New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ambari 23549] Blueprint configuration processor updates for HDFS NameNode Federation #985

Merged
merged 11 commits into from Apr 16, 2018

Conversation

Projects
None yet
4 participants
@rnettleton
Contributor

rnettleton commented Apr 12, 2018

What changes were proposed in this pull request?

The Blueprint configuration processor was updated to account for HDFS NameNode Federation deployments. The following changes were introduced:

  1. The "hadoop-env/dfs_ha_initial_cluster_id" property will now be set automatically by the Blueprint configuration processor to be the cluster's name, unless this setting was already customized by the user.

  2. In addition to setting the cluster ID, this change should also implement some minor configuration conveniences that will make Blueprint creation for Federated clusters simpler:

    • Add property updater support for "dfs.namenode.shared.edits.dir.NAMESERVICE" properties.
    • Add property update support for "dfs.namenode.servicerpc-address.NAMESERVICE.NAMENODE" properties
  3. Unit tests were updated to validate this change.

How was this patch tested?

  1. Ran the ambari-server unit test suite (passed).

  2. Deployed a 4-node HDFS NameNode Federation cluster using a Blueprint, and verified that the HDFS services are all started as expected, and that the hdfs-site configuration includes the expected configuration values.

Please review Ambari Contributing Guide before opening a pull request.

@rnettleton rnettleton self-assigned this Apr 12, 2018

@rnettleton rnettleton requested review from adoroszlai and benyoka Apr 12, 2018

clusterConfig.setProperty(HADOOP_ENV_CONFIG_TYPE_NAME, HDFS_HA_INITIAL_CLUSTER_ID_PROPERTY_NAME, clusterName);
} else {
LOG.warn("Cluster name could not obtained, this may indicate a deployment or configuration error.");
}

This comment has been minimized.

@benyoka

benyoka Apr 12, 2018

Contributor

Shouldn't missing cluster name mean an exception or are there valid cases when cluster name could be missing?

This comment has been minimized.

@rnettleton

rnettleton Apr 13, 2018

Contributor

That's a good question. I've checked the usages of the getClusterName() method throughout the server code, and some cases treat this as an exception case, and some just proceed assuming that the name is always non-null.

I don't think there are any valid cases when the cluster name would be missing, so I'll update this code to throw an exception in the event that a null cluster name is found.

This comment has been minimized.

@rnettleton

rnettleton Apr 13, 2018

Contributor

Updated the patch to include more error-checking in this area, and added a unit test to verify that the expected exception is thrown in the event that getClusterName() returns null.

@asfgit

This comment has been minimized.

asfgit commented Apr 12, 2018

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/1818/
Test FAILed.
Test FAILured.

@adoroszlai

This comment has been minimized.

Contributor

adoroszlai commented Apr 13, 2018

@rnettleton When merging this, please make sure to select "Squash and merge" from the dropdown.

https://help.github.com/articles/about-pull-request-merges/#squash-and-merge-your-pull-request-commits

@benyoka

LGTM, one minor question

@asfgit

This comment has been minimized.

asfgit commented Apr 13, 2018

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/1850/
Test PASSed.

@rnettleton rnettleton merged commit 4f0fbd2 into apache:trunk Apr 16, 2018

1 check passed

Jenkins Build SUCCESS No test results found.
Details

@rnettleton rnettleton deleted the rnettleton:AMBARI-23549-trunk branch Apr 16, 2018

nitirajrathore added a commit to nitirajrathore/ambari that referenced this pull request Apr 26, 2018

[Ambari 23549] Blueprint configuration processor updates for HDFS Nam…
…eNode Federation (apache#985)

* Adding property substitution support for various Federation-related properties

* Minor changes to Blueprint Configuration Processor for NameNode Federation.

* Updates to unit tests and minor refactorings

* Updates to unit tests and minor changes.

* Updated error handling for cluster name and new unit test assertions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment