Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-1065. OM and DN should persist SCM certificate as the trust root. Contributed by Ajay Kumar. #754

Closed
wants to merge 140 commits into from

Conversation

ajayydv
Copy link
Contributor

@ajayydv ajayydv commented Apr 19, 2019

No description provided.

@ajayydv ajayydv added the ozone label Apr 19, 2019
@ajayydv ajayydv requested a review from xiaoyuyao April 19, 2019 00:21
@ajayydv ajayydv self-assigned this Apr 19, 2019
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 30 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
0 mvndep 63 Maven dependency ordering for branch
+1 mvninstall 1095 trunk passed
+1 compile 1084 trunk passed
+1 checkstyle 142 trunk passed
+1 mvnsite 201 trunk passed
+1 shadedclient 1054 branch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 183 trunk passed
+1 javadoc 131 trunk passed
_ Patch Compile Tests _
0 mvndep 23 Maven dependency ordering for patch
+1 mvninstall 140 the patch passed
+1 compile 1040 the patch passed
+1 javac 1040 the patch passed
+1 checkstyle 134 the patch passed
+1 mvnsite 157 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 649 patch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 209 the patch passed
+1 javadoc 124 the patch passed
_ Other Tests _
-1 unit 75 common in the patch failed.
+1 unit 63 container-service in the patch passed.
-1 unit 1266 integration-test in the patch failed.
+1 unit 62 ozone-manager in the patch passed.
+1 asflicense 57 The patch does not generate ASF License warnings.
7910
Reason Tests
Failed junit tests hadoop.hdds.scm.net.TestNodeSchemaManager
hadoop.hdds.scm.net.TestNetworkTopologyImpl
hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
hadoop.ozone.TestOzoneConfigurationFields
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
hadoop.hdds.scm.pipeline.TestRatisPipelineUtils
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.om.TestContainerReportWithKeys
hadoop.ozone.client.rpc.TestCommitWatcher
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-754/1/artifact/out/Dockerfile
GITHUB PR #754
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 051e1ddd5144 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 518f47b
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-754/1/artifact/out/patch-unit-hadoop-hdds_common.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-754/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-754/1/testReport/
Max. process+thread count 3724 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/container-service hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-754/1/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 24 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
0 mvndep 322 Maven dependency ordering for branch
+1 mvninstall 1058 trunk passed
+1 compile 995 trunk passed
+1 checkstyle 136 trunk passed
+1 mvnsite 254 trunk passed
+1 shadedclient 1058 branch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 174 trunk passed
+1 javadoc 124 trunk passed
_ Patch Compile Tests _
0 mvndep 21 Maven dependency ordering for patch
+1 mvninstall 142 the patch passed
+1 compile 907 the patch passed
+1 javac 907 the patch passed
+1 checkstyle 133 the patch passed
+1 mvnsite 159 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 615 patch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 212 the patch passed
+1 javadoc 128 the patch passed
_ Other Tests _
-1 unit 68 common in the patch failed.
+1 unit 65 container-service in the patch passed.
-1 unit 905 integration-test in the patch failed.
+1 unit 50 ozone-manager in the patch passed.
+1 asflicense 40 The patch does not generate ASF License warnings.
7444
Reason Tests
Failed junit tests hadoop.hdds.scm.net.TestNodeSchemaManager
hadoop.hdds.scm.net.TestNetworkTopologyImpl
hadoop.ozone.TestOzoneConfigurationFields
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
hadoop.hdds.scm.pipeline.TestRatisPipelineUtils
hadoop.ozone.client.rpc.TestCommitWatcher
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-754/2/artifact/out/Dockerfile
GITHUB PR #754
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 0ce971fa6b0a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 317fcba
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-754/2/artifact/out/patch-unit-hadoop-hdds_common.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-754/2/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-754/2/testReport/
Max. process+thread count 4310 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/container-service hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-754/2/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

yangwwei and others added 3 commits April 20, 2019 21:51
…correct units when the default value is used. Contributed by starphin.
…AM heartbeat. Contributed by Abhishek Modi.
datanodeDetails.setCertSerialId(getX509Certificate(pemEncodedCert).
getSerialNumber().toString());
persistDatanodeDetails(datanodeDetails);
// Get SCM CA certificate and store it in filesystem.
String pemEncodedRootCert = secureScmClient.getCACertificate();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we get the CA certificate based on the DN certificate signed by CA? Does that container a signer certificate id?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As of now we don't have functionality to look up certificates by subject or scm id. getCACertificate returns default certificate for SCM who signed it.

@@ -80,6 +80,7 @@
public abstract class DefaultCertificateClient implements CertificateClient {

private static final String CERT_FILE_NAME_FORMAT = "%s.crt";
private static final String CA_CERT_PREFIX = "CA-";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remind me where do we actually use this root CA certificate in the code? I don't see reference in this patch. Should we use it in Block token verification?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes for block token and DT validation. It will be used to establish trust of chain.

Copy link
Contributor

@xiaoyuyao xiaoyuyao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. Just two questions added inline.

Inigo Goiri and others added 15 commits April 22, 2019 13:23
… handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen.
…ationTokenSecretManager. Contributed by CR Hota.
…ds for containers in closing state after a restart. (apache#755)
…estBase.testUpdateTrackingUrl fails intermittent. Contributed by Prabhu Joseph.
… fails intermittent. Contributed by Prabhu Joseph.
…core.

Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
…tInterceptor#invokeConcurrent. Contributed by Shen Yinjie.
bibinchundatt and others added 13 commits May 15, 2019 13:30
…vided storage. Contributed by Ashvin Agrawal
Adding a protected-scope getter for the DistCpOptions, so that a subclass does
not need to save its own copy of the inputOptions supplied to its constructor,
if it wishes to override the createInputFileListing method with logic similar
to the original implementation, i.e. calling CopyListing#buildListing with a path and input options.

Author:    Andrew Olson
* @throws CertificateException - on Error.
*
*/
void storeCertificate(String pemEncodedCert, boolean force)
void storeCertificate(String pemEncodedCert, boolean force, boolean caCert)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't you write a new function, called storeRootCertificate. That avoids adding this false argument to lots of other parts of the code.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, let's add a new function as @anuengineer suggested.

macroadster and others added 6 commits May 16, 2019 16:39
…s special characters in hdfs file path. Contributed by Srinivasu Majeti.

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 0 Docker mode activated.
-1 patch 40 #754 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
Subsystem Report/Notes
GITHUB PR #754
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-754/3/console
versions git=2.7.4
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

arp7 and others added 5 commits May 20, 2019 09:09
apache#725)

* HDDS-1422. Exception during DataNode shutdown. Contributed by Arpit Agarwal.

Change-Id: I6db6bdd19839a45e5341ed7e745cd38f68af8378

* Suppress spurious findbugs warning.

* Remove log file that got committed in error
… inconsistent read after replace/overwrite.

Contributed by Ben Roling.

S3Guard will now track the etag of uploaded files and, if an S3
bucket is versioned, the object version.

You can then control how to react to a mismatch between the data
in the DynamoDB table and that in the store: warn, fail, or, when
using versions, return the original value.

This adds two new columns to the table: etag and version.
This is transparent to older S3A clients -but when such clients
add/update data to the S3Guard table, they will not add these values.
As a result, the etag/version checks will not work with files uploaded by older clients.

For a consistent experience, upgrade all clients to use the latest hadoop version.
@ajayydv ajayydv closed this May 20, 2019
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 0 Docker mode activated.
-1 patch 51 #754 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
Subsystem Report/Notes
GITHUB PR #754
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-754/4/console
versions git=2.7.4
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@ajayydv ajayydv deleted the HDDS-1065 branch May 20, 2019 16:20
shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
Author: Prateek Maheshwari <pmaheshwari@apache.org>

Closes apache#754 from prateekm/version-updates
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet