Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-27347 Port FileWatcher from ZK to autodetect keystore/truststore changes in TLS connections #4869

Merged
merged 6 commits into from Nov 11, 2022

Conversation

anmolnar
Copy link
Contributor

@anmolnar anmolnar commented Nov 9, 2022

This patch is the port of ZooKeeper's FileWatcher's functionality which we can take advantage to detect changes in truststore / keystore files for TLS. Cert / key renewal processes don't need HBase services to be restarted with this patch.

cc @bbeaudreault @Apache9

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 30s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 10s Maven dependency ordering for branch
+1 💚 mvninstall 3m 32s master passed
+1 💚 compile 4m 40s master passed
+1 💚 checkstyle 1m 24s master passed
+1 💚 spotless 0m 54s branch has no errors when running spotless:check.
+1 💚 spotbugs 3m 25s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 10s Maven dependency ordering for patch
+1 💚 mvninstall 2m 47s the patch passed
+1 💚 compile 3m 38s the patch passed
+1 💚 javac 3m 38s the patch passed
+1 💚 checkstyle 1m 1s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 9m 18s Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚 spotless 0m 41s patch has no errors when running spotless:check.
+1 💚 spotbugs 3m 36s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 23s The patch does not generate ASF License warnings.
43m 34s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #4869
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 24d85d40da07 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 41c7bd3
Default Java Eclipse Adoptium-11.0.17+8
Max. process+thread count 79 (vs. ulimit of 30000)
modules C: hbase-common hbase-client hbase-server U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/1/console
versions git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@wchevreuil wchevreuil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, overall, just have some minor remarks.

}
// Note: we don't care about delete events
if (shouldResetContext) {
if (LOG.isDebugEnabled()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Maybe worth info logging? My thought here is that this wouldn't be a frequent event, yet a important one to get logged at higher level than debugging.

resetContext.run();
} else {
if (LOG.isDebugEnabled()) {
LOG.debug("Ignoring watch event and keeping previous default SSL context. Event kind: "
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: parameterized logging?

* revision</a>
*/
@InterfaceAudience.Private
public final class FileChangeWatcher {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a straight copy from Zookeeper? Couldn't we just reuse the ZK impl directly, as we already have ZK as a dependency?

Copy link
Contributor Author

@anmolnar anmolnar Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Urm... I don't have a strong opinion. We were following this pattern in the entire implementation of TLS. It's probably better to stay on the safe side and avoid sideeffects of ZK non-backward compatible changes. They're quite unlikely though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for clarifying. Not a big deal for me. And on the flip side, it would be actually a bit weird to depend on ZK for a functionality that isn't really ZK specific, so I'm ok with this copy approach.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 45s Docker mode activated.
-0 ⚠️ yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 12s Maven dependency ordering for branch
+1 💚 mvninstall 2m 33s master passed
+1 💚 compile 1m 9s master passed
+1 💚 shadedjars 3m 50s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 48s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 11s Maven dependency ordering for patch
+1 💚 mvninstall 2m 28s the patch passed
+1 💚 compile 1m 10s the patch passed
+1 💚 javac 1m 10s the patch passed
+1 💚 shadedjars 3m 53s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 47s the patch passed
_ Other Tests _
+1 💚 unit 1m 56s hbase-common in the patch passed.
+1 💚 unit 1m 17s hbase-client in the patch passed.
+1 💚 unit 193m 26s hbase-server in the patch passed.
219m 1s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #4869
Optional Tests javac javadoc unit shadedjars compile
uname Linux d4cc90e5e458 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 41c7bd3
Default Java Eclipse Adoptium-11.0.17+8
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/1/testReport/
Max. process+thread count 2449 (vs. ulimit of 30000)
modules C: hbase-common hbase-client hbase-server U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/1/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 23s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗 mvndep 0m 16s Maven dependency ordering for branch
+1 💚 mvninstall 3m 7s master passed
+1 💚 compile 4m 8s master passed
+1 💚 checkstyle 1m 10s master passed
+1 💚 spotless 0m 49s branch has no errors when running spotless:check.
+1 💚 spotbugs 3m 16s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 9s Maven dependency ordering for patch
+1 💚 mvninstall 2m 41s the patch passed
+1 💚 compile 4m 4s the patch passed
+1 💚 javac 4m 4s the patch passed
-0 ⚠️ checkstyle 0m 16s hbase-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 10m 35s Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚 spotless 0m 51s patch has no errors when running spotless:check.
+1 💚 spotbugs 4m 17s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 36s The patch does not generate ASF License warnings.
46m 49s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #4869
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 2ad66b07bbad 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / da99738
Default Java Eclipse Adoptium-11.0.17+8
checkstyle https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-common.txt
Max. process+thread count 80 (vs. ulimit of 30000)
modules C: hbase-common hbase-client hbase-server U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/console
versions git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 34s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 11s Maven dependency ordering for branch
+1 💚 mvninstall 2m 59s master passed
+1 💚 compile 1m 31s master passed
+1 💚 shadedjars 4m 29s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 57s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 10s Maven dependency ordering for patch
+1 💚 mvninstall 2m 42s the patch passed
+1 💚 compile 1m 27s the patch passed
+1 💚 javac 1m 27s the patch passed
+1 💚 shadedjars 4m 14s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 54s the patch passed
_ Other Tests _
+1 💚 unit 2m 1s hbase-common in the patch passed.
+1 💚 unit 1m 21s hbase-client in the patch passed.
+1 💚 unit 212m 29s hbase-server in the patch passed.
241m 11s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #4869
Optional Tests javac javadoc unit shadedjars compile
uname Linux bd1885fef336 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / da99738
Default Java Temurin-1.8.0_352-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/testReport/
Max. process+thread count 2376 (vs. ulimit of 30000)
modules C: hbase-common hbase-client hbase-server U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 13s Docker mode activated.
-0 ⚠️ yetus 0m 5s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 13s Maven dependency ordering for branch
+1 💚 mvninstall 2m 42s master passed
+1 💚 compile 1m 24s master passed
+1 💚 shadedjars 3m 47s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 1m 0s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 13s Maven dependency ordering for patch
+1 💚 mvninstall 2m 36s the patch passed
+1 💚 compile 1m 26s the patch passed
+1 💚 javac 1m 26s the patch passed
+1 💚 shadedjars 3m 46s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 57s the patch passed
_ Other Tests _
+1 💚 unit 2m 27s hbase-common in the patch passed.
+1 💚 unit 1m 23s hbase-client in the patch passed.
+1 💚 unit 241m 19s hbase-server in the patch passed.
270m 18s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #4869
Optional Tests javac javadoc unit shadedjars compile
uname Linux 8c48b0875401 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / da99738
Default Java Eclipse Adoptium-11.0.17+8
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/testReport/
Max. process+thread count 2484 (vs. ulimit of 30000)
modules C: hbase-common hbase-client hbase-server U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4869/2/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@wchevreuil wchevreuil merged commit 047f4e2 into apache:master Nov 11, 2022
@anmolnar anmolnar deleted the HBASE-27347 branch November 13, 2022 17:39
@anmolnar
Copy link
Contributor Author

Thanks @wchevreuil for merging the patch. Please cherry pick it to branch-2 as well.

bbeaudreault pushed a commit to HubSpot/hbase that referenced this pull request Nov 16, 2022
…e changes in TLS connections (apache#4869)

Signed-off-by: wchevreuil@apache.org
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants