BIGTOP-3300. Add puppet manifests for hadoop-kms. #615

iwasakims · 2020-03-12T05:16:55Z

No description provided.

iwasakims · 2020-03-13T14:25:42Z

I manually tested that HDFS transparent encryption works on following config.yaml::

docker:
        memory_limit: "8g"
        image: "bigtop/puppet:trunk-centos-7"
distro: centos
components: [hdfs, yarn, kms]
enable_local_repo: true
smoke_test_components: [hdfs, yarn]

test steps::

$ cd provisioner/docker
$ ./docker-hadoop.sh -c 3
$ ./docker-hadoop.sh --exec 3 /bin/bash

 # hdfs dfs -mkdir /user/root/zone1
 
 # hadoop key create key1
 key1 has been successfully created with options Options{cipher='AES/CTR/NoPadding', bitLength=128, description='null', attributes=null}.
 org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@1dde4cb2 has been updated.
 
 # sudo -u hdfs hdfs crypto -createZone -keyName key1 -path /user/root/zone1
 Added encryption zone /user/root/zone1
 
 # hdfs dfs -put /etc/hosts /user/root/zone1/
 # hdfs dfs -get /user/root/zone1/hosts /tmp/

iwasakims · 2020-03-13T14:26:56Z

also tested on kerberos enabled cluster on config.yaml::

docker:
        memory_limit: "8g"
        image: "bigtop/puppet:trunk-centos-7"
distro: centos
components: [kerberos, hdfs, yarn, kms]
enable_local_repo: true
smoke_test_components: [hdfs]

with the following configs in bigtop-deploy/puppet/hieradata/bigtop/cluster.yaml::

# Kerberos
hadoop::hadoop_security_authentication: "kerberos"
kerberos::krb_site::domain: "bigtop.apache.org"
kerberos::krb_site::realm: "BIGTOP.APACHE.ORG"
kerberos::krb_site::kdc_server: "%{hiera('bigtop::hadoop_head_node')}"
kerberos::krb_site::kdc_port: "88"
kerberos::krb_site::admin_port: "749"
kerberos::krb_site::keytab_export_dir: "/var/lib/bigtop_keytabs"
hadoop::kerberos_realm: "%{hiera('kerberos::krb_site::realm')}"

test steps::

$ cd provisioner/docker
$ ./docker-hadoop.sh -c 3
$ ./docker-hadoop.sh --exec 3 /bin/bash

 # kinit -kt /etc/hdfs.keytab hdfs/$(hostname --fqdn)
 # hadoop key create key1
 # hdfs dfs -mkdir /zone1
 # hdfs crypto -createZone -keyName key1 -path /zone1
 # hdfs dfs -put /etc/hosts /zone1/
 # hdfs dfs -get /zone1/hosts /tmp/

iwasakims · 2020-03-13T14:30:01Z

TODOs in follow-up JIRAs are

HA deployment with multiple KMS instances
https enabled configruation

evans-ye · 2020-03-14T15:20:09Z

bigtop-deploy/puppet/hieradata/bigtop/cluster.yaml

+#kerberos::krb_site::kdc_server: "%{hiera('bigtop::hadoop_head_node')}"
+#kerberos::krb_site::kdc_port: "88"
+#kerberos::krb_site::admin_port: "749"
+#kerberos::krb_site::keytab_export_dir: "/var/lib/bigtop_keytabs"


May I know why the name is changed to from site to krb_site? I think this may break compatibility.

The Hiera variables are not injected unless the name space of properties match the class name of Puppet classes. The relevant class name was changed from site to krb_site in 3386a9d. The commented out configs (as example) seem to be left unchanged.

OK. That was me ;) Glad you catch this. Thanks!

evans-ye · 2020-03-14T15:21:39Z

bigtop-deploy/puppet/modules/hadoop/templates/core-site.xml

+<% if @kms_host %>
+  <property>
+    <name>hadoop.security.key.provider.path</name>
+    <value>kms://http@<%= @kms_host %>:<%= @kms_port %>/kms</value>


I'm zero knowledge about this, just wondering can this be https?

Yes. We need additional configurations in files such as ssl-client.xml, ssl-server.xml and server.xml (of Tomcat) for that. I would like to address that in another JIRA since all services of HDFS and YARN should be cared when we enable HTTPS on Web-UI and REST API.

Sure. No problem and thanks for the explanation.

evans-ye · 2020-03-14T15:22:42Z

bigtop-deploy/puppet/modules/kerberos/manifests/init.pp

-      default          => [ ],
+      true      => [ 'HTTP' ],
+      'enabled' => [ 'HTTP' ],
+      default   => [ ],


The original code won't work hence you refactor here?

Yes. The credential of HTTP/host@REALM did not written to keytab file since boolean true was not covered in the conditional of previous code.

evans-ye · 2020-03-14T15:23:22Z

This is super awesome feature! I've left some comments. Thanks!

evans-ye · 2020-03-15T11:24:44Z

Looks nice. +1.

evans-ye · 2020-03-15T11:28:19Z

bigtop-deploy/puppet/modules/hadoop/templates/kms-site.xml

+<% if @hadoop_security_authentication == "kerberos" -%>
+  <property>
+    <name>hadoop.kms.authentication.kerberos.keytab</name>
+    <value>/etc/kms.keytab</value>


I'm not aware that there's a convention to put keytab under /etc, is it?

I used the keytab created by kerberos::host_keytab resource. Other modules using the host_keytab resource such as HDFS and YARN seem to follow this convention as I can see.

Got It. Pretty comprehensive. Thanks!

evans-ye · 2020-03-15T11:29:32Z

bigtop-deploy/puppet/modules/hadoop/templates/kms-site.xml

+
+  <property>
+    <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.principal</name>
+    <value>kms/#HOSTNAME#</value>


Is this intentional for replacement?

This part came from the original kms-site.xml bundled with Hadoop. The value is used only if the value of hadoop.kms.authentication.signer.secret.provider is changed to zookeeper. ZKSignerSecretProvider is a feature for HA setup by which multiple KMS instances share the same signer secret via ZooKeeper. Since KMS HA is not supported in this patch, it is left as is.

Thanks for the detailed explanation!

iwasakims · 2020-03-16T00:59:06Z

Thanks, @evans-ye.

* BIGTOP-3300. Add puppet manifests for hadoop-kms. * fixed role assingment, kms kerberos configs and bugs in kerberos module. * tightened permission of kms-env.sh containing keystore password.

iwasakims added 2 commits March 13, 2020 14:19

BIGTOP-3300. Add puppet manifests for hadoop-kms.

de495ca

fixed role assingment, kms kerberos configs and bugs in kerberos module.

81b991b

iwasakims force-pushed the BIGTOP-3300 branch from 9a4112b to 81b991b Compare March 13, 2020 14:20

tightened permission of kms-env.sh containing keystore password.

93f8a85

evans-ye reviewed Mar 14, 2020

View reviewed changes

evans-ye reviewed Mar 15, 2020

View reviewed changes

evans-ye merged commit 6307ef3 into apache:master Mar 15, 2020

BIGTOP-3300. Add puppet manifests for hadoop-kms. #615

BIGTOP-3300. Add puppet manifests for hadoop-kms. #615

Uh oh!

Conversation

iwasakims commented Mar 12, 2020

Uh oh!

iwasakims commented Mar 13, 2020

Uh oh!

iwasakims commented Mar 13, 2020

Uh oh!

iwasakims commented Mar 13, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

evans-ye commented Mar 14, 2020

Uh oh!

evans-ye commented Mar 15, 2020

Uh oh!

evans-ye Mar 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

evans-ye Mar 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iwasakims commented Mar 16, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

evans-ye Mar 15, 2020 •

edited

Loading

evans-ye Mar 15, 2020 •

edited

Loading