[AMBARI-24616] Disable Kerberos from Ambari UI didn't clean up keytab directories #2334

rlevas · 2018-09-17T15:29:44Z

What changes were proposed in this pull request?

Disable Kerberos from Ambari UI didn't clean up keytab directories,

stderr:

2018-09-08 05:27:19,276 - Failed to remove identity for amsmon/ctr-e138-1518143905142-467151-01-000002.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:19,298 - Failed to remove identity for amsmon/ctr-e138-1518143905142-467151-01-000006.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:19,325 - Failed to remove identity for amsmon/ctr-e138-1518143905142-467151-01-000005.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:19,348 - Failed to remove identity for amsmon/ctr-e138-1518143905142-467151-01-000003.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:19,465 - Failed to remove identity for dn/ctr-e138-1518143905142-467151-01-000003.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:19,491 - Failed to remove identity for dn/ctr-e138-1518143905142-467151-01-000002.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:19,515 - Failed to remove identity for dn/ctr-e138-1518143905142-467151-01-000005.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:19,539 - Failed to remove identity for dn/ctr-e138-1518143905142-467151-01-000004.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:19,671 - Failed to remove identity for hbase/ctr-e138-1518143905142-467151-01-000006.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:19,696 - Failed to remove identity for hbase/ctr-e138-1518143905142-467151-01-000003.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:19,723 - Failed to remove identity for hbase/ctr-e138-1518143905142-467151-01-000004.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:19,744 - Failed to remove identity for hbase/ctr-e138-1518143905142-467151-01-000002.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:19,959 - Failed to remove identity for nm/ctr-e138-1518143905142-467151-01-000005.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:19,987 - Failed to remove identity for nm/ctr-e138-1518143905142-467151-01-000006.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:20,049 - Failed to remove identity for nn/ctr-e138-1518143905142-467151-01-000003.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:20,376 - Failed to remove identity for HTTP/ctr-e138-1518143905142-467151-01-000002.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:20,399 - Failed to remove identity for HTTP/ctr-e138-1518143905142-467151-01-000004.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:20,420 - Failed to remove identity for HTTP/ctr-e138-1518143905142-467151-01-000003.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:20,441 - Failed to remove identity for HTTP/ctr-e138-1518143905142-467151-01-000005.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:20,590 - Failed to remove identity for yarn-ats-hbase/ctr-e138-1518143905142-467151-01-000003.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:20,617 - Failed to remove identity for yarn-ats-hbase/ctr-e138-1518143905142-467151-01-000002.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:20,647 - Failed to remove identity for yarn-ats-hbase/ctr-e138-1518143905142-467151-01-000004.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:20,677 - Failed to remove identity for yarn-ats-hbase/ctr-e138-1518143905142-467151-01-000005.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:20,768 - Failed to remove identity for zookeeper/ctr-e138-1518143905142-467151-01-000006.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type.
2018-09-08 05:27:20,798 - Failed to remove identity for zookeeper/ctr-e138-1518143905142-467151-01-000004.hwx.site@EXAMPLE.COM from the Ambari database - Object: null is not a known Entity type

[root@ctr-e138-1518143905142-467151-01-000002 ~]# ls -lrt /etc/security/keytabs/
total 56
-rw-r----- 1 ambari-qa  hadoop 318 Sep  7 23:00 kerberos.service_check.090718.keytab
-r-------- 1 slava      slava  353 Sep  7 23:05 ambari.server.keytab
-r--r----- 1 root       hadoop 538 Sep  7 23:05 spnego.service.keytab
-r-------- 1 cstm-ams   hadoop 548 Sep  7 23:05 ams-monitor.keytab
-r-------- 1 cstm-hdfs  hadoop 533 Sep  7 23:05 nfs.service.keytab
-r--r----- 1 cstm-hbase hadoop 338 Sep  7 23:05 hbase.headless.keytab
-r-------- 1 yarn-ats   hadoop 328 Sep  7 23:05 yarn-ats.hbase-client.headless.keytab
-r-------- 1 cstm-hdfs  hadoop 528 Sep  7 23:05 dn.service.keytab
-r-------- 1 yarn-ats   hadoop 588 Sep  7 23:05 yarn-ats.hbase-regionserver.service.keytab
-r--r----- 1 ambari-qa  hadoop 333 Sep  7 23:05 smokeuser.headless.keytab
-r-------- 1 cstm-hbase hadoop 543 Sep  7 23:05 hbase.service.keytab
-r-------- 1 cstm-hdfs  hadoop 528 Sep  7 23:05 nn.service.keytab
-r-------- 1 yarn-ats   hadoop 588 Sep  7 23:05 yarn-ats.hbase-master.service.keytab
-r-------- 1 cstm-hdfs  hadoop 333 Sep  7 23:05 hdfs.headless.keytab

This occurred for several reasons related to many iterations of changes to the Kerberos enable and clean up processes. This patch attempts to fix the inconsistencies the lead to the cleanup failures - whether a service, component, or host was removed or Kerberos was being disabled. Now the keytab files, principals/accounts, and Ambari DB records are being properly cleaned up.

How was this patch tested?

Manually tested various scenarios when Kerberos was enabled

remove a service
remove a component
move a component
remove a host
regenerate keytab files
regenerate missing keytab file
disable Kerberos
Ambari upgrade
stack upgrade

Unit tests were updated and all passed.

Please review Ambari Contributing Guide before opening a pull request.

… directories

rlevas · 2018-09-17T15:30:28Z

Sorry for the large patch, there were lots of changes needed to clean up the code.

asfgit · 2018-09-17T16:15:43Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/3982/
Test PASSed.

ghost · 2018-09-17T16:15:34Z

ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java

+
        // *****************************************************************
        // Create stage to remove principals
+        // - this should be the last opterion that deals with principals and keytab files since the


Thanks. Fixing.

ghost · 2018-09-17T16:15:51Z

ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java

+
        // *****************************************************************
        // Create stage to delete principals
+        // - this should be the last opterion that deals with principals and keytab files since the


Thanks. Fixing.

...-server/src/main/java/org/apache/ambari/server/controller/utilities/RemovableIdentities.java

ghost · 2018-09-17T16:17:26Z

...-server/src/main/java/org/apache/ambari/server/events/publishers/AgentCommandsPublisher.java

+  }
+
+  /**
+   * KerberosCommandParameterProcessor is an abstract class providing common implementions for processing


implementions

thanks.. fixing.

ghost · 2018-09-17T16:19:45Z

...-server/src/main/java/org/apache/ambari/server/events/publishers/AgentCommandsPublisher.java

+
+        if (keytabFilePath != null) {
+          String sha1Keytab = DigestUtils.sha256Hex(keytabFilePath);
+          File keytabFile = new File(dataDir + File.separator + hostName + File.separator + sha1Keytab);


you might prefer https://docs.oracle.com/javase/8/docs/api/java/nio/file/Paths.html#get-java.lang.String-java.lang.String...- as it is less error-prone

I totally agree, but this was the code that was already there and I didn't want to mess with it.

ncole

A couple import fixes and nits (optional).

ncole · 2018-09-17T18:36:31Z

ambari-server/src/main/java/org/apache/ambari/server/controller/DeleteIdentityHandler.java

 import org.apache.ambari.server.state.kerberos.KerberosDescriptor;
 import org.apache.ambari.server.state.svccomphost.ServiceComponentHostServerActionEvent;
 import org.apache.ambari.server.utils.StageUtils;
+import org.springframework.util.CollectionUtils;


Should probably be org.apache.commons.collections.CollectionUtils

I knew I was going to mess this up. Fixed other places where my IDE choose the wrong impl. Nice catch.

ncole · 2018-09-17T18:37:16Z

ambari-server/src/main/java/org/apache/ambari/server/controller/DeleteIdentityHandler.java

-      requestParams);
-    customCommandExecutionHelper.addExecutionCommandsToStage(actionExecContext, stage, requestParams, null);
-    stageContainer.addStage(stage);
+    if(!CollectionUtils.isEmpty(hostNames)) {


nit: CollectionUtils.isNotEmpty(...)

ncole · 2018-09-17T18:37:43Z

ambari-server/src/main/java/org/apache/ambari/server/controller/DeleteIdentityHandler.java

+
+      ActionExecutionContext actionExecContext = new ActionExecutionContext(
+          cluster.getClusterName(),
+          "REMOVE_KEYTAB",


Do we have a constant somewhere for this?

ncole · 2018-09-17T18:39:35Z

ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosKeytabDAO.java

+import org.apache.ambari.server.orm.entities.KerberosKeytabPrincipalEntity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.util.CollectionUtils;


org.apache.commons.collections.CollectionUtils

ncole · 2018-09-17T18:40:02Z

ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosKeytabPrincipalDAO.java

 import org.apache.ambari.server.orm.entities.KerberosKeytabPrincipalEntity;
 import org.apache.ambari.server.orm.entities.KerberosKeytabServiceMappingEntity;
 import org.apache.ambari.server.orm.entities.KerberosPrincipalEntity;
+import org.springframework.util.CollectionUtils;


org.apache.commons.collections.CollectionUtils

ncole · 2018-09-17T18:40:56Z

ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosKeytabDAO.java

+   */
+  public boolean removeIfNotReferenced(KerberosKeytabEntity kerberosKeytabEntity) {
+    if (kerberosKeytabEntity != null) {
+      if (!CollectionUtils.isEmpty(kerberosKeytabEntity.getKerberosKeytabPrincipalEntities())) {


nit: CollectionUtils.isNotEmpty(...)

… directories

asfgit · 2018-09-17T22:03:51Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/3985/
Test FAILed.
Test FAILured.

asfgit · 2018-09-17T22:07:16Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/3988/
Test PASSed.

ambari-server/src/main/java/org/apache/ambari/server/controller/DeleteIdentityHandler.java

[AMBARI-24616] Disable Kerberos from Ambari UI didn't clean up keytab…

6bb9ea0

… directories

rlevas added kerberos cleanup labels Sep 17, 2018

rlevas self-assigned this Sep 17, 2018

rlevas requested review from a user, jonathan-hurley, ncole, rnettleton, smolnar82 and zeroflag September 17, 2018 15:29

rlevas mentioned this pull request Sep 17, 2018

[AMBARI-24616] Disable Kerberos from Ambari UI didn't clean up keytab directories #2335

Merged

ghost approved these changes Sep 17, 2018

View reviewed changes

rnettleton approved these changes Sep 17, 2018

View reviewed changes

ncole approved these changes Sep 17, 2018

View reviewed changes

rlevas added 2 commits September 17, 2018 16:39

[AMBARI-24616] Disable Kerberos from Ambari UI didn't clean up keytab…

1186c2f

… directories

[AMBARI-24616] Disable Kerberos from Ambari UI didn't clean up keytab…

7d47a62

… directories

zeroflag approved these changes Sep 18, 2018

View reviewed changes

smolnar82 reviewed Sep 18, 2018

View reviewed changes

ambari-server/src/main/java/org/apache/ambari/server/controller/DeleteIdentityHandler.java Show resolved Hide resolved

rlevas merged commit 620539f into apache:trunk Sep 18, 2018

[AMBARI-24616] Disable Kerberos from Ambari UI didn't clean up keytab directories #2334

[AMBARI-24616] Disable Kerberos from Ambari UI didn't clean up keytab directories #2334

Uh oh!

Conversation

rlevas commented Sep 17, 2018

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

rlevas commented Sep 17, 2018

Uh oh!

asfgit commented Sep 17, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ncole left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asfgit commented Sep 17, 2018

Uh oh!

asfgit commented Sep 17, 2018

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants