Add support for kerberized solr instance (#1138)

[jackson-chris]

Fixes issue #1138.

Signed-off-by: Christopher Jackson chris.jackson@us.ibm.com

---

Thank you for contributing to JanusGraph!

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

• Is there an issue associated with this PR? Is it referenced in the commit message?
• Does your PR body contain #xyz where xyz is the issue number you are trying to resolve?
• Has your PR been rebased against the latest commit within the target branch (typically master)?
• Is your initial contribution a single, squashed commit?

For code changes:

• Have you written and/or updated unit tests to verify your changes?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE.txt file, including the main LICENSE.txt file in the root of this repository?
• If applicable, have you updated the NOTICE.txt file, including the main NOTICE.txt file found in the root of this repository?

For documentation related changes:

• Have you ensured that format looks appropriate for the output in which it is rendered?
• If this PR is a documentation-only change, have you added a [skip ci]
  tag to the first line of your commit message to avoid spending CPU cycles in
  Travis CI when no code, tests, or build configuration are modified?

Note:

Please ensure that once the PR is submitted, you check Travis CI for build issues and submit an update to your PR as soon as possible.

[jerryjch]

Thanks for the PR.
My main question is that if HBase (which uses Zookeeper) is Kerberized, how is the configuration going to affect HBase's access to Zookeeper. Can the keytab files or jaas files be shared to access the same Zookeeper, or will there be any conflicts?

[jackson-chris]

Background

HBase will use the typical hadoop configuration files to configure itself. For example the files founds in /etc/hbase/conf and /etc/hadoop/conf. If Hadoop Core Services (HDFS, YARN, etc.) and HBase are configured for Kerberos it will be reflected by the pertinent properties in these files.

Both Zookeeper and SOLR gets its kerberos configuration from flags set when the corresponding service processes are started.

Each of these aforementioned services will have their own JAAS configuration files. In each of those files will be various JAAS configuration elements, some for the server processes of that service and potentially some for the client processes. Sometimes the client/server portions are split out into unique jass configuration files.

Keytabs aren't typically shared between the services. Each service will have a unique principal, sometimes multiple and the keytab is a stored credential(s) for a specific principal(s).

Janus on Kerberized Hadoop

When working with Janus in a Hadoop environment, lets say using HBase as the storage backend and SOLR as the indexing back end there are two main things that need to happen for Janus to work properly:

1. The HBase and Hadoop configuration folders (ex. /etc/hbase/conf and /etc/hadoop/conf) must be on the classpath. This will ensure that when Janus attempts to communicate with HBase it knows that Kerberos is in play and does the necessary things automatically for authentication.

2. Janus makes use of the SOLRJ library, and that library must also be made aware that Kerberos is in play. Hence this PR. The changes in this PR configure the solrj clients (both standalone and cloud) to make use of a JAAS configuration file to handle the necessary authentication.

   The JAAS configuration file needs one section called: Client. In Solr Cloud mode this configuration pulls double duty as it handles authentication for both internode requests and requests to ZooKeeper (Since Solr Cloud Mode uses Zookeeper).

Your Question

Even if SOLR and HBase share the same Zookeeper there should be no conflicts with using them together with Janus. As a user I can authenticate as a single principal by using a Keytab or TGT and then start using Janus which will in turn interact with HBase, Solr, Zookeeper, etc.

Sorry for the long winded answer but I thought the background and context would be helpful. Please let me know if you have any other questions or concerns.

Related Reading

https://lucene.apache.org/solr/guide/7_2/kerberos-authentication-plugin.html

[jerryjch]

Thanks for the very good explanation!
Just to be clear. Suppose Zookeeper server is a secure server. HBase's Zookeeper client will share the the same JAAS configuration as the one introduced by this PR, and share the same 'client' section?

[jackson-chris]

I will answer your question in two parts.

1. The HBase daemons (Master, and RegionServer) will NOT share this configuration. They will be configured to use their own JAAS configuration files which will dictate authentication configuration with Zookeeper.

2. It is not immediately clear to me if the HBase Client API (used by Janus Graph) will share the configuration. Once we call System.setProperty("java.security.auth.login.config", kerberosConfig); that property will persist for the lifecycle of that particular JVM or until that property is overwritten. However, I don't know that the HBase Client code uses that property to configure auth settings to interact with Zookeeper.

   I took a look at the HBase documentation and from this section I took note of:

   Before 2.2.0 version, the client environment must be logged in to Kerberos from KDC or keytab via the kinit command before communication with the HBase cluster will be possible.

   and this section I took note of:

   In cases where ZooKeeper is used for service discovery or sharing state with the client, the znodes created by HBase will also allow anyone (regardless of authentication) to read these znodes (clusterId, master address, meta location, etc), but only the HBase user can modify them.

   Based upon the information in the docs I think that the HBase Client code will also not share this configuration.

Further Notes:

The JAAS configuration file I have included in my PR is just for testing purposes and used for the unit tests and isn't shipped with the product.

This PR expects the end user to add additional configuration in the janusgraph.properties file that specifies kerberos is being used and the location of the JAAS configuration file we want to use for authentication.

[jackson-chris]

Any more feedback @jerryjch?

[jerryjch]

@blindenvy I will get back to it. Sorry about the delay,

[pluradj]

@blindenvy I think it would be worthwhile to add some pointers in the Solr docs to some of the Kerberos resources you mentioned in the discussion above. Although we don't need to completely duplicate those docs, making it easier for other JanusGraph users to find them would be helpful, especially with respect to the HBase, ZooKeeper, and Solr interplay.

I'd suggest that you add the docs to the solr.adoc file.

[pluradj]

I'm not sure how valuable this parameter is if it's configured as GLOBAL_OFFLINE. Do users typically toggle Kerberos on and off? If not, it seems like you could just use KERBEROS_CONFIG as the only config parameter for whether to use Kerberos or not.

[jackson-chris]

Do users typically toggle Kerberos on and off?

No I would say that is not a typical use case.

[pluradj]

If it makes sense to support changing the KERBEROS_ENABLED setting per connection (for example, if Solr or HBase support it also), then you should declare it as ConfigOption.Type.MASKABLE rather than ConfigOption.Type.GLOBAL_OFFLINE.

[jackson-chris]

After discussion I will make the change to MASKABLE.

[pluradj]

LGTM. @blindenvy thanks for working on this.

[jerryjch]

@blindenvy I am still a little worried about setting the java.security.auth.login.config property for the whole JVM programmably for Solr only that may overwrite setting that may come from other places (e.g in the java command line from gremlin server) and/or for other clients.
Can we do a check in the new code to see if the system property java.security.auth.login.config is set or not, and if it is, give a warning but don't overwrite it?

[jerryjch]

HBase secure config with Zookeeper is here.

[jackson-chris]

@jerryjch @pluradj I agree with both your assessments. There is some concern with overriding a prior set configuration, also its typical for jaas config to be set via command line through jvm options instead of programatically.

Thus, I have removed the jaas config option from janusgraph properties. Instead System.getProperty("java.security.auth.login.config") will be used to pick up the appropriate jaas config.

This will require that end users add this JVM option when invoking janus graph when a kerberized solr instance is in play.

For instance to run gremlin.sh you would need to do:

export JAVA_OPTIONS="-Djava.security.auth.login.config=/absolute/path/jaas.conf"
$JANUSGRAPH_HOME/bin/gremlin.sh

[jerryjch]

This approach of setting the property from the command line looks good.