Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

METRON-1844: Allow for LDAP to be used for authentication and roles #1246

Closed
wants to merge 14 commits into from

Conversation

Projects
None yet
4 participants
@justinleet
Copy link
Contributor

commented Oct 25, 2018

Contributor Comments

I went through the defunct Knox SSO feature branch and pulled out the LDAP bits. I made sure @simonellistonball is on the first commit to maintain attribution. When we merge this, we'll need to make sure that's properly maintained.

What this does is setup the set of LDAP configs, while maintaining backwards compatibility with the JDBC stuff by default.

I'd say the main objection I'd expect would be around unit/integration testing. Right now there's not anything in this PR, although we could probably pull in the EmbeddedLDAP from the branch and make sure the role endpoint works with it for integration.

I have run this up on full dev and tested against an actual OpenLDAP instance using https://github.com/osixia/docker-openldap. Instructions will be provided in comment to duplicate this testing.

Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:

For all changes:

  • Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
  • Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
  • Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

  • Have you included steps to reproduce the behavior or problem that is being changed or addressed?

  • Have you included steps or a guide to how the change may be verified and tested manually?

  • Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:

    mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh 
    
  • Have you written or updated unit tests and or integration tests to verify your changes?

  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

  • Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:

    cd site-book
    mvn site
    

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

@justinleet

This comment has been minimized.

Copy link
Contributor Author

commented Oct 25, 2018

To run this up against an OpenLDAP instance, make sure Docker is installed and running, we'll be using https://github.com/osixia/docker-openldap.

OpenLDAP setup

Setup Users

There is an LDIF file we'll load into this container at https://gist.github.com/justinleet/b8f0350f27b32d4d705f45b9c1ba8c6e

It'll set up an admin user (admin), a regular user (sam), and a user without a role (tom).

We'll want to save that file somewhere. I'll be using /tmp/ldif/test.ldif in this example.

Run the container.

Run the container with

docker run \
  -p 33389:389 \
  --name metron_ldap_demo \
  --env LDAP_ORGANISATION=Hadoop \
  --env LDAP_DOMAIN=apache.org \
  --env LDAP_TLS_FALSE \
  --volume /tmp/ldif:/container/service/slapd/assets/config/bootstrap/ldif/custom \
  osixia/openldap:1.2.2 --copy-service

This does a few things

  • Exposes the LDAP port (389) on the host's port 33389. The host port is arbitrarily chosen
  • Makes sure we have the appropriate org and domain for the users
  • Makes sure we aren't using TLS. There are options for in the configs, but I wasn't able to get the certs lined up properly (although I was able to see both sides connecting, then failing).
  • Makes the test.ldif available in the right place
  • --copy-service needs to be provided for some reason or another. I didn't really dig into why, it's just part of the documentation

This will spin up the OpenLDAP instance and let it run.

Ambari LDAP setup

From the command line of the vagrant machine, run
netstat -rn | grep "^0.0.0.0 " | cut -d " " -f10. It should return the address it can use to address the host machine. E.g. for me it was "10.0.2.2". This is needed to be able to hit the Docker container from inside the Vagrant VM.

A new tab is added "Security" with the various LDAP related properties.
In here, we'll went to set

  • LDAP URL = ldap://10.0.2.2:33389
    • Make sure to fill in the IP from above!
  • Bind User = cn=admin,dc=apache,dc=org
  • Bind User Password = admin

The other fields are set already set to defaults to match the modified Knox users-ldif provided, so nothing should change.
These values are

  • User dn pattern = uid={0},ou=people,dc=hadoop,dc=apache,dc=org
  • User password attribute = userPassword
  • Group Search Base = ou=people,dc=hadoop,dc=apache,dc=org
  • User Search Filter is empty
  • Group Search Base = ou=groups,dc=hadoop,dc=apache,dc=org
  • Group Search Filter = member={0}
  • LDAP group role attribute = cn

Additionally, we also need to tell REST we will be using LDAP.

In the REST tab

  • Active Spring profiles = dev,ldap

Restart Metron REST

Testing LDAP

Go to the Swagger UI. You should meet a login screen. There are 3 users that should work in LDAP

  • admin/admin-password
    • {SSHA} format in LDAP
  • sam/sam-password
    • Plaintext format in LDAP
  • tom/tom-password
    • Plaintext format in LDAP

In Swagger, the UserController has another endpoint: "/whoami/roles". Using this endpoint should return the roles for each user

  • admin has ROLE_ADMIN
  • sam has ROLE_USER
  • tom has no roles.

These ROLES can be altered by changing the LDAP groups each user belongs to, e.g. see https://gist.github.com/justinleet/b8f0350f27b32d4d705f45b9c1ba8c6e#file-test-ldif-L95 for how "sam" is added to the "user" group.

This same login should also apply to the Alerts UI and Management UI. Login via JDBC should also not be able to happen (e.g. user/password login should no longer work).

SSL Note

If anyone is brave enough to try, or wants to help me, you can run the container with

docker run \
  -p 33389:389 \
  -p 33636:636 \
  --name metron_ldap_demo \
  --env LDAP_ORGANISATION=Hadoop \
  --env LDAP_DOMAIN=apache.org \
  --env LDAP_TLS_VERIFY_CLIENT=try \
  --volume /tmp/ldif:/container/service/slapd/assets/config/bootstrap/ldif/custom \
  osixia/openldap:1.2.2 --copy-service

This will setup the appropriate SSL port and map it to 33636. LDAP_TLS_VERIFY_CLIENT=try seems necessary to be able to connect from outside the container.

In the Vagrant box, it's necessary to import the appropriate certificates to the truststore to be used. Generally this can be done with

keytool -importcert -keystore <keystore file> -alias <unique alias> -storepass <password> -file <certificate file>

Additionally, in the configs, there are settings for "LDAP Truststore" and "LDAP Truststore Password". These should be set to match the keystore you imported certificates to.

I attempted to pull the certs out of the container itself, as well as to use

echo -n | openssl s_client -showcerts -connect 10.0.2.2:33636 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' >> ldapserver.pem

in order to get certs. Neither seemed to make SSL happy, I can only assume my sacrificial offerings have been lacking lately.

The log error in Vagrant:

Warning: no suitable certificate found - continuing without client authentication
*** Certificate chain

while the container gives:

5bd1d66c conn=1011 fd=13 ACCEPT from IP=172.17.0.1:35946 (IP=0.0.0.0:636)
TLS: can't accept: No certificate was found..
5bd1d66c conn=1011 fd=13 closed (TLS negotiation failure)

Given that this is a negotiation failure, rather than an inability to make it to the handshake, I'm kind of inclined to not troubleshoot it too much here.

@justinleet justinleet force-pushed the justinleet:ldap_fixed_author branch from ee6e6d9 to 52f76b7 Oct 26, 2018

@justinleet

This comment has been minimized.

Copy link
Contributor Author

commented Oct 29, 2018

I updated the Ambari config to use a toggle On/Off for the LDAP functionality. The adjustment to the spring profiles takes place in the background, so the user doesn't have to worry about it.

metron_ldap_userdn = config['configurations']['metron-security-env']['metron.ldap.bind.dn']
metron_ldap_password = config['configurations']['metron-security-env']['metron.ldap.bind.password']
metron_ldap_user_pattern = config['configurations']['metron-security-env']['metron.ldap.user.dnpattern']
metron_ldap_user_password = config['configurationsmetron_spring_profiles_active']['metron-security-env']['metron.ldap.user.password']

This comment has been minimized.

Copy link
@anandsubbu

anandsubbu Oct 30, 2018

Contributor

Looks like a mispaste. metron_spring_profiles_active should be removed.

This comment has been minimized.

Copy link
@justinleet

justinleet Oct 30, 2018

Author Contributor

Good catch. Fixed in latest commit

@justinleet

This comment has been minimized.

Copy link
Contributor Author

commented Nov 1, 2018

Updated the instructions above with corrected LDAP URL

<value-attributes>
<empty-value-valid>true</empty-value-valid>
</value-attributes>
<value>jdbc</value>

This comment has been minimized.

Copy link
@merrimanr

merrimanr Nov 1, 2018

Contributor

Are we using a "jdbc" profile anywhere? From what I can tell jdbc authentication is used by default but we don't actually use a profile for that.

This comment has been minimized.

Copy link
@justinleet

justinleet Nov 1, 2018

Author Contributor

good catch. I thought I'd rolled that back. I'll udpate

@merrimanr

This comment has been minimized.

Copy link
Contributor

commented Nov 1, 2018

I tested this in full dev using the instructions in the README and with the Knox sample ldap. Both worked fine. I also tested using legacy JDBC authentication and it continued to work. Nice job! +1

@anandsubbu

This comment has been minimized.

Copy link
Contributor

commented Nov 2, 2018

My +1 as well.

I spun up a multi-node cluster and set this up using the demo LDAP service provided by Knox. I was able to authenticate using the demo users for all the Swagger, Alerts and Management UI interfaces. Nicely done, @justinleet !

@asfgit asfgit closed this in d0411f6 Nov 2, 2018

asfgit pushed a commit that referenced this pull request Nov 2, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.