entityanalytics_ad: new package for Active Directory user collection

[efd6]

Proposed commit message

See title.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

System testing requires an external Active Directory server to collect the user details from. Set this up or get access to an existing server.

This testing depends on 8.14.0-SHAPSHOT including f871b5c5f76b45c5724599d49bd0b0909959783d. The current DRA does not. It is possible to get around this by bringing up the stack as normal with elastic-package and inserting a copy of the relevant filebeat into the docker agent filesystem. First start an 8.13.0 stack.

In the x-pack/filebeat directory:

PACKAGES=tar.gz PLATFORMS=linux/amd64 DEV=true SNAPSHOT=true mage -v package
tar zxvf build/distributions/filebeat-8.14.0-SNAPSHOT-linux-x86_64.tar.gz filebeat-8.14.0-SNAPSHOT-linux-x86_64/filebeat
elastic-package stack up -v -d --version=8.13.0
docker cp filebeat-8.14.0-SNAPSHOT-linux-x86_64/filebeat <container ID>:/usr/share/elastic-agent/data/elastic-agent-1eb18c/components

Then in the integrations repo in the entityanalytics_ad package directory:

AD_ADDR=<ldap host url> # e.g. ldap://localhost
AD_USER=<ldap user> # e.g. CN=Administrator,CN=Users,DC=testserver,DC=local
AD_PASS=<password>
AD_BASE_DN=<base dn> # e.g. CN=Users,DC=testserver,DC=local
sed -i -r "\
    s|ldapurl|${AD_ADDR}|; \
    s/ldapbasedn/${AD_BASE_DN}/; \
    s/ldapuser/${AD_USER}/; \
    s/ldappassword/${AD_PASS}/; \
" data_stream/user/_dev/test/system/test-default-config.yml
elastic-package stack up -v -d --version=8.14.0-SNAPSHOT # See note above about stack version; do not run this if the filebeat build was required.
eval "$(elastic-package stack shellinit)"
elastic-package test system

(assumes GNU sed; if on another OS, install and use gsed)

Related issues

• Closes [Entity Analytics] Active Directory #8559

Screenshots

[jaredburgettelastic]

Unfortunately, I am unable to run an active directory server on my M2 mac. Only testbuild versions of VirtualBox are available for ARM-based chips, and when I try to boot up a Windows 2022 ISO on that testbuild VirtualBox, it runs into a critical error.
So with that said, right now I can only provide feedback based on the information in the PR and the active directory specs. Hope some of this is helpful:

• In the ingest pipeline, I see a single field being mapped to user.account.* , specifically password_change_date . Is there any reason for only that field? I see a couple others that seem like they would map directly to the "exported fields", such as user.account.change_date and user.account.create_date mapping to when_changed and when_created, respectively. I also don't see user.account.* defined in ECS today, is this something that is planned to be added to ECS? https://www.elastic.co/guide/en/ecs/8.13/ecs-user.html#ecs-user-nestings
• Each entityanalytics_ad record has a groups field and a user field. My assumption is that they correspond to one another (i.e., "this user belongs to these groups"). If that is true, we should be able to provide a mapping to the ECS user.group.* field. Here is where ECS defines that nested mapping for user.group.* , and here is where ECS defines what groups can contain.
• Lastly, I don't see anything regarding email addresses being mapped, but it's also not in any of the sample data. It would be great if I could have gotten an AD server set up to see if it indeed allows for/provides email data, but at the very least I see a few attributes in the AD spec that should corresponding to it (here and here). If we have this data, it would be mapped to user.email .

[efd6]

@jaredburgettelastic Thanks.

The field use here reflect the use in the Okta EA package. The fields that are mapped are the ones that were available for me to identify absent any documentation for the expected fields in the AD LDAP schema.

From what you've raised, given the use in the Okta package, I think that this is likely OK to merge from the schema perspective with the anticipation that additional fields will be identified in future that will direct further ECS mappings being added. Does that sound reasonable to you?

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[jaredburgettelastic]

@efd6 Sure thing, we can take an iterative approach. Thank you!

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: a711a23

History

• 💚 Build #10403 succeeded 2deeca0
• 💔 Build #10005 failed 092b4e3
• 💔 Build #10002 failed fc89b65

cc @efd6

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
97.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[chemamartinez]

LGTM

[elasticmachine]

Package entityanalytics_ad - 0.0.1 containing this change is available at https://epr.elastic.co/search?package=entityanalytics_ad

[mbudge]

Is this integration available yet?

I can't see it on the available integrations.

Thanks

[jamiehynds]

@mbudge yep, it's available. 8.14 is the minimum required Kibana version though, so won't appear until you're running 8.14.

Also check to make sure the 'Display beta integrations' toggle is enabled (under the categories in Fleet). That often catches folks out when integrations are in beta/tech preview.

[Gelios7]

When testing the work of the Entity_AD integration, I found that the settings of the integration itself are limited to the directory "CN=Users" at the program level. For example, most of our users do not find "CN=Users" in the directory, and in the settings I cannot set a user path that does not include "CN=Users" for "Active Directory User", because I get an error in the logs, too refers to the "Active Directory Base DN" setting, it is not possible to specify the root "Base DN" to read the assets of all users in Active Directory.
For example, our users are on different paths:
CN=test_user,OU=technical_account,OU=Admins,DC=test,DC=doman,DC=int
That is, you need to be able to specify the "Active Directory Base DN" or the root path: DC=test,DC=doman,DC=int or specify several paths where users are located:
OU=technical_account,OU=Admins,DC=test,DC=doman,DC=int
OU=Administrator_users ,OU=Admins,DC=test,DC=doman,DC=int
OU=Main_Office_Users ,DC=test,DC=domain,DC=int
The first option, in my opinion, will be correct, because the connector will see all users in the domain.
This also applies to the "Active Directory User" setting, it is necessary to be able to specify the path to any user, not only those included in the path "CN=Users", for example:
CN=test_user,OU=technical_account,OU=Admins,DC=test,DC=doman,DC=int

If this is not done, the integration will be inferior to use and will have limited functionality.
I can assume that the same situation can be for "device" assets.