x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher: add device handling #35807

efd6 · 2023-06-19T01:35:13Z

What does this PR do?

Adds device support to the Azure AD entity analytics input provider.

Why is it important?

Milestone task needed for security solution.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine · 2023-06-19T02:16:35Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-06-21T21:27:18.235+0000
Duration: 71 min 49 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3048
Skipped	178
Total	3226

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-06-19T03:21:23Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

bhapas

Looks Good.

Just a thought though : There are instances where a lot of code is duplicated between users groups and devices. Not sure if we can think of a solution that helps to add more entities in future in a simple way?

x-pack/filebeat/input/entityanalytics/provider/azuread/azure.go

x-pack/filebeat/input/entityanalytics/provider/azuread/azure_test.go

bhapas · 2023-06-20T09:07:14Z

x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher/device.go

+	"github.com/elastic/elastic-agent-libs/mapstr"
+)
+
+// TODO: Implement fetchers for the registered owners and users


A github issue link to the TODO ?

x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher/device.go

x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher/group.go

mergify · 2023-06-20T10:16:24Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b st5940-azure-device upstream/st5940-azure-device
git merge upstream/main
git push upstream st5940-azure-device

x-pack/filebeat/input/entityanalytics/provider/azuread/azure.go

x-pack/filebeat/docs/inputs/input-entity-analytics.asciidoc

x-pack/filebeat/input/entityanalytics/provider/azuread/azure.go

taylor-swanson · 2023-06-20T20:01:13Z

x-pack/filebeat/input/entityanalytics/provider/azuread/azure.go

@@ -165,6 +165,21 @@ func (p *azure) runFullSync(inputCtx v2.Context, store *kvstore.Store, client be
 		tracker.Wait()
 	}

+	if len(state.devices) != 0 {
+		tracker := kvstore.NewTxTracker(ctx)
+


I'm wondering, do we want a single set of writer markers to bound both users and devices? If we want them separate, do we want to attach any sort of field to indicate what the write markers are associated with? (device or user).

Thoughts, @andrewkroh?

Good point, either is acceptable to me. If it is two separate marks then someone who is querying would need to be know if it was associated to the users or devices.

ISTM a single set makes more sense. devices are users are really pretty much the same kind of entity (at the moment more so, with the TODO additions less so).

x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher/device.go

taylor-swanson · 2023-06-20T20:09:05Z

x-pack/filebeat/input/entityanalytics/provider/azuread/statestore.go

+
+		return nil
+	}); err != nil && !errIsItemNotFound(err) {
+		return nil, fmt.Errorf("unable to get users from state: %w", err)


s/users/devices

mergify · 2023-06-21T16:54:35Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b st5940-azure-device upstream/st5940-azure-device
git merge upstream/main
git push upstream st5940-azure-device

…evice handling

Wrap update and sync publication of users and devices together into single tracker/write marker operations.

…evice handling (#35807) (cherry picked from commit a98c576)

…evice handling (#35807) (#35878) (cherry picked from commit a98c576) Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

efd6 added enhancement Filebeat Filebeat Team:Security-External Integrations backport-skip Skip notification from the automated backport with mergify 8.9-candidate labels Jun 19, 2023

efd6 self-assigned this Jun 19, 2023

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 19, 2023

efd6 force-pushed the st5940-azure-device branch 2 times, most recently from 71e0e07 to 9d1adf7 Compare June 19, 2023 02:10

efd6 marked this pull request as ready for review June 19, 2023 03:21

efd6 requested a review from a team as a code owner June 19, 2023 03:21

bhapas reviewed Jun 20, 2023

View reviewed changes

andrewkroh reviewed Jun 20, 2023

View reviewed changes

x-pack/filebeat/input/entityanalytics/provider/azuread/azure.go Outdated Show resolved Hide resolved

taylor-swanson reviewed Jun 20, 2023

View reviewed changes

efd6 force-pushed the st5940-azure-device branch from 9d1adf7 to 267f68a Compare June 21, 2023 02:18

andrewkroh approved these changes Jun 21, 2023

View reviewed changes

taylor-swanson approved these changes Jun 21, 2023

View reviewed changes

efd6 added 4 commits June 22, 2023 06:56

x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher: add d…

a9f2c7d

…evice handling

address pr comments

9737957

address pr comment

2d21dd8

address pr comment

6f4faa5

Wrap update and sync publication of users and devices together into single tracker/write marker operations.

efd6 force-pushed the st5940-azure-device branch from bda997c to 6f4faa5 Compare June 21, 2023 21:26

andrewkroh approved these changes Jun 21, 2023

View reviewed changes

efd6 added backport-v8.9.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Jun 21, 2023

efd6 merged commit a98c576 into elastic:main Jun 21, 2023
21 checks passed

mergify bot pushed a commit that referenced this pull request Jun 21, 2023

x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher: add d…

53fd045

…evice handling (#35807) (cherry picked from commit a98c576)

mergify bot mentioned this pull request Jun 21, 2023

[8.9](backport #35807) x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher: add device handling #35878

Merged

efd6 added a commit that referenced this pull request Jun 22, 2023

x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher: add d…

1245830

…evice handling (#35807) (#35878) (cherry picked from commit a98c576) Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher: add device handling #35807

x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher: add device handling #35807

efd6 commented Jun 19, 2023

elasticmachine commented Jun 19, 2023 •

edited by jenkins-beats-ci bot

Build stats

Test stats 🧪

elasticmachine commented Jun 19, 2023

bhapas left a comment

bhapas Jun 20, 2023

mergify bot commented Jun 20, 2023

taylor-swanson Jun 20, 2023

andrewkroh Jun 21, 2023

efd6 Jun 21, 2023

taylor-swanson Jun 20, 2023

mergify bot commented Jun 21, 2023

x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher: add device handling #35807

x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher: add device handling #35807

Conversation

efd6 commented Jun 19, 2023

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Jun 19, 2023 • edited by jenkins-beats-ci bot

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Jun 19, 2023

bhapas left a comment

Choose a reason for hiding this comment

bhapas Jun 20, 2023

Choose a reason for hiding this comment

mergify bot commented Jun 20, 2023

taylor-swanson Jun 20, 2023

Choose a reason for hiding this comment

andrewkroh Jun 21, 2023

Choose a reason for hiding this comment

efd6 Jun 21, 2023

Choose a reason for hiding this comment

taylor-swanson Jun 20, 2023

Choose a reason for hiding this comment

mergify bot commented Jun 21, 2023

elasticmachine commented Jun 19, 2023 •

edited by jenkins-beats-ci bot