Add Lumos Package

[ethanzh]

Proposed commit message

This PR introduces a package for Lumos (lumos.com). The Lumos product itself has an Activity Log, and we would like customers to be able to ship their Activity Logs directly to their Elastic instance.

In order to support this, we added an /activity_logs endpoint to our API, which returns data in this form:

{
    "items":
        [{
            "actor": {
                "actor_type": "Lumos"
            },
            "event_began_at": "2023-12-29T19:26:16",
            "event_hash": "b1205b5116d3d798035a359dea4313382d6409ff02350e4600bce2e161c1c2bbe91a05de8c3ed7c3b1efce9751d9548523dc2fbca67191ca1a060976581c5215",
            "event_metadata": {},
            "event_type": "SEND_REQUEST_APPROVAL_FOR_ACCESS_REQUEST_SLACK",
            "event_type_user_friendly": "Lumos attempted to send a Slack notification to this approver requesting approval for an access request",
            "outcome": "Succeeded",
            "targets": [
                {
                    "access_length": "Unlimited",
                    "app": {
                        "app_id": "salesforce.com",
                        "instance_id": "00D5f000005vnLlEAI",
                        "user_friendly_label": "Salesforce - Dev"
                    },
                    "business_justification": "read only test with new permission for flows on staging",
                    "permissions": [
                        {
                            "label": "Read Only [Profile]",
                            "source": "MANUAL",
                            "type": "PERMISSION",
                            "value": "Read Only [Profile]"
                        }
                    ],
                    "requester_user": {
                        "email": "cypress@lumostester.com",
                        "family_name": "Tests",
                        "given_name": "Cypress"
                    },
                    "target_type": "Access Request",
                    "target_user": {
                        "email": "bellatrix.lestrange@lumostester.com",
                        "family_name": "Lestrange",
                        "given_name": "Bellatrix"
                    }
                },
                {
                    "email": "albus@lumostester.com",
                    "family_name": "Dumbledore",
                    "given_name": "Albus",
                    "target_type": "User"
                }
            ]
        }
    ],
    "limit": 50,
    "links": {
        "first": "/activity_logs?since=2023-12-06T15%3A39%3A42Z&offset=0",
        "last": "/activity_logs?since=2023-12-06T15%3A39%3A42Z&offset=400",
        "next": "/activity_logs?since=2023-12-06T15%3A39%3A42Z&offset=50",
        "prev": null,
        "self": "/activity_logs?since=2023-12-06T15%3A39%3A42Z"
    },
    "offset": 0,
    "total": 446
}

Essentially, we support filtering responses by ISO8601-encoded strings until and since. And we also return a next link in the response in order to simplify the pagination logic on the Elastic side.

Because this is a 0.1.0, we do not do anything fancy with the log once we receive it, we simply log it in the json column. A next step will be to extract out common fields from the JSON response (e.g. actor.email).

Our API is authenticated through a bearer token that the user provides.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Screenshots

[ethanzh]

/test

[ethanzh]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[ethanzh]

Hi @jamiehynds -- thanks for adding the tags. Please let me know if there is any other information I can provide here to make it easier on you all's end :)

[ethanzh]

Really appreciate the review @efd6 I'll address comments and turn this around as quickly as I can

[ethanzh]

Hey @efd6, thank you again for the previous review.

I've incorporated all of your feedback and re-pushed the branch. The biggest change here is the addition of a proper system test, similar to the 1Password implementation you linked me to.

Please let me know what else, if anything, needs to be improved here. Thank you!

[efd6]

/test

[efd6]

Can you include some pipeline test cases so that we can build on the logic for event classification as it is added?

[ethanzh]

/test

[ethanzh]

@efd6 Thank you again, you've been super helpful. Since your last review I've done the following:

• Add processor logic in integest_pipeline/default.yml to set event.outcome
• Add processor in default.yml to set event.kind=event.
• Add processor in default.yml to set event.type=info. In a future iteration I will refine event.type to be more granular, as well as correctly setting event.category by creating a mapping from event.action -> event.category
• Fixed-up the whitespaces. Apologies for that, can't seem to get my editor to respect the 2 space by default
• Rename json to lumos_activity to remove the need to manually rename many columns
• README.md formatting
• Newline in pipeline test file

[efd6]

/test

[ethanzh]

Shoot, I'll double-check that the README files are in-sync. Looks like that's what's failing CI

[ethanzh]

Just re-ran the build, apologies for that oversight

[efd6]

/test

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[efd6]

Last nit, then LGTM

[ethanzh]

Made that one final tweak

[efd6]

/test

[efd6]

Suggested change

😄 Too many.

[ethanzh]

Hah, third time is the charm!

[efd6]

/test

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 902795f

History

• 💚 Build #9634 succeeded b05abec
• 💚 Build #9632 succeeded b5f5d40
• 💔 Build #9631 failed 88ac9b5
• 💚 Build #9600 succeeded 29c7a96

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
81.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[efd6]

Thanks

[ethanzh]

Hey @efd6 one more question. I'm seeing this build failure on the main branch -- is this a result of something I did incorrectly?

[efd6]

No, there is nothing wrong in what was done here. I will look into what is going on.

[elasticmachine]

Package lumos - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=lumos

[ethanzh]

Hi @efd6, I've noticed that https://epr.elastic.co/search?package=lumos is still blank. Let me know if there's something I can do on my end to help debug/remediate. Thank you

[ethanzh]

Additionally, should we expect Lumos to be showing up in this catalog: https://www.elastic.co/integrations/data-integrations?

[efd6]

The package is not GA, so it won't show up without the experimental flag, https://epr.elastic.co/search?package=lumos&experimental=true