[Auth0] Update Auth0 Integration to use Management API for Log Ingestion

[terrancedejesus]

Summary

The Auth0 integration seems to use an outdated approach to retrieving and ingesting event logs. Rather the management API can be used to retrieve these logs instead as done by companies like DataDog and Splunk.

We should look to update this integration to use these same APIs, making this integration setup more feasible. TRADE is looking to add a prebuilt threat detection ruleset for this data, however, have ran into several hurdles and blockers with setting up this integration that has halted progress.

Please let us know if there is anything we can help with.

cc @andrewkroh @jamiehynds

References:

• [Meta] Add Auth0 Prebuilt Threat Detection Ruleset detection-rules#3780

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[andrewkroh]

seems to use an outdated approach

@terrancedejesus, Can you please clarify what is outdated about auth0 custom webhooks?

[terrancedejesus]

@andrewkroh - Thank you for the reply! If I understand correctly, this integration was built to rely on an Auth0 stream that pushes logs via HTTP POST requests to a custom web server that stores locally. Then the Elastic Agent reads these logs and ingests them into the stack, is that correct? My assumption is that at the time this integration was built (~2 years ago) the management API may have not been available.

If so, this issue was opened to adjust how logs are retrieved, using the management API where logs can be pulled by the agent and then ingested as typically done with SaaS integrations. From the small blurb Retrieve logs by checkpoint, this appears to be how DataDog and Splunk ingest Auth0 logs as well.

[cpascale43]

Hi @terrancedejesus! Looking to understand this one a bit more, are the events exposed via the management API the same as the ones we currently collect?

[terrancedejesus]

@cpascale43 hello! I am not sure but worth the investigation. Their notes suggest that this is how Splunk and DataDog ingest logs for monitoring.

Digging a bit, I see that the DataDog integration sets up the Auth0 log stream to point directly to the DataDog API which requires an API key to their instance. If management API does not seem viable, then we could explore this route and contact Auth0 to add an OOTB Elastic stream option, which is missing.

At the moment, it seems that we can update the integration or at least documentation to setup this integration with Elastic. At the moment, setting up a custom public web server just to accept POST requests and store logs for the agent to push to the stack seems like it could be updated.

The agent running this integration must be able to accept requests from the Internet in order for Auth0 to be able connect. Auth0 requires that the webhook accept requests over HTTPS. So you must either configure the integration with a valid TLS certificate or use a reverse proxy in front of the integration.

I could also be misunderstanding the steps to set this up if someone is willing to help guide me through setup.

[jamiehynds]

@narph to simplify the integration, a move to a typical API based integration makes sense for Auth0. The Management API didn't exist when we built the integration, hence the unusual workflow in place currently.

The endpoint to hit is: https://auth0.com/docs/deploy-monitor/logs/retrieve-log-events-using-mgmt-api
An Auth0 account can be created here.

Ok to remove tech clarification and add to the next iteration?

[andrewkroh]

I think we need to keep the existing http_endpoint input otherwise we risk breaking some users. The push-based "Custom Webhook" delivery is still fully supported by auth0. Because of the complexity in setting up webhook delivery to Elastic Agent, I suggest we de-emphasize this mechanism and steer users towards the API collection approach.

[andrewkroh]

This is what I think needs done.

1. Add a new input to the existing data stream to pull data from the API.
2. Disable all the inputs by default (see [Integrations] Data streams disabled by default #6104). Make the user choose one. This will make upgrades go smoothly after the new input is added.

Assuming that the data format from the API is the same as the webhook, then there should not be much ingest pipeline work.