feat(ti_misp): add MISP API rate-limit response headers

[navnit-elastic]

Proposed commit message

feat(ti_misp): add MISP API rate-limit headers to httpjson

- Add MISP API rate-limit response headers (X-Rate-Limit-*) to httpjson.
- Remove the unused http_request_rate_limit option. It did not enable
  httpjson rate limiting without request.rate_limit.remaining; behavior
  is superseded by MISP response headers.

• https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-httpjson#request-rate-limit

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Reference taken from: #9073

The following is a guide for how to run MISP locally, observe API rate limit behaviour and run the Elastic MISP integration against that local instance.

MISP: Configure, run and generate an API token

• Clone https://github.com/misp/misp-docker.
• Follow the misp-docker README's Getting Started instructions.
• In MISP, Administration > List Auth Keys, add a new key and copy its value.

MISP: Enforce Rate Limit

• In MISP, Administration > List Roles, locate admin role and click on edit icon.
• Enable Enforce search rate limit option and configure rate limit.

MISP: Load data and query

• In MISP, Sync Actions > Feeds, click "Load default feed metadata", enable some feeds and click "Fetch all events" on them.
• In MISP, API > Rest client, you can construct and run queries. It can generate cURL commands (to which you may need to add the --insecure option).

MISP integration in Elastic

When adding a MISP integration policy use the following settings:

• MISP URL: https://172.17.0.1 (special IP for the docker host)
• MISP API Token: <your-api-token> (from earlier setup)
• Initial Interval: 0s (or more)
• Interval: 10s
• Preserve original event: yes
• SSL: "verification_mode: none"
• Enable request tracing: yes

Related issues

• Closes [Integration Name]: Respect Rate Limit for openMISP integration #18268

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[kcreddy]

Instead of hardcoding the API token, it would be slightly cleaner to replace it with a placeholder like <your-api-token> since the instructions already say "copy its value" from the earlier step.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 09096ba

History

• 💚 Build #41649 succeeded 5c876f3
• 💚 Build #41599 succeeded 2459002

cc @navnit-elastic

[kcreddy]

LGTM

[elastic-vault-github-plugin-prod]

Package ti_misp - 1.42.0 containing this change is available at https://epr.elastic.co/package/ti_misp/1.42.0/