[filebeat][threatintel] MISP pagination fixes

[chrisberkhout]

Update the Filebeat Threat Intel module's misp fileset with fixes from the MISP integration.

Proposed commit message

[filebeat][threatintel] MISP pagination fixes (#)

Update the HTTP JSON input configuration for the Threat Intel module's
misp fileset with pagination fixes that were done earlier in the
Agent-based MISP integration, in these PRs:

- Fix timestamp format sent to API
  https://github.com/elastic/integrations/pull/6482

- Fix duplicate requests for page 1
  https://github.com/elastic/integrations/pull/6495

- Keep the same timestamp for later pages
  https://github.com/elastic/integrations/pull/6649

- Pagination fixes
  https://github.com/elastic/integrations/pull/9073

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

To run existing tests for the Threat Intel module:

cd x-pack/filebeat
mage pythonIntegTest

For a guide to running against a local instance of MISP, please refer to the "How to test this PR locally" section of elastic/integrations#9073.

Related issues

This issue has discussion of these bugs (and links to a user report specific to the Threat Intel module):

• Relates [ti_misp] Wrong timestamp cause no events fetched from MISP integrations#8554

PRs with the corresponding changes in the Agent-based integration:

• Relates ti_misp: fix timestamp format sent to API integrations#6482
• Relates ti_misp - Fix duplicate requests for page 1 integrations#6495
• Relates [ti_misp] Keep the same timestamp for later pages integrations#6649
• Relates [ti_misp] Pagination fixes integrations#9073

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @chrisberkhout? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Duration: 135 min 23 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 51b4f5c

Failed CI Steps

• :ubuntu: Unit Tests

History

• 💔 Build #2657 failed e65debe

cc @chrisberkhout

[efd6]

The buildkite failure looks to be unrelated. Do you know the cause?

[chrisberkhout]

The buildkite failure looks to be unrelated. Do you know the cause?

Seems to be a problem with build scripting for generating pipelines. I guess these aren't run for all builds. I tried to run it locally but was missing some buildkite agent dependency and stopped there.