[ti_misp] Pagination fixes

[chrisberkhout]

Proposed commit message

[ti_misp] Pagination fixes (#9073)

This change fixes several bugs in pagination logic.

For the threat (events) and threat_attributes (attributes) data streams:

* Access the timestamp correctly in the cursor data, as a string not a
  `time.Time` value.
* Reuse the initial timestamp value until a new one comes from the
  server (rather resetting it with `initial_interval` until a
  timestamp appears in the cursor data).
* Add `"order":"timestamp"` parameter to requests, so that MISP queries
  the database with an explicit ORDER BY clause.
* Adjust the system test stubs so that the tests cover several page
  sequences, require correct timestamp values after the first sequence,
  and match with the new `order` parameter.

For the threat_attributes (attributes) data stream only:

* Correct the expression for extracting the timestamp from an event,
  to account for the fact that the split uses `keep_parent: false`.

Also:

* Correct the PR link in an earlier change log entry.

Discussion

While the system tests do now require that later page sequences use the correct timestamp values, I relied on manual checks to validate that the timestamp value won't be reinitialized following an empty page before data arrives. I used MISP v2.4.183, via misp-docker, as described below.

Adding the order parameter changes the SQL query generated by MISP as follows (for events):

 SELECT `Event`.`id`, `Event`.`attribute_count`
 FROM `misp`.`events` AS `Event`
 WHERE `Event`.`timestamp` >= 1707200151
+ORDER BY `Event`.`timestamp` asc
 LIMIT 10

The order parameter is mentioned in MISP's main documentation but not in it's OpenAI documentation. The documentation says it's "Only available for /events/restSearch", but in fact the restSearch code is shared and it works for /attributes/restSearch as well.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

The following is a guide for how to run MISP locally, observe its interactions with the database and run the Elastic MISP integration against that local instance.

MISP: Configure, run and generate an API token

• Clone https://github.com/misp/misp-docker.
• To enable SQL logging in MariaDB, edit docker-compse.yml and under services.db add command: --general-log --general-log-file=general.log.
• Follow the misp-docker README's Getting Started instructions.
• In MISP, Administration > List Auth Keys, add a new key and copy its value.

MISP: Load data and query

• In MISP, Sync Actions > Feeds, click "Load default feed metadata", enable some feeds and click "Fetch all events" on them.
• In MISP, API > Rest client, you can construct and run queries. It can generate cURL commands (to which you may need to add the --insecure option).

SQL logs

Tail the MariaDB log and highlight queries of interest (SELECTs with a LIMIT that's not 1), as follows:

docker exec -it misp-docker-db-1 tail -f /var/lib/mysql/general.log | \
  grep -P 'SELECT .* LIMIT (?!1[^\d])\d+|$'

MISP integration in Elastic

When adding a MISP integration policy use the following settings:

• MISP URL: https://172.17.0.1 (special IP for the docker host)
• MISP API Token: F2wJWSS4qhRyVOdjMemTNiMW9W9dqdEFgGPjJIGP (from earlier setup)
• Initial Interval: 0s (or more)
• Interval: 10s
• Preserve original event: yes
• SSL: "verification_mode: none"
• Enable request tracing: yes

MISP: other settings (optional)

Additional settings can be added to misp-docker/core/files/configure_misp.sh, for example:

sudo -u www-data /var/www/MISP/app/Console/cake Admin setSetting -q -f "debug" 2
sudo -u www-data /var/www/MISP/app/Console/cake Admin setSetting -q -f "site_admin_debug" true

Setting debug to 2 will show a summary of SQL queries at the bottom of the page in the browser, but to see all queries, including those generated by API requests, use MariaDB's general log as above.

Check live settings as follows:

docker exec misp-docker-misp-core-1 \
  bash -c 'sudo -u www-data /var/www/MISP/app/Console/cake Admin getSetting'

Change a live setting as follows:

docker exec misp-docker-misp-core-1 \
  bash -c 'sudo -u www-data /var/www/MISP/app/Console/cake Admin setSetting --force "debug" 2'

Related issues

• Closes [ti_misp] Wrong timestamp cause no events fetched from MISP #8554

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: a788295

cc @chrisberkhout

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[P1llus]

This looks great! Thanks for such a detailed PR and test description!

No comments from me, only thing I noticed was that while .Unix was removed from one part, the default value still uses it, I assume thats intended since the original value is not Unix?

[chrisberkhout]

while .Unix was removed from one part, the default value still uses it, I assume thats intended since the original value is not Unix?

Yes, it's intended. now (parseDuration "-{{initial_interval}}") will make a time.Time, so we call .Unix on that to get the right format. Later when we're reading the cursor value (which came from the server) we'll have a string with no .Unix.

[elasticmachine]

Package ti_misp - 1.31.0 containing this change is available at https://epr.elastic.co/search?package=ti_misp