[ML] Add context information to search slow log entries

[jgowdyelastic]

We should be able to add context information to all searches so they appear in the search slow log. This applies to all the plugins the ML team maintains.

v8.7

• ml [JG]
  [ML] Adding ML execution context to es requests #148746

v8.8

• transforms [WR]
  [ML] Transforms: Adding execution context to ES requests. #153649
• aiops [WR]
  [ML] AIOps: Adding execution context to ES requests. #153778
• data_visualizer [QN]
  [ML] Add execution context to Field statistic table for Discover and Index data visualizer #154404
• Update client side code which uses the data.search plugin [WR]
  [ML] AIOps: Adds execution context to client side data.search requests #154891

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[sophiec20]

Does this show user id as well as component and plugin?
Are we targeting this for 8.7?

[peteharverson]

@sophiec20 in answer to your questions:

Does this show user id as well as component and plugin?

The information added in #148746 does not include user ID. We will need to investigate if this is possible. We aren't aware of any other plugins which currently add the user.

Are we targeting this for 8.7?

The changes for the ml plugin are in for 8.7, but transforms, aiops and data_visualizer will follow in a later release.

[sophiec20]

The information added in #148746 does not include user ID. We will need to investigate if this is possible. We aren't aware of any other plugins which currently add the user.

I wasn't saying it "must", at this stage it is just a question. Would still be useful to investigate what is possible.

[jgowdyelastic]

I think the decision on whether we add user ID to the slow log would need to be made by someone in the elasticsearch, security or core kibana teams. We could be exposing sensitive information.
Also, I don't think it should be added to the part of the log which we are able to customise (the "elasticsearch.slowlog.id" property) as this is only being used to identify the location in app or endpoint which has triggered the search.

User ID could be added as a separate property in the log, but this is not something we could do.

example log document:

{
  "@timestamp": "2023-01-13T10:10:21.985Z",
  "log.level": "TRACE",
  "elasticsearch.slowlog.id": "a90d5297-fd77-4ea0-ac0d-c302963d7e75;kibana:application:ml:%2Fjobs%2Fnew_job%2Fsingle_metric;application:ml:%2Fapi%2Fml%2Fjobs%2Fnew_job_line_chart",
  "elasticsearch.slowlog.message": "[gtfsbeat-8.0.0-2019][0]",
  "elasticsearch.slowlog.search_type": "QUERY_THEN_FETCH",
  "elasticsearch.slowlog.source": "{\\\"size\\\":0,\\\"query\\\":{\\\"bool\\\":{\\\"must\\\":[{\\\"match_all\\\":{\\\"boost\\\":1.0}},{\\\"range\\\":{\\\"@timestamp\\\":{\\\"boost\\\":1.0}}}],\\\"boost\\\":1.0}},\\\"track_total_hits\\\":2147483647,\\\"aggregations\\\":{\\\"times\\\":{\\\"date_histogram\\\":{\\\"field\\\":\\\"@timestamp\\\",\\\"fixed_interval\\\":\\\"7200000ms\\\",\\\"offset\\\":0,\\\"order\\\":{\\\"_key\\\":\\\"asc\\\"},\\\"keyed\\\":false,\\\"min_doc_count\\\":0,\\\"extended_bounds\\\":{\\\"min\\\":1556141284000,\\\"max\\\":1560430139000}}}}}",
  "elasticsearch.slowlog.stats": "[]",
  "elasticsearch.slowlog.took": "132.3ms",
  "elasticsearch.slowlog.took_millis": 132,
  "elasticsearch.slowlog.total_hits": "2882527 hits",
  "elasticsearch.slowlog.total_shards": 1,
  "ecs.version": "1.2.0",
  "service.name": "ES_ECS",
  "event.dataset": "elasticsearch.index_search_slowlog",
  "process.thread.name": "elasticsearch[node-0][search][T#10]",
  "log.logger": "index.search.slowlog.query",
  "trace.id": "efcb4fa736fa31bd8a36695e83b7507b",
  "elasticsearch.cluster.uuid": "XPf0bmrNSRy74IXugIAcUg",
  "elasticsearch.node.id": "ymkG7GGFR9WAhCP7xmsfHQ",
  "elasticsearch.node.name": "node-0",
  "elasticsearch.cluster.name": "default-8.7.0-2427"
}

[peteharverson]

Closing issue as all the planned work has been completed for 8.8.