Kibana audit logging and usage data

[elasticmachine]

Original comment by @tbragin:

At the moment, the Kibana log contains all requests made by Kibana server to the backend.

respons [09:26:33.815]  POST /elasticsearch/logstash-2014*/_field_stats?level=indices 200 61ms - 9.0B
respons [09:26:33.859]  POST /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1452446450804 200 17ms - 9.0B
respons [09:26:36.699]  POST /elasticsearch/_mget?timeout=0&ignore_unavailable=true&preference=1452446450804 200 8ms - 9.0B
respons [09:26:36.711]  POST /elasticsearch/_mget?timeout=0&ignore_unavailable=true&preference=1452446450804 200 5ms - 9.0B
respons [09:26:45.022]  POST /elasticsearch/.kibana/dashboard/_search?size=100 200 13ms - 9.0B

When we add REST endpoints for accessing dashboards, visualizations, and saved searches, and move that functionality from front-end to back-end, a rudimentary log of accesses to those objects will be available in the Kibana log as well.

As part of Shield, we may want to enhance this capability to provide a true audit log by doing the following:

• Add "user name", so that we can know who accessed what
• Translate URL into "object name" and "action"
• Store this information in the ES audit log index

[elasticmachine]

Original comment by @skearns64:

the Kibana log contains all requests made by Kibana server to the backend

Do you mean that the Kibana log contains all requests from the browser to the Kibana backend, or does it also log the requests from the Kibana backend to ES?

Translate URL into "object name" and "action"

++ to the idea - I sort of think of the Kibana backend as having an API that the browser-part of Kibana calls, so I think of each of these APIs as having an "action type" and each action as having "loggable metadata". With or without Shield, each log would have an action, and some structured metadata, similar to how ES logs the the node, and index, etc. Then, with Shield installed, the loggable metadata for every action could simply have a user associated with it.

Also, note that to some extent, I think this plays into what we do about LINK REDACTED which might change how the Kibana backend communicates with ES (now it's always "as" the user, that issue proposes moving all requests to the .kibana index to use the "Kibana Server" user.

[elasticmachine]

Original comment by @seang-es:

We also need to track login/logout/session state change events.

[elasticmachine]

Original comment by @tbragin:

There was a request to optionally log full request bodies here: LINK REDACTED

[elasticmachine]

Original comment by @damianpfister:

Recently a customer came up with an interesting way of doing this via a code change. No idea whether it will work or is viable though (waiting for feedback from them).

Ref: LINK REDACTED

[elasticmachine]

Original comment by @gmoskovicz:

@tbragin any news regarding this ?

[elasticmachine]

Original comment by @tbragin:

@alexfrancoeur @joshbressers Could you comment?

[elasticmachine]

Original comment by @alexfrancoeur:

I'm not sure how granular the current audit log files are for Elasticsearch, but I was under the impression that you get everything. Which I believe LINK REDACTED is meant to address with additional filtering capabilities. I do not believe we have any immediate plans for introducing new Kibana audit logs, but am wondering how far filtering gets us for this use case.

@joshbressers thoughts?

[elasticmachine]

Original comment by @gmoskovicz:

I think that this is very important. Elasticsearch audit logs are fantastic, but from my experience with real use cases (troubleshooting things sometimes) it's almost impossible to correlate and event from Kibana with what happened in Elasticsearch. It would be great to have some sort of audit logging in Kibana as well to see who was doing what and investigate things on each end rather than just at the Elasticsearch level.

Keep in mind that some of the audit logs are per node, and take things at the shard level, so when you do a query, trying to match what happened when opening a dashboard inside Elasticsearch is very hard.

[elasticmachine]

Original comment by @joshbressers:

I have this as a note on my big roadmap, I do think we should formalize the plan here. Ideally we provide a consistent experience across the stack.

There are more urgent things on my list than this, but it is on my list.

[elasticmachine]

Original comment by @LeeDr:

I'm responding to a user on Discuss right now who wants to see which users open which dashboards and which dashboards get used those most. Short answer is that I don't think it's possible;

I don't think we have logging that can show you that right now.

I tried turning on Elasticsearch audit logging. That does show the username as they open a dashboard (something like a GET request) but it doesn't log the body of the request or response so you don't get the title (or id) of the dashboard that user opened.

I turned on Kibana verbose logging and don't get either username or dashboard name there.

And I turned on the slowlog logging in Elasticsearch for the .kibana index and that also doesn't show the username opening the dashboard or the name of the dashboard.

[arisonl]

Tying Elasticsearch queries to Kibana users for security and performance purposed, as per linked ERs: #16493

[jportner]

I renamed this issue to more accurately describe the linked enhancement requests, and opened another issue with a smaller scope for just audit logging, see #52125.

[tbragin]

@kobelb Should this issue be closed in favor of #52125 and #54836 ?

[kobelb]

@tbragin This issue was originally left open because it encompasses requests for both audit logging and more general usage data. However, I see the detriment in leaving this issue around as we continue to conflate the two enhancement requests. I'd prefer we just close out this issue. @arisonl and @alexfrancoeur, do you happen to know if we have another issue for usage data collection?

[arisonl]

@kobelb @tbragin I was going through the audit logging ERs as we merged the new Kibana audit logging capability and it became evident that an issue around usage is required: #81130 I will update the corresponding ERs to point to this one.

[kobelb]

Thanks, @arisonl!

[lukeelmers]

Looks like we intended to close this in favor of #81130, but forgot to do so. Closing now.