Index audit trail loses events on a rolling upgrade

[nik9000]

The index based audit trail loses data on a rolling upgrade because the component doesn't start until the template for the audit trail is upgraded to the newest version and that doesn't happen until the a node with the new code is elected the master. I feel like we should document this somewhere or fix it.

[elasticmachine]

Pinging @elastic/es-security

[jaymode]

I feel like we should document this somewhere or fix it.

I agree that this is not ideal, but this has been the behavior for a long time and is also part of why we plan to remove this feature and recommend filebeat (#29881).

We currently say:

The index output type should be used in conjunction with the logfile output type Because it is possible for the index output type to lose messages if the target index is unavailable, the access.log should be used as the official record of events.

We could make that statement a bit more generic as there are other cases where audit events can be lost.

[nik9000]

Sorry! I didn't see that. I was scanning but not reading carefully. I wonder if we should use a warning tag?

[jaymode]

++ to use a warning tag. I had to go searching for that statement; it wasn't where I first thought it would be.

@lcawl do you have any thoughts on how to make the warning clearer?