[event log] unable to set some old event log alias to hidden when underlying indices are not hidden

[pmuellr]

Kibana version: 7.16.0

Describe the bug:

Customer sees the following message (and similar related ones) at Kibana start time:

error setting existing \".kibana-event-log-7.9.2-000012\" index aliases - error setting existing index aliases for index .kibana-event-log-7.9.2-000012 to is_hidden: illegal_state_exception: [illegal_state_exception] Reason: alias [.kibana-event-log-7.9.2] has is_hidden set to true on indices [.kibana-event-log-7.9.2-000012] but does not have is_hidden set to true on indices [.kibana-event-log-7.9.2-000015,.kibana-event-log-7.9.2-000014,.kibana-event-log-7.9.2-000013]; alias must have the same is_hidden setting on all indices

It seems like this is likely coming from the code that changes all the existing event log indices to hidden. I haven't looked, but I believe this is just an informative message - the event log continues on after logging this error. Presumably some of the event log resources will not be marked hidden, but that won't affect the runtime at all. I believe, upon a restart, it will run the same code again - and perhaps it will work this time, if it happened before because of some timing issue. But I also wouldn't be surprised if for some reason it will run into the same condition and produce the same (or similar) log messages.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

Quick peek; it looks like this code is not waiting on the promise returned from asyncForEach() ... and same for the two calls to asyncForEach() below it. That would explain the message - the code to set the indices to hidden hasn't completed yet, by the time it tries to set the alias to hidden.

kibana/x-pack/plugins/event_log/server/es/init.ts

Lines 45 to 85 in 72e714c

	async setExistingIndexTemplatesToHidden() {
	let indexTemplates: estypes.IndicesGetTemplateResponse = {};
	try {
	// look up existing index templates and update index.hidden to true if that
	// setting is currently false or undefined
	// since we are updating to the new index template API and converting new event log
	// indices to hidden in the same PR, we only need to use the legacy template API to
	// look for and update existing event log indices.
	indexTemplates = await this.esContext.esAdapter.getExistingLegacyIndexTemplates(
	this.esContext.esNames.indexPattern
	);
	} catch (err) {
	// errors when trying to get existing index templates
	// should not block the rest of initialization, log the error and move on
	this.esContext.logger.error(`error getting existing index templates - ${err.message}`);
	}
	asyncForEach(Object.keys(indexTemplates), async (indexTemplateName: string) => {
	try {
	const hidden: string | boolean = indexTemplates[indexTemplateName]?.settings?.index?.hidden;
	// Check to see if this index template is hidden
	if (hidden !== true && hidden !== 'true') {
	this.esContext.logger.debug(
	`setting existing "${indexTemplateName}" index template to hidden.`
	);
	await this.esContext.esAdapter.setLegacyIndexTemplateToHidden(
	indexTemplateName,
	indexTemplates[indexTemplateName]
	);
	}
	} catch (err) {
	// errors when trying to update existing index templates to hidden
	// should not block the rest of initialization, log the error and move on
	this.esContext.logger.error(
	`error setting existing "${indexTemplateName}" index template to hidden - ${err.message}`
	);
	}
	});
	}

[tshayan]

Just wanted to report that I just encountered this issue after upgrading Kibana from 7.13.4 to 7.16.2. Everything seems to be working normally but I see the below message every time that Kibana is restarted:

{"type":"log","@timestamp":"2022-01-12T13:49:53-05:00","tags":["error","plugins","eventLog"],"pid":7,"message":"error setting existing ".kibana-admin-event-log-7.13.4-000004" index aliases - error setting existing index aliases for index .kibana-admin-event-log-7.13.4-000004 to is_hidden: illegal_state_exception: [illegal_state_exception] Reason: alias [.kibana-admin-event-log-7.13.4] has is_hidden set to true on indices [.kibana-admin-event-log-7.13.4-000004] but does not have is_hidden set to true on indices [.kibana-admin-event-log-7.13.4-000003,.kibana-admin-event-log-7.13.4-000001,.kibana-admin-event-log-7.13.4-000002]; alias must have the same is_hidden setting on all indices"}
{"type":"log","@timestamp":"2022-01-12T13:49:53-05:00","tags":["error","plugins","eventLog"],"pid":7,"message":"error setting existing ".kibana-admin-event-log-7.13.4-000003" index aliases - error setting existing index aliases for index .kibana-admin-event-log-7.13.4-000003 to is_hidden: illegal_state_exception: [illegal_state_exception] Reason: alias [.kibana-admin-event-log-7.13.4] has is_hidden set to true on indices [.kibana-admin-event-log-7.13.4-000003] but does not have is_hidden set to true on indices [.kibana-admin-event-log-7.13.4-000001,.kibana-admin-event-log-7.13.4-000004,.kibana-admin-event-log-7.13.4-000002]; alias must have the same is_hidden setting on all indices"}
{"type":"log","@timestamp":"2022-01-12T13:49:53-05:00","tags":["error","plugins","eventLog"],"pid":7,"message":"error setting existing ".kibana-admin-event-log-7.13.4-000001" index aliases - error setting existing index aliases for index .kibana-admin-event-log-7.13.4-000001 to is_hidden: illegal_state_exception: [illegal_state_exception] Reason: alias [.kibana-admin-event-log-7.13.4] has is_hidden set to true on indices [.kibana-admin-event-log-7.13.4-000001] but does not have is_hidden set to true on indices [.kibana-admin-event-log-7.13.4-000003,.kibana-admin-event-log-7.13.4-000004,.kibana-admin-event-log-7.13.4-000002]; alias must have the same is_hidden setting on all indices"}
{"type":"log","@timestamp":"2022-01-12T13:49:53-05:00","tags":["error","plugins","eventLog"],"pid":7,"message":"error setting existing ".kibana-admin-event-log-7.13.4-000002" index aliases - error setting existing index aliases for index .kibana-admin-event-log-7.13.4-000002 to is_hidden: illegal_state_exception: [illegal_state_exception] Reason: alias [.kibana-admin-event-log-7.13.4] has is_hidden set to true on indices [.kibana-admin-event-log-7.13.4-000002] but does not have is_hidden set to true on indices [.kibana-admin-event-log-7.13.4-000003,.kibana-admin-event-log-7.13.4-000001,.kibana-admin-event-log-7.13.4-000004]; alias must have the same is_hidden setting on all indices"}

[buzzdeee]

saw these as well after upgrading from 7.15.0 -> 7.16.2

[danparisi]

Same for me, after upgrading to v. 7.16.2