[Ingest] check and create default indices during setup

[neptunian]

#62343

• add function setup endpoint that creates a placeholder index to supress errors that no matching index exists
• add readFromDocValues property to each index pattern field to prevent apps like Discovery and Management -> Index Patterns -> (index pattern) from creating a new index pattern from the placeholder index and overwriting our entire index pattern (see Index Pattern overwritten when an index exists #65284). The value should be the same as doc_values

Test:

Go to Dashboads and click on a dashboard for system or nginx FIlebeat. Before there could be many notices about no matching idices:

Now you should not see any toast errors:

This PR does not resolve these errors.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[neptunian]

@elasticmachine merge upstream

[ruflin]

Nit:

Suggested change

	// create a placeholder index to supress errors in the kibana Dashboards app
	// create a placeholder indicies to supress errors in the kibana Dashboards app

Could you also add a link here to the open Kibana issue?

[ruflin]

I tested this locally and got two errors.

The first one I assume comes from a missing field in the Endpoint package:

This one is about the missing @timestamp field. I wonder if this is solved if we add an @timestamp field with a data to the doc we create?

[neptunian]

@ruflin I'm not creating a documents in this PR, I'm just creating an index. where are you getting the timestamp error?

[ruflin]

@neptunian One was the endpoint dashboard, the other one I think was just on discovery page selecting one of the index pattern. I didn't ingest any data.

[neptunian]

@ruflin it looks like the reason it's getting that @timestamp error is because if I PUT an index, whenever I visit the index patterns in the management page or the Discovery app it will overwrite whatever index pattern already exists (logs-*) with the existing index that's matching it, which in this case is an empty index.

Same with events. The errors for these fields may be because they were overwritten by the new placeholder index.

So, even though, during setup, a lot of fields were added to the index pattern due to installation of the default packages, they will be overwritten visiting these pages if no indices exist or they exist with a few fields. They create the patterns automatically.

[neptunian]

@elasticmachine merge upstream

[ruflin]

I tested this locally. No errors show up and dashboards seem to still work as expected. I checked 2 system dashboards.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 2e9f693

History

• 💔 Build #45717 failed b0e46ec
• 💔 Build #45683 failed 4a2634b
• 💚 Build #45117 succeeded c6c4ee1
• 💚 Build #44536 succeeded a2152e8
• 💚 Build #44218 succeeded ff5d11f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[jasonrhodes]

Hey all, I would love to understand more about this and what the plan is to get around this more completely. This change broke some assumptions we were making in the Logs and Metrics apps around when we can assume "logs-" and "metrics-" indices would exist (i.e. their existence should imply the intention to ingest data of that type). Our assumption was a bit flaky, but this change also feel pretty tenuous. Thanks!

[ruflin]

@jasonrhodes I think we haven't figured out the final solution yet. The above is kind of a hack but otherwise users see lots of errors because of the way index patterns work today. I would hope we can change the way index patterns work ;-) Happy to sync up if it didn't happen yet.