Migrations only run once #32237

jbudz · 2019-02-28T19:04:38Z

Start Kibana
Start Elasticsearch
Migrations run
Restore unmigrated .kibana, find missing .kibana shard, upgrade elasticsearch, etc

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "[illegal_argument_exception] application privileges must refer to at least one resource"
}

 log   [18:58:00.266] [error][status][plugin:spaces@6.6.1] Status changed from red to red - mapping set to strict, dynamic introduction of [space] within [doc] is not allowed: [strict_dynamic_mapping_exception] mapping set to strict, dynamic introduction of [space] within [doc] is not allowed

cc @elastic/kibana-security for awareness

The text was updated successfully, but these errors were encountered:

elasticmachine · 2019-02-28T19:04:40Z

Pinging @elastic/kibana-operations

kobelb · 2019-02-28T20:50:00Z

That spaces error occurs when the default space document is deleted. We insert that only once on startup.

Are we restoring an unmigrated .kibana from a pre 6.5 Kibana?

jbudz · 2019-02-28T22:48:33Z

Yup in this case 6.4.3.

kobelb · 2019-03-01T01:27:28Z

It feels like a lot of stuff in the future will break if users skip the migrations and run from an old index.

guillain · 2019-05-30T08:46:03Z

Hi,

I've the same issue after to have upgraded elastic.co from 6.8.0 to 7.1.1.

My first error after the upgrade was "mapping set to strict, dynamic introduction of [references] within [doc] is not allowed" so I deleted the .kibana* indexes.
The second error was this one and as I'm on the elastic cloud don't know how can I solve it (as the last time that I deleted my Kibana instance it was impossibe to recreate it, hopefully I try on dev pf ^^)

Thanks for your support and guidance ;)

elasticmachine · 2020-03-26T22:19:42Z

Pinging @elastic/kibana-platform (Team:Platform)

pgayvallet · 2020-03-30T06:50:46Z

After initial migration, we are also migrating every object during actual access. However, we indeed are only creating the mapping once, during startup. Meaning that this

start kibana, migration is performed
restore backup from previous version (and therefor previous schema) without kbn restart
insert SO object with NEW schema

Will cause that kind of errors, which are basically fatals atm, during next access of the index:

mapping set to strict, dynamic introduction of [XXX] within [YYY] is not allowed: [strict_dynamic_mapping_exception] mapping set to strict.

Not sure what we should be doing to try to avoid that though. We would need a way to detect that the index schema has been modified from an external source and reapply migration? Or identify this specific error and (try to) reapply migration when it occur?

BTW

Restore unmigrated .kibana, find missing .kibana shard, upgrade elasticsearch, etc

Restore unmigrated .kibana

If not already, we really should document that such operation should not be done with a running kbn instance

find missing .kibana shard

This is a real and valid issue.

upgrade elasticsearch

Not sure to see how what would cause the mapping to differ at some point?

joshdover · 2020-05-07T14:46:43Z

This could be solved by the same mechanism proposed in #52202 to prevent old nodes from interfering with upgraded/migrated indices.

rudolf · 2021-01-14T16:00:39Z

Without continuous polling we can't detect an index being changed/restored and unless the polling window is really small we won't be able to prevent the problems from happening. So I don't think we will ever be able to support Kibana running while it's indices are removed/restored to a previous version, but we should aim to reduce the impact.

To reduce the impact:

Kibana will stop auto-creating it's system indices Kibana should not auto create it's system indices if these don't exist #81790 This will ensure that if Kibana's indices are deleted Kibana doesn't accept writes into an auto-created index with the wrong mappings.
Kibana will only read/write saved objects into a version specific index. So together with the previous point, this will prevent Kibana from doing any reads/writes when an index from an earlier version is restored details in https://github.com/elastic/kibana/blob/master/rfcs/text/0013_saved_object_migrations.md#L0-L1 and implementation tracked in Implement SavedObject Migrations v2 #75780 Upon restarting the process Kibana will perform a migration and become operational again.

If writing to a non-existing index causes a fatal error Kibana will automatically shutdown and in most deployments restart itself, which means for most users Kibana will automatically recover from an earlier version's index being restored.

blankhang · 2021-06-28T01:38:52Z

{
"statusCode": 500,
"error": "Internal Server Error",
"message": "[illegal_argument_exception]: application privileges must refer to at least one resource"
}
elk 7.13.0

docker stack ps elk
ID             NAME                      IMAGE                  NODE      DESIRED STATE   CURRENT STATE         ERROR                         PORTS
p55l69sfipll   elk_elasticsearch.1       elasticsearch:7.13.0   test2     Running         Running 8 hours ago                                 
626vmzz8zadi    \_ elk_elasticsearch.1   elasticsearch:7.13.0   test2     Shutdown        Failed 8 hours ago    "task: non-zero exit (127)"   
lyqsr2tzgj03   elk_kibana.1              kibana:7.13.0          test2     Running         Running 2 days ago                                  
2wc3tpoenh9i   elk_logstash.1            logstash:7.13.0        test2     Running         Running 2 days ago

blankhang · 2021-06-28T02:08:49Z

kibana log


Search
Filter...
Lines
100
Actions   
      
{"type":"log","@timestamp":"2021-06-28T01:56:06+00:00","tags":["error","plugins","taskManager"],"pid":952,"message":"[WorkloadAggregator]: Error: Invalid workload: {\"took\":0,\"timed_out\":false,\"_shards\":{\"total\":0,\"successful\":0,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":{\"value\":0,\"relation\":\"eq\"},\"max_score\":0,\"hits\":[]}}"}

{"type":"log","@timestamp":"2021-06-28T01:56:12+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:56:22+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:56:32+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:56:42+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:56:52+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:57:02+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:57:06+00:00","tags":["error","plugins","taskManager"],"pid":952,"message":"[WorkloadAggregator]: Error: Invalid workload: {\"took\":0,\"timed_out\":false,\"_shards\":{\"total\":0,\"successful\":0,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":{\"value\":0,\"relation\":\"eq\"},\"max_score\":0,\"hits\":[]}}"}

{"type":"log","@timestamp":"2021-06-28T01:57:12+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:57:22+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:57:32+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:57:42+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:57:52+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:58:02+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:58:06+00:00","tags":["error","plugins","taskManager"],"pid":952,"message":"[WorkloadAggregator]: Error: Invalid workload: {\"took\":0,\"timed_out\":false,\"_shards\":{\"total\":0,\"successful\":0,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":{\"value\":0,\"relation\":\"eq\"},\"max_score\":0,\"hits\":[]}}"}

{"type":"log","@timestamp":"2021-06-28T01:58:12+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:58:22+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:58:32+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:58:42+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:58:52+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:59:02+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:59:06+00:00","tags":["error","plugins","taskManager"],"pid":952,"message":"[WorkloadAggregator]: Error: Invalid workload: {\"took\":2,\"timed_out\":false,\"_shards\":{\"total\":0,\"successful\":0,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":{\"value\":0,\"relation\":\"eq\"},\"max_score\":0,\"hits\":[]}}"}

{"type":"log","@timestamp":"2021-06-28T01:59:12+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:59:22+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:59:32+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:59:42+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T01:59:52+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:00:02+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:00:06+00:00","tags":["error","plugins","taskManager"],"pid":952,"message":"[WorkloadAggregator]: Error: Invalid workload: {\"took\":0,\"timed_out\":false,\"_shards\":{\"total\":0,\"successful\":0,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":{\"value\":0,\"relation\":\"eq\"},\"max_score\":0,\"hits\":[]}}"}

{"type":"log","@timestamp":"2021-06-28T02:00:12+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:00:22+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:00:32+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:00:42+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"response","@timestamp":"2021-06-28T02:00:45+00:00","tags":[],"pid":952,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:36.0) Gecko/20100101 Firefox/36.0","host":"120.77.246.103:5601","connection":"Keep-Alive","accept-encoding":"gzip"},"remoteAddress":"10.0.0.2","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:36.0) Gecko/20100101 Firefox/36.0"},"res":{"statusCode":302,"responseTime":2},"message":"GET / 302 2ms"}

{"type":"response","@timestamp":"2021-06-28T02:00:45+00:00","tags":[],"pid":952,"method":"get","statusCode":200,"req":{"url":"/login?next=%2F","method":"get","headers":{"user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:36.0) Gecko/20100101 Firefox/36.0","host":"120.77.246.103:5601","connection":"Keep-Alive","accept-encoding":"gzip"},"remoteAddress":"10.0.0.2","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:36.0) Gecko/20100101 Firefox/36.0"},"res":{"statusCode":200,"responseTime":30,"contentLength":132438},"message":"GET /login?next=%2F 200 30ms - 129.3KB"}

{"type":"log","@timestamp":"2021-06-28T02:00:52+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:01:02+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:01:06+00:00","tags":["error","plugins","taskManager"],"pid":952,"message":"[WorkloadAggregator]: Error: Invalid workload: {\"took\":0,\"timed_out\":false,\"_shards\":{\"total\":0,\"successful\":0,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":{\"value\":0,\"relation\":\"eq\"},\"max_score\":0,\"hits\":[]}}"}

{"type":"log","@timestamp":"2021-06-28T02:01:12+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:01:22+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:01:32+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:01:42+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:01:52+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:02:02+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:02:06+00:00","tags":["error","plugins","taskManager"],"pid":952,"message":"[WorkloadAggregator]: Error: Invalid workload: {\"took\":0,\"timed_out\":false,\"_shards\":{\"total\":0,\"successful\":0,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":{\"value\":0,\"relation\":\"eq\"},\"max_score\":0,\"hits\":[]}}"}

{"type":"log","@timestamp":"2021-06-28T02:02:12+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:02:22+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:02:32+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:02:42+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:02:52+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:03:02+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:03:06+00:00","tags":["error","plugins","taskManager"],"pid":952,"message":"[WorkloadAggregator]: Error: Invalid workload: {\"took\":0,\"timed_out\":false,\"_shards\":{\"total\":0,\"successful\":0,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":{\"value\":0,\"relation\":\"eq\"},\"max_score\":0,\"hits\":[]}}"}

{"type":"log","@timestamp":"2021-06-28T02:03:12+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:03:22+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:03:32+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:03:42+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:03:52+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:04:02+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:04:06+00:00","tags":["error","plugins","taskManager"],"pid":952,"message":"[WorkloadAggregator]: Error: Invalid workload: {\"took\":0,\"timed_out\":false,\"_shards\":{\"total\":0,\"successful\":0,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":{\"value\":0,\"relation\":\"eq\"},\"max_score\":0,\"hits\":[]}}"}

{"type":"log","@timestamp":"2021-06-28T02:04:12+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:04:22+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:04:32+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:04:42+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:04:52+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:05:02+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:05:06+00:00","tags":["error","plugins","taskManager"],"pid":952,"message":"[WorkloadAggregator]: Error: Invalid workload: {\"took\":0,\"timed_out\":false,\"_shards\":{\"total\":0,\"successful\":0,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":{\"value\":0,\"relation\":\"eq\"},\"max_score\":0,\"hits\":[]}}"}

{"type":"log","@timestamp":"2021-06-28T02:05:12+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:05:22+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:05:32+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:05:42+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:05:52+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:06:02+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:06:06+00:00","tags":["error","plugins","taskManager"],"pid":952,"message":"[WorkloadAggregator]: Error: Invalid workload: {\"took\":0,\"timed_out\":false,\"_shards\":{\"total\":0,\"successful\":0,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":{\"value\":0,\"relation\":\"eq\"},\"max_score\":0,\"hits\":[]}}"}

{"type":"log","@timestamp":"2021-06-28T02:06:12+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:06:22+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:06:32+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:06:42+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:06:52+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:07:02+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:07:06+00:00","tags":["error","plugins","taskManager"],"pid":952,"message":"[WorkloadAggregator]: Error: Invalid workload: {\"took\":0,\"timed_out\":false,\"_shards\":{\"total\":0,\"successful\":0,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":{\"value\":0,\"relation\":\"eq\"},\"max_score\":0,\"hits\":[]}}"}

{"type":"error","@timestamp":"2021-06-28T02:07:07+00:00","tags":[],"pid":952,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:124:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:78:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:73:17)\n    at interceptRequest (/usr/share/kibana/src/core/server/http/lifecycle/on_post_auth.js:58:36)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Request._invoke (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:397:30)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)"},"url":"http://kibana.test.storlead.com/","message":"Internal Server Error"}

{"type":"response","@timestamp":"2021-06-28T02:07:07+00:00","tags":[],"pid":952,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":{"host":"kibana.test.storlead.com","x-real-ip":"163.125.80.73","x-forwarded-for":"163.125.80.73","x-forwarded-proto":"https","connection":"close","cache-control":"max-age=0","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"zh,zh-CN;q=0.9,en;q=0.8"},"remoteAddress":"10.0.0.2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59"},"res":{"statusCode":500,"responseTime":43,"contentLength":151},"message":"GET / 500 43ms - 151.0B"}

{"type":"error","@timestamp":"2021-06-28T02:07:09+00:00","tags":[],"pid":952,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:124:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:78:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:73:17)\n    at interceptRequest (/usr/share/kibana/src/core/server/http/lifecycle/on_post_auth.js:58:36)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Request._invoke (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:397:30)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)"},"url":"http://kibana.test.storlead.com/","message":"Internal Server Error"}

{"type":"response","@timestamp":"2021-06-28T02:07:09+00:00","tags":[],"pid":952,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":{"host":"kibana.test.storlead.com","x-real-ip":"163.125.80.73","x-forwarded-for":"163.125.80.73","x-forwarded-proto":"https","connection":"close","cache-control":"max-age=0","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"zh,zh-CN;q=0.9,en;q=0.8"},"remoteAddress":"10.0.0.2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59"},"res":{"statusCode":500,"responseTime":33,"contentLength":151},"message":"GET / 500 33ms - 151.0B"}

{"type":"error","@timestamp":"2021-06-28T02:07:10+00:00","tags":[],"pid":952,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:124:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:78:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:73:17)\n    at interceptRequest (/usr/share/kibana/src/core/server/http/lifecycle/on_post_auth.js:58:36)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Request._invoke (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:397:30)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)"},"url":"http://kibana.test.storlead.com/","message":"Internal Server Error"}

{"type":"response","@timestamp":"2021-06-28T02:07:10+00:00","tags":[],"pid":952,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":{"host":"kibana.test.storlead.com","x-real-ip":"163.125.80.73","x-forwarded-for":"163.125.80.73","x-forwarded-proto":"https","connection":"close","cache-control":"max-age=0","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"zh,zh-CN;q=0.9,en;q=0.8"},"remoteAddress":"10.0.0.2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59"},"res":{"statusCode":500,"responseTime":40,"contentLength":151},"message":"GET / 500 40ms - 151.0B"}

{"type":"error","@timestamp":"2021-06-28T02:07:11+00:00","tags":[],"pid":952,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:124:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:78:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:73:17)\n    at interceptRequest (/usr/share/kibana/src/core/server/http/lifecycle/on_post_auth.js:58:36)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Request._invoke (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:397:30)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)"},"url":"http://kibana.test.storlead.com/","message":"Internal Server Error"}

{"type":"response","@timestamp":"2021-06-28T02:07:11+00:00","tags":[],"pid":952,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":{"host":"kibana.test.storlead.com","x-real-ip":"163.125.80.73","x-forwarded-for":"163.125.80.73","x-forwarded-proto":"https","connection":"close","cache-control":"max-age=0","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"zh,zh-CN;q=0.9,en;q=0.8"},"remoteAddress":"10.0.0.2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59"},"res":{"statusCode":500,"responseTime":33,"contentLength":151},"message":"GET / 500 33ms - 151.0B"}

{"type":"error","@timestamp":"2021-06-28T02:07:12+00:00","tags":[],"pid":952,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:124:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:78:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:73:17)\n    at interceptRequest (/usr/share/kibana/src/core/server/http/lifecycle/on_post_auth.js:58:36)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Request._invoke (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:397:30)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)"},"url":"http://kibana.test.storlead.com/","message":"Internal Server Error"}

{"type":"response","@timestamp":"2021-06-28T02:07:12+00:00","tags":[],"pid":952,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":{"host":"kibana.test.storlead.com","x-real-ip":"163.125.80.73","x-forwarded-for":"163.125.80.73","x-forwarded-proto":"https","connection":"close","cache-control":"max-age=0","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"zh,zh-CN;q=0.9,en;q=0.8"},"remoteAddress":"10.0.0.2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59"},"res":{"statusCode":500,"responseTime":34,"contentLength":151},"message":"GET / 500 34ms - 151.0B"}

{"type":"log","@timestamp":"2021-06-28T02:07:12+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"error","@timestamp":"2021-06-28T02:07:12+00:00","tags":[],"pid":952,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:124:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:78:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:73:17)\n    at interceptRequest (/usr/share/kibana/src/core/server/http/lifecycle/on_post_auth.js:58:36)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Request._invoke (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:397:30)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)"},"url":"http://kibana.test.storlead.com/","message":"Internal Server Error"}

{"type":"response","@timestamp":"2021-06-28T02:07:12+00:00","tags":[],"pid":952,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":{"host":"kibana.test.storlead.com","x-real-ip":"163.125.80.73","x-forwarded-for":"163.125.80.73","x-forwarded-proto":"https","connection":"close","cache-control":"max-age=0","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"zh,zh-CN;q=0.9,en;q=0.8"},"remoteAddress":"10.0.0.2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59"},"res":{"statusCode":500,"responseTime":46,"contentLength":151},"message":"GET / 500 46ms - 151.0B"}

{"type":"error","@timestamp":"2021-06-28T02:07:13+00:00","tags":[],"pid":952,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:124:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:78:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:73:17)\n    at interceptRequest (/usr/share/kibana/src/core/server/http/lifecycle/on_post_auth.js:58:36)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Request._invoke (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:397:30)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)"},"url":"http://kibana.test.storlead.com/","message":"Internal Server Error"}

{"type":"response","@timestamp":"2021-06-28T02:07:13+00:00","tags":[],"pid":952,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":{"host":"kibana.test.storlead.com","x-real-ip":"163.125.80.73","x-forwarded-for":"163.125.80.73","x-forwarded-proto":"https","connection":"close","cache-control":"max-age=0","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"zh,zh-CN;q=0.9,en;q=0.8"},"remoteAddress":"10.0.0.2","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59"},"res":{"statusCode":500,"responseTime":34,"contentLength":151},"message":"GET / 500 34ms - 151.0B"}

{"type":"log","@timestamp":"2021-06-28T02:07:22+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:07:32+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:07:42+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:07:52+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

{"type":"log","@timestamp":"2021-06-28T02:08:02+00:00","tags":["warning","environment"],"pid":952,"message":"Detected an unhandled Promise rejection.\nResponseError: Saved object index alias [.kibana_7.13.0] not found: index_not_found_exception"}

blankhang · 2021-06-28T02:09:52Z

docker-stack.yml

version: '3.3'

services:

  elasticsearch:
    image: elasticsearch:7.13.0
    ports:
      - "9200:9200"
      - "9300:9300"
    configs:
      - source: elastic_config
        target: /usr/share/elasticsearch/config/elasticsearch.yml
    environment:
      ES_JAVA_OPTS: "-Xmx512m -Xms512m"
      ELASTIC_PASSWORD: changeme
      # Use single node discovery in order to disable production mode and avoid bootstrap checks.
      # see: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
      discovery.type: single-node
      # Force publishing on the 'elk' overlay.
      network.publish_host: _eth0_
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1

  logstash:
    image: logstash:7.13.0
    ports:
      - "5044:5044"
      - "5000:5000"
      - "9600:9600"
    configs:
      - source: logstash_config
        target: /usr/share/logstash/config/logstash.yml
      - source: logstash_pipeline
        target: /usr/share/logstash/pipeline/logstash.conf
    environment:
      LS_JAVA_OPTS: "-Xmx512m -Xms512m"
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1

  kibana:
    image: kibana:7.13.0
    ports:
      - "5601:5601"
    configs:
      - source: kibana_config
        target: /usr/share/kibana/config/kibana.yml
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1

configs:

  elastic_config:
    file: ./elasticsearch/config/elasticsearch.yml
  logstash_config:
    file: ./logstash/config/logstash.yml
  logstash_pipeline:
    file: ./logstash/pipeline/logstash.conf
  kibana_config:
    file: ./kibana/config/kibana.yml

networks:
  elk:
    driver: overlay

blankhang · 2021-06-28T02:17:39Z

here is the source

pgayvallet · 2021-06-28T06:13:11Z

@blankhang if you have a specific problem, please open a new issue with the detail and description.

jbudz added bug Fixes for quality problems that affect the customer experience Team:Operations Team label for Operations Team labels Feb 28, 2019

jbudz mentioned this issue Aug 9, 2019

Kibana: mapping set to strict, dynamic introduction of [references] within [doc] is not allowed #35061

Closed

jbudz mentioned this issue Dec 18, 2019

Kibana no such index - Kibana re-adds index incorrectly. #53322

Closed

tylersmalley added Feature:Saved Objects Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc and removed Team:Operations Team label for Operations Team labels Mar 26, 2020

joshdover assigned rudolf Jan 14, 2021

rudolf mentioned this issue Feb 15, 2021

Enable v2 so migrations, disable in FTR tests #89297

Merged

9 tasks

rudolf closed this as completed Feb 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrations only run once #32237

Migrations only run once #32237

jbudz commented Feb 28, 2019 •

edited

Loading

elasticmachine commented Feb 28, 2019

kobelb commented Feb 28, 2019

jbudz commented Feb 28, 2019

kobelb commented Mar 1, 2019

guillain commented May 30, 2019

elasticmachine commented Mar 26, 2020

pgayvallet commented Mar 30, 2020

joshdover commented May 7, 2020

rudolf commented Jan 14, 2021

blankhang commented Jun 28, 2021 •

edited

Loading

blankhang commented Jun 28, 2021

blankhang commented Jun 28, 2021

blankhang commented Jun 28, 2021 •

edited

Loading

pgayvallet commented Jun 28, 2021

Migrations only run once #32237

Migrations only run once #32237

Comments

jbudz commented Feb 28, 2019 • edited Loading

elasticmachine commented Feb 28, 2019

kobelb commented Feb 28, 2019

jbudz commented Feb 28, 2019

kobelb commented Mar 1, 2019

guillain commented May 30, 2019

elasticmachine commented Mar 26, 2020

pgayvallet commented Mar 30, 2020

joshdover commented May 7, 2020

rudolf commented Jan 14, 2021

blankhang commented Jun 28, 2021 • edited Loading

blankhang commented Jun 28, 2021

blankhang commented Jun 28, 2021

blankhang commented Jun 28, 2021 • edited Loading

pgayvallet commented Jun 28, 2021

jbudz commented Feb 28, 2019 •

edited

Loading

blankhang commented Jun 28, 2021 •

edited

Loading

blankhang commented Jun 28, 2021 •

edited

Loading