1000 shards limit in kibana 7.0.0

[fenchu]

Kibana version: 7.0.0

Elasticsearch version: 7.0.0

Server OS version: Linux Debian 4.9.88-1+deb9u1

Browser version: NA

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.):
installed via docker-compose

Describe the bug:
Had to delete everythin up to latest 1 week, this worked fine in kibana6.7.0

Steps to reproduce:

1. just upgraded docker from 6.7.0 to 7.0.0 and kibana refuses to start because of too many shards open (limit seem to be 1000)

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.0.0
docker pull docker.elastic.co/kibana/kibana:7.0.0

2. deleted a lots of shards

Expected behavior:

Screenshots (if relevant):

Errors in browser console (if relevant):
FATAL [index_creation_exception] failed to create index [.kibana_3] :: {"path":"/.kibana_3","query":{},"body":"{"mappings":{"dynamic":"strict","properties":{"apm-telemetry":{"properties":{"has_any_services":{"type":"boolean"},"services_per_agent":{"properties":{"python":{"type":"long","null_value":0},"java":{"type":"long","null_value":0},"nodejs":{"type":"long","null_value":0},"js-base":{"type":"long","null_value":0},"ruby":{"type":"long","null_value":0},"go":{"type":"long","null_value":0}}}}},"canvas-workpad":{"dynamic":false,"properties":{"name":{"type":"text","fields":{"keyword":{"type":"keyword"}}},"@timestamp":{"type":"date"},"@created":{"type":"date"}}},"config":{"dynamic":"true","properties":{"buildNum":{"type":"keyword"}}},"dashboard":{"properties":{"description":{"type":"text"},"hits":{"type":"integer"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text"}}},"optionsJSON":{"type":"text"},"panelsJSON":{"type":"text"},"refreshInterval":{"properties":{"display":{"type":"keyword"},"pause":{"type":"boolean"},"section":{"type":"integer"},"value":{"type":"integer"}}},"timeFrom":{"type":"keyword"},"timeRestore":{"type":"boolean"},"timeTo":{"type":"keyword"},"title":{"type":"text"},"uiStateJSON":{"type":"text"},"version":{"type":"integer"}}},"graph-workspace":{"properties":{"description":{"type":"text"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text"}}},"numLinks":{"type":"integer"},"numVertices":{"type":"integer"},"title":{"type":"text"},"version":{"type":"integer"},"wsState":{"type":"text"}}},"index-pattern":{"properties":{"fieldFormatMap":{"type":"text"},"fields":{"type":"text"},"intervalName":{"type":"keyword"},"notExpandable":{"type":"boolean"},"sourceFilters":{"type":"text"},"timeFieldName":{"type":"keyword"},"title":{"type":"text"},"type":{"type":"keyword"},"typeMeta":{"type":"keyword"}}},"infrastructure-ui-source":{"properties":{"name":{"type":"text"},"description":{"type":"text"},"metricAlias":{"type":"keyword"},"logAlias":{"type":"keyword"},"fields":{"properties":{"container":{"type":"keyword"},"host":{"type":"keyword"},"pod":{"type":"keyword"},"tiebreaker":{"type":"keyword"},"timestamp":{"type":"keyword"}}}}},"kql-telemetry":{"properties":{"optInCount":{"type":"long"},"optOutCount":{"type":"long"}}},"map":{"properties":{"description":{"type":"text"},"title":{"type":"text"},"version":{"type":"integer"},"bounds":{"type":"geo_shape"},"mapStateJSON":{"type":"text"},"layerListJSON":{"type":"text"},"uiStateJSON":{"type":"text"}}},"maps-telemetry":{"properties":{"mapsTotalCount":{"type":"long"},"timeCaptured":{"type":"date"},"attributesPerMap":{"properties":{"dataSourcesCount":{"properties":{"min":{"type":"long"},"max":{"type":"long"},"avg":{"type":"long"}}},"layersCount":{"properties":{"min":{"type":"long"},"max":{"type":"long"},"avg":{"type":"long"}}},"layerTypesCount":{"dynamic":"true","properties":{}},"emsVectorLayersCount":{"dynamic":"true","properties":{}}}}}},"migrationVersion":{"dynamic":"true","type":"object"},"ml-telemetry":{"properties":{"file_data_visualizer":{"properties":{"index_creation_count":{"type":"long"}}}}},"namespace":{"type":"keyword"},"search":{"properties":{"columns":{"type":"keyword"},"description":{"type":"text"},"hits":{"type":"integer"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text"}}},"sort":{"type":"keyword"},"title":{"type":"text"},"version":{"type":"integer"}}},"server":{"properties":{"uuid":{"type":"keyword"}}},"space":{"properties":{"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":2048}}},"description":{"type":"text"},"initials":{"type":"keyword"},"color":{"type":"keyword"},"_reserved":{"type":"boolean"}}},"telemetry":{"properties":{"enabled":{"type":"boolean"}}},"timelion-sheet":{"properties":{"description":{"type":"text"},"hits":{"type":"integer"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text"}}},"timelion_chart_height":{"type":"integer"},"timelion_columns":{"type":"integer"},"timelion_interval":{"type":"keyword"},"timelion_other_interval":{"type":"keyword"},"timelion_rows":{"type":"integer"},"timelion_sheet":{"type":"text"},"title":{"type":"text"},"version":{"type":"integer"}}},"type":{"type":"keyword"},"updated_at":{"type":"date"},"upgrade-assistant-reindex-operation":{"dynamic":true,"properties":{"indexName":{"type":"keyword"},"status":{"type":"integer"}}},"upgrade-assistant-telemetry":{"properties":{"ui_open":{"properties":{"overview":{"type":"long","null_value":0},"cluster":{"type":"long","null_value":0},"indices":{"type":"long","null_value":0}}},"ui_reindex":{"properties":{"close":{"type":"long","null_value":0},"open":{"type":"long","null_value":0},"start":{"type":"long","null_value":0},"stop":{"type":"long","null_value":0}}},"features":{"properties":{"deprecation_logging":{"properties":{"enabled":{"type":"boolean","null_value":true}}}}}}},"url":{"properties":{"accessCount":{"type":"long"},"accessDate":{"type":"date"},"createDate":{"type":"date"},"url":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":2048}}}}},"visualization":{"properties":{"description":{"type":"text"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"text"}}},"savedSearchRefName":{"type":"keyword"},"title":{"type":"text"},"uiStateJSON":{"type":"text"},"version":{"type":"integer"},"visState":{"type":"text"}}},"references":{"type":"nested","properties":{"name":{"type":"keyword"},"type":{"type":"keyword"},"id":{"type":"keyword"}}}},"_meta":{"migrationMappingPropertyHashes":{"config":"87aca8fdb053154f11383fce3dbf3edf","migrationVersion":"4a1746014a75ade3a714e1db5763276f","type":"2f4316de49999235636386fe51dc06c1","namespace":"2f4316de49999235636386fe51dc06c1","updated_at":"00da57df13e94e9d98437d13ace4bfe0","references":"7997cf5a56cc02bdc9c93361bde732b0","telemetry":"e1c8bc94e443aefd9458932cc0697a4d","graph-workspace":"cd7ba1330e6682e9cc00b78850874be1","space":"0d5011d73a0ef2f0f615bb42f26f187e","ml-telemetry":"257fd1d4b4fdbb9cb4b8a3b27da201e9","apm-telemetry":"a8adf824cc1b83d3bf4539a036a14462","map":"23d7aa4a720d4938ccde3983f87bd58d","maps-telemetry":"a4229f8b16a6820c6d724b7e0c1f729d","canvas-workpad":"b0a1706d356228dbdcb4a17e6b9eb231","infrastructure-ui-source":"10acdf67d9a06d462e198282fd6d4b81","upgrade-assistant-reindex-operation":"a53a20fe086b72c9a86da3cc12dad8a6","upgrade-assistant-telemetry":"56702cec857e0a9dacfb696655b4ff7b","index-pattern":"66eccb05066c5a89924f48a9e9736499","visualization":"52d7a13ad68a150c4525b292d23e12cc","search":"181661168bbadd1eff5902361e2a0d5c","dashboard":"eb3789e1af878e73f85304333240f65f","url":"c7f66a0df8b1b52f17c28c4adb111105","server":"ec97f1c5da1a19609a60874e5af1100c","kql-telemetry":"d12a98a6f19a2d273696597547e064ee","timelion-sheet":"9a2a2748877c7a7b582fef201ab1d4cf"}}},"settings":{"number_of_shards":1,"auto_expand_replicas":"0-1"}}","statusCode":400,"response":"{"error":{"root_cause":[{"type":"index_creation_exception","reason":"failed to create index [.kibana_3]"}],"type":"validation_exception","reason":"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1686]/[1000] maximum shards open;"},"status":400}"}
Provide logs and/or server output (if relevant):

Any additional context:

[tylersmalley]

Starting in 7.0 there is a default soft limit of 1000 shards per node in the cluster. You can read more about this limit here. Do note, that this limit can be adjusted with the cluster setting cluster.max_shards_per_node. Having too many shards open can definitely lead to performance issues and I would suggest analyzing your situation.

Additional information:

How many shards should I have in my Elasticsearch cluster?

Resizing Elasticsearch shards for fun and profit