[7.x] [server/logging/tests] use --oss to load kibana faster and avoid timeouts (#32162)

[spalger]

Backports the following commits to 7.x:

• [server/logging/tests] use --oss to load kibana faster and avoid timeouts ([server/logging/tests] use --oss to load kibana faster and avoid timeout #32162)

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[00:00:27]             └-> "before all" hook
[00:00:27]               │ info [logstash/example_pipelines] Loading "mappings.json"
[00:00:27]               │ info [logstash/example_pipelines] Loading "data.json.gz"
[00:00:27]               │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-debian-1551310512087769495] [.kibana_2/U6KsuzF9TESM5FE95iVxHQ] deleting index
[00:00:27]               │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-debian-1551310512087769495] [.kibana_1/GX8-9zICSHOB57oCSvHTYw] deleting index
[00:00:27]               │ info [logstash/example_pipelines] Deleted existing index [".kibana_2",".kibana_1"]
[00:00:27]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-debian-1551310512087769495] [.kibana] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[00:00:27]               │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-debian-1551310512087769495] [.kibana/Vv_Jt9ZwTK6JDCim-0aDvQ] deleting index
[00:00:27]               │ info [o.e.i.IndicesService] [kibana-ci-immutable-debian-1551310512087769495] [.kibana/Vv_Jt9ZwTK6JDCim-0aDvQ] failed to delete index
[00:00:27]               │      org.elasticsearch.env.ShardLockObtainFailedException: [.kibana][0]: obtaining shard lock timed out after 0ms
[00:00:27]               │      	at org.elasticsearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:736) ~[elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.env.NodeEnvironment.shardLock(NodeEnvironment.java:655) ~[elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.env.NodeEnvironment.lockAllForIndex(NodeEnvironment.java:601) ~[elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectorySafe(NodeEnvironment.java:554) ~[elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.indices.IndicesService.deleteIndexStoreIfDeletionAllowed(IndicesService.java:792) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.indices.IndicesService.deleteIndexStore(IndicesService.java:779) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:683) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:324) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:258) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$5(ClusterApplierService.java:472) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at java.lang.Iterable.forEach(Iterable.java:75) [?:?]
[00:00:27]               │      	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:470) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:459) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:413) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:164) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.1.0-SNAPSHOT.jar:7.1.0-SNAPSHOT]
[00:00:27]               │      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
[00:00:27]               │      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
[00:00:27]               │      	at java.lang.Thread.run(Thread.java:834) [?:?]
[00:00:27]               │ proc [kibana]   log   [00:00:01.344] [warning] Error loading maps telemetry: Error: Automatic index creation failed
[00:00:27]               │ info [logstash/example_pipelines] Deleted existing index ".kibana"
[00:00:27]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-debian-1551310512087769495] [.kibana] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:00:28]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-debian-1551310512087769495] [.kibana/wESHJwo7RHekqI4SVemAyQ] update_mapping [_doc]
[00:00:28]               │ info [logstash/example_pipelines] Created index ".kibana"
[00:00:28]               │ debg [logstash/example_pipelines] ".kibana" settings {"index":{"number_of_replicas":"1","number_of_shards":"1"}}
[00:00:28]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-debian-1551310512087769495] [.logstash] creating index, cause [auto(bulk api)], templates [.logstash-management], shards [1]/[1], mappings [doc]
[00:00:28]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-1551310512087769495] updating number_of_replicas to [0] for indices [.logstash]
[00:00:28]               │ info [logstash/example_pipelines] Indexed 22 docs into ".logstash"
[00:00:28]               │ info [migrations] Creating index .kibana_2.
[00:00:28]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-debian-1551310512087769495] [.kibana_2] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:00:28]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-1551310512087769495] updating number_of_replicas to [0] for indices [.kibana_2]
[00:00:28]               │ info [migrations] Reindexing .kibana to .kibana_1
[00:00:28]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-debian-1551310512087769495] [.kibana_1] creating index, cause [api], templates [], shards [1]/[1], mappings [_doc]
[00:00:28]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-1551310512087769495] updating number_of_replicas to [0] for indices [.kibana_1]
[00:00:28]               │ info [o.e.c.m.MetaDataCreateIndexService] [kibana-ci-immutable-debian-1551310512087769495] [.tasks] creating index, cause [auto(task api)], templates [], shards [1]/[1], mappings [task]
[00:00:28]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-1551310512087769495] updating number_of_replicas to [0] for indices [.tasks]
[00:00:29]               │ info [o.e.t.LoggingTaskListener] [kibana-ci-immutable-debian-1551310512087769495] 1025 finished with response BulkByScrollResponse[took=8.4ms,timed_out=false,sliceId=null,updated=0,created=0,deleted=0,batches=0,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[00:00:29]               │ info [o.e.c.m.MetaDataDeleteIndexService] [kibana-ci-immutable-debian-1551310512087769495] [.kibana/wESHJwo7RHekqI4SVemAyQ] deleting index
[00:00:29]               │ info [migrations] Migrating .kibana_1 saved objects to .kibana_2
[00:00:29]               │ info [migrations] Pointing alias .kibana to .kibana_2.
[00:00:29]               │ info [migrations] Finished in 523ms.
[00:00:29]               │ debg applying update to kibana config: {"accessibility:disableAnimations":true}
[00:00:29]               │ info [o.e.c.m.MetaDataMappingService] [kibana-ci-immutable-debian-1551310512087769495] [.kibana_2/jeJ9nmRyRWGC6sPirDab1g] update_mapping [_doc]
[00:00:30]               │ debg navigating to logstashPipelines url: http://localhost:5620/app/kibana#/management/logstash/pipelines
[00:00:30]               │ debg navigate to: http://localhost:5620/app/kibana#/management/logstash/pipelines
[00:00:30]               │ proc [kibana]   log   [00:00:04.305] [error][spaces] Unable to navigate to space "default", redirecting to Space Selector. Error: Saved object [space/default] not found
[00:00:31]               │ debg ... sleep(700) start
[00:00:31]               │ debg ... sleep(700) end
[00:00:31]               │ debg returned from get, calling refresh
[00:00:31]               │ debg currentUrl = http://localhost:5620/#/management/logstash/pipelines
[00:00:31]               │          appUrl = http://localhost:5620/app/kibana#/management/logstash/pipelines
[00:00:31]               │ debg findByCssSelector [data-test-subj="kibanaChrome"]
[00:04:31]               │ debg --- retry.try error: [POST http://localhost:9515/session/51e27987b4ed8fc48453cc8d87f3b8e1/element / {"using":"css selector","value":"[data-test-subj=\"kibanaChrome\"]"}] no such element: Unable to locate element: {"method":"css selector","selector":"[data-test-subj="kibanaChrome"]"}
[00:04:31]               │        (Session info: headless chrome=72.0.3626.119)
[00:04:31]               │        (Driver info: chromedriver=2.42.591071 (0b695ff80972cc1a65a5cd643186d2ae582cd4ac),platform=Linux 4.9.0-8-amd64 x86_64)
[00:04:32]               │ debg --- retry.try error: retry.try timeout: [POST http://localhost:9515/session/51e27987b4ed8fc48453cc8d87f3b8e1/element / {"using":"css selector","value":"[data-test-subj=\"kibanaChrome\"]"}] no such element: Unable to locate element: {"method":"css selector","selector":"[data-test-subj="kibanaChrome"]"}
[00:04:32]               │        (Session info: headless chrome=72.0.3626.119)
...

Looks like Elasticsearch failed to delete an index... we might need to implement a retry in the esArchiver.

[spalger]

retest

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request