Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test: appsearch/stats metricset integration and system tests #19739

Closed
ycombinator opened this issue Jul 8, 2020 · 9 comments · Fixed by #19744
Closed

Flaky test: appsearch/stats metricset integration and system tests #19739

ycombinator opened this issue Jul 8, 2020 · 9 comments · Fixed by #19744
Labels
flaky-test Unstable or unreliable test cases. Team:Services (Deprecated) Label for the former Integrations-Services team

Comments

@ycombinator
Copy link
Contributor

ycombinator commented Jul 8, 2020 •
edited
Loading

Flaky Test

The test occasionally times out while bringing up the Docker container for appsearch. This error from the stack trace of the system test (see below) seems relevant:

compose.service: ERROR: 404 Client Error: Not Found ("manifest for docker.elastic.co/integrations-ci/beats-appsearch:7.5.0-1 not found")

Stack Trace

Integration test

00:48:46  FAILURES:
00:48:46  Package: github.com/elastic/beats/v7/x-pack/metricbeat/module/appsearch/stats
00:48:46  Test:    TestFetch
00:48:46  /usr/local/go/src/testing/testing.go:1377 +0x11c
00:48:46  /usr/local/go/src/time/sleep.go:168 +0x52
00:48:46  /usr/local/go/src/testing/testing.go:961 +0x68a
00:48:46  /usr/local/go/src/testing/testing.go:1202 +0xa7
00:48:46  /usr/local/go/src/testing/testing.go:909 +0x19a
00:48:46  /usr/local/go/src/testing/testing.go:1200 +0x522
00:48:46  /usr/local/go/src/testing/testing.go:1117 +0x300
00:48:46  _testmain.go:96 +0x338
00:48:46  /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
00:48:46  /usr/local/go/src/os/wait_waitid.go:31 +0xbf
00:48:46  /usr/local/go/src/os/exec_unix.go:22 +0x6e
00:48:46  /usr/local/go/src/os/exec.go:125
00:48:46  /usr/local/go/src/os/exec/exec.go:506 +0xf3
00:48:46  /usr/local/go/src/os/exec/exec.go:342 +0x87
00:48:46  /go/src/github.com/elastic/beats/libbeat/tests/compose/wrapper.go:368 +0x155
00:48:46  /go/src/github.com/elastic/beats/libbeat/tests/compose/wrapper.go:315 +0x21d
00:48:46  /go/src/github.com/elastic/beats/libbeat/tests/compose/wrapper.go:278 +0x9d
00:48:46  /go/src/github.com/elastic/beats/libbeat/tests/compose/project.go:332 +0x141
00:48:46  /go/src/github.com/elastic/beats/libbeat/tests/compose/project.go:161 +0x1cf
00:48:46  /go/src/github.com/elastic/beats/libbeat/tests/compose/compose.go:83 +0x2cc
00:48:46  /go/src/github.com/elastic/beats/libbeat/tests/compose/compose.go:100 +0x50f
00:48:46  /go/src/github.com/elastic/beats/libbeat/tests/compose/compose.go:124 +0xa8
00:48:46  /go/src/github.com/elastic/beats/x-pack/metricbeat/module/appsearch/stats/stats_integration_test.go:19 +0x7e
00:48:46  /usr/local/go/src/testing/testing.go:909 +0x19a
00:48:46  /usr/local/go/src/testing/testing.go:960 +0x652
00:48:46  /usr/local/go/src/runtime/netpoll.go:184 +0x55
00:48:46  /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0xe4
00:48:46  /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
00:48:46  /usr/local/go/src/internal/poll/fd_unix.go:169 +0x253
00:48:46  /usr/local/go/src/net/fd_unix.go:202 +0x66
00:48:46  /usr/local/go/src/net/net.go:184 +0xa1
00:48:46  /usr/local/go/src/net/http/transport.go:1758 +0xe8
00:48:46  /usr/local/go/src/bufio/bufio.go:100 +0x19a
00:48:46  /usr/local/go/src/bufio/bufio.go:138 +0xa0
00:48:46  /usr/local/go/src/net/http/transport.go:1911 +0x2af
00:48:46  /usr/local/go/src/net/http/transport.go:1580 +0xbe2
00:48:46  /usr/local/go/src/net/http/transport.go:2210 +0x1c3
00:48:46  /usr/local/go/src/net/http/transport.go:1581 +0xc07
00:48:46  /usr/local/go/src/runtime/netpoll.go:184 +0x55
00:48:46  /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0xe4
00:48:46  /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
00:48:46  /usr/local/go/src/internal/poll/fd_unix.go:169 +0x253
00:48:46  /usr/local/go/src/os/file_unix.go:259
00:48:46  /usr/local/go/src/os/file.go:116 +0xa7
00:48:46  /usr/local/go/src/bytes/buffer.go:204 +0x159
00:48:46  /usr/local/go/src/io/io.go:388 +0x3fb
00:48:46  /usr/local/go/src/io/io.go:364
00:48:46  /usr/local/go/src/os/exec/exec.go:312 +0x7b
00:48:46  /usr/local/go/src/os/exec/exec.go:440 +0x35
00:48:46  /usr/local/go/src/os/exec/exec.go:439 +0xa9e
00:48:46  /usr/local/go/src/os/exec/exec.go:448 +0x13a
00:48:46  /usr/local/go/src/os/exec/exec.go:447 +0xb8d
00:48:46  ----
00:48:46  SUMMARY:
00:48:46    Fail:     1
00:48:46    Skip:     0
00:48:46    Pass:     0
00:48:46    Packages: 1
00:48:46    Duration: 10m20.016540644s
00:48:46    Coverage Report: /go/src/github.com/elastic/beats/x-pack/metricbeat/build/TEST-go-integration-appsearch.html
00:48:46    JUnit Report:    /go/src/github.com/elastic/beats/x-pack/metricbeat/build/TEST-go-integration-appsearch.xml
00:48:46    Output File:     /go/src/github.com/elastic/beats/x-pack/metricbeat/build/TEST-go-integration-appsearch.out
00:48:46  >> go test: Integration-appsearch Test Failed
00:48:46  Error: failed modules: appsearch

System test

======================================================================
ERROR: test suite for <class 'test_appsearch.Test'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/python-env/build/ve/docker/lib/python3.7/site-packages/nose/suite.py", line 210, in run
    self.setUp()
  File "/tmp/python-env/build/ve/docker/lib/python3.7/site-packages/nose/suite.py", line 293, in setUp
    self.setupContext(ancestor)
  File "/tmp/python-env/build/ve/docker/lib/python3.7/site-packages/nose/suite.py", line 316, in setupContext
    try_run(context, names)
  File "/tmp/python-env/build/ve/docker/lib/python3.7/site-packages/nose/util.py", line 471, in try_run
    return func()
  File "/go/src/github.com/elastic/beats/x-pack/metricbeat/tests/system/xpack_metricbeat.py", line 22, in setUpClass
    super(XPackTest, self).setUpClass()
  File "/go/src/github.com/elastic/beats/x-pack/metricbeat/tests/system/../../../../metricbeat/tests/system/metricbeat.py", line 31, in setUpClass
    super(BaseTest, self).setUpClass()
  File "/go/src/github.com/elastic/beats/libbeat/tests/system/beat/beat.py", line 151, in setUpClass
    self.compose_up_with_retries()
  File "/go/src/github.com/elastic/beats/libbeat/tests/system/beat/beat.py", line 166, in compose_up_with_retries
    raise e
  File "/go/src/github.com/elastic/beats/libbeat/tests/system/beat/beat.py", line 162, in compose_up_with_retries
    self.compose_up()
  File "/go/src/github.com/elastic/beats/libbeat/tests/system/beat/compose.py", line 78, in compose_up
    container.name_without_project)
Exception: Container appsearch_1 unexpectedly finished on startup
-------------------- >> begin captured logging << --------------------
compose.config.config: DEBUG: Using configuration files: /go/src/github.com/elastic/beats/x-pack/metricbeat/module/activemq/docker-compose.yml
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
compose.parallel: DEBUG: Pending: {<Container: activemq_1556499397b1afe2_activemq_1 (5d4efe)>}
compose.parallel: DEBUG: Starting producer thread for <Container: activemq_1556499397b1afe2_activemq_1 (5d4efe)>
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: <Container: activemq_1556499397b1afe2_activemq_1 (5d4efe)>
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: {<Container: activemq_1556499397b1afe2_activemq_1 (5d4efe)>}
compose.parallel: DEBUG: Starting producer thread for <Container: activemq_1556499397b1afe2_activemq_1 (5d4efe)>
compose.parallel: DEBUG: Finished processing: <Container: activemq_1556499397b1afe2_activemq_1 (5d4efe)>
compose.parallel: DEBUG: Pending: set()
compose.network: INFO: Removing network activemq_1556499397b1afe2_default
compose.config.config: DEBUG: Using configuration files: /go/src/github.com/elastic/beats/x-pack/metricbeat/module/appsearch/docker-compose.yml
compose.config.environment: WARNING: The PWD variable is not set. Defaulting to a blank string.
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
compose.service: INFO: Pulling appsearch (docker.elastic.co/integrations-ci/beats-appsearch:7.5.0-1)...
docker.auth: DEBUG: Looking for auth config
docker.auth: DEBUG: No auth config in memory - loading from filesystem
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
docker.auth: DEBUG: Looking for auth entry for 'docker.elastic.co'
docker.auth: DEBUG: No entry found
docker.auth: DEBUG: No auth config found
compose.service: ERROR: 404 Client Error: Not Found ("manifest for docker.elastic.co/integrations-ci/beats-appsearch:7.5.0-1 not found")
compose.network: INFO: Creating network "appsearch_7918a246f6f8_default" with the default driver
compose.service: INFO: Building appsearch
docker.api.build: DEBUG: Looking for auth config
docker.api.build: DEBUG: No auth config in memory - loading from filesystem
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
docker.api.build: DEBUG: Sending auth config ()
compose.service: WARNING: Image for service appsearch was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
compose.project: DEBUG: appsearch has upstream changes (elasticsearch)
compose.parallel: DEBUG: Pending: {<Service: elasticsearch>, <Service: appsearch>}
compose.parallel: DEBUG: Starting producer thread for <Service: elasticsearch>
compose.parallel: DEBUG: Pending: {ServiceName(project='appsearch_7918a246f6f8', service='elasticsearch', number=1)}
compose.parallel: DEBUG: Starting producer thread for ServiceName(project='appsearch_7918a246f6f8', service='elasticsearch', number=1)
compose.service: DEBUG: Added config hash: 488f6d9399e49f9be0f25c812c7041537a3e8aa7262ebe9848f33f94f0dd6736
compose.parallel: DEBUG: Pending: {<Service: appsearch>}
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: {<Service: appsearch>}
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: ServiceName(project='appsearch_7918a246f6f8', service='elasticsearch', number=1)
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: <Service: elasticsearch>
compose.parallel: DEBUG: Pending: {<Service: appsearch>}
compose.parallel: DEBUG: Starting producer thread for <Service: appsearch>
compose.parallel: DEBUG: Pending: {ServiceName(project='appsearch_7918a246f6f8', service='appsearch', number=1)}
compose.parallel: DEBUG: Starting producer thread for ServiceName(project='appsearch_7918a246f6f8', service='appsearch', number=1)
compose.service: DEBUG: Added config hash: 77be3b874a22f03f207e5dd4b69059f10a106d3b585a92d695131311b9280243
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: ServiceName(project='appsearch_7918a246f6f8', service='appsearch', number=1)
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: <Service: appsearch>
compose.parallel: DEBUG: Pending: set()
compose.config.config: DEBUG: Using configuration files: /go/src/github.com/elastic/beats/x-pack/metricbeat/module/appsearch/docker-compose.yml
compose.config.environment: WARNING: The PWD variable is not set. Defaulting to a blank string.
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
compose.parallel: DEBUG: Pending: {<Container: appsearch_7918a246f6f8_elasticsearch_1 (07260b)>}
compose.parallel: DEBUG: Starting producer thread for <Container: appsearch_7918a246f6f8_elasticsearch_1 (07260b)>
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: <Container: appsearch_7918a246f6f8_elasticsearch_1 (07260b)>
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: {<Container: appsearch_7918a246f6f8_appsearch_1 (a14948)>, <Container: appsearch_7918a246f6f8_elasticsearch_1 (07260b)>}
compose.parallel: DEBUG: Starting producer thread for <Container: appsearch_7918a246f6f8_appsearch_1 (a14948)>
compose.parallel: DEBUG: Starting producer thread for <Container: appsearch_7918a246f6f8_elasticsearch_1 (07260b)>
compose.parallel: DEBUG: Finished processing: <Container: appsearch_7918a246f6f8_appsearch_1 (a14948)>
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: <Container: appsearch_7918a246f6f8_elasticsearch_1 (07260b)>
compose.parallel: DEBUG: Pending: set()
compose.network: INFO: Removing network appsearch_7918a246f6f8_default
compose.config.config: DEBUG: Using configuration files: /go/src/github.com/elastic/beats/x-pack/metricbeat/module/appsearch/docker-compose.yml
compose.config.environment: WARNING: The PWD variable is not set. Defaulting to a blank string.
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
compose.service: INFO: Pulling appsearch (docker.elastic.co/integrations-ci/beats-appsearch:7.5.0-1)...
docker.auth: DEBUG: Looking for auth config
docker.auth: DEBUG: No auth config in memory - loading from filesystem
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
docker.auth: DEBUG: Looking for auth entry for 'docker.elastic.co'
docker.auth: DEBUG: No entry found
docker.auth: DEBUG: No auth config found
compose.service: ERROR: 404 Client Error: Not Found ("manifest for docker.elastic.co/integrations-ci/beats-appsearch:7.5.0-1 not found")
compose.network: INFO: Creating network "appsearch_7918a246f6f8_default" with the default driver
compose.project: DEBUG: appsearch has upstream changes (elasticsearch)
compose.parallel: DEBUG: Pending: {<Service: appsearch>, <Service: elasticsearch>}
compose.parallel: DEBUG: Starting producer thread for <Service: elasticsearch>
compose.parallel: DEBUG: Pending: {ServiceName(project='appsearch_7918a246f6f8', service='elasticsearch', number=1)}
compose.parallel: DEBUG: Starting producer thread for ServiceName(project='appsearch_7918a246f6f8', service='elasticsearch', number=1)
compose.service: DEBUG: Added config hash: 488f6d9399e49f9be0f25c812c7041537a3e8aa7262ebe9848f33f94f0dd6736
compose.parallel: DEBUG: Pending: {<Service: appsearch>}
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: {<Service: appsearch>}
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: {<Service: appsearch>}
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: ServiceName(project='appsearch_7918a246f6f8', service='elasticsearch', number=1)
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: <Service: elasticsearch>
compose.parallel: DEBUG: Pending: {<Service: appsearch>}
compose.parallel: DEBUG: Starting producer thread for <Service: appsearch>
compose.parallel: DEBUG: Pending: {ServiceName(project='appsearch_7918a246f6f8', service='appsearch', number=1)}
compose.parallel: DEBUG: Starting producer thread for ServiceName(project='appsearch_7918a246f6f8', service='appsearch', number=1)
compose.service: DEBUG: Added config hash: 77be3b874a22f03f207e5dd4b69059f10a106d3b585a92d695131311b9280243
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: ServiceName(project='appsearch_7918a246f6f8', service='appsearch', number=1)
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: <Service: appsearch>
compose.parallel: DEBUG: Pending: set()
compose.config.config: DEBUG: Using configuration files: /go/src/github.com/elastic/beats/x-pack/metricbeat/module/appsearch/docker-compose.yml
compose.config.environment: WARNING: The PWD variable is not set. Defaulting to a blank string.
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
compose.parallel: DEBUG: Pending: {<Container: appsearch_7918a246f6f8_elasticsearch_1 (b5666c)>}
compose.parallel: DEBUG: Starting producer thread for <Container: appsearch_7918a246f6f8_elasticsearch_1 (b5666c)>
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: <Container: appsearch_7918a246f6f8_elasticsearch_1 (b5666c)>
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: {<Container: appsearch_7918a246f6f8_elasticsearch_1 (b5666c)>, <Container: appsearch_7918a246f6f8_appsearch_1 (c65324)>}
compose.parallel: DEBUG: Starting producer thread for <Container: appsearch_7918a246f6f8_elasticsearch_1 (b5666c)>
compose.parallel: DEBUG: Starting producer thread for <Container: appsearch_7918a246f6f8_appsearch_1 (c65324)>
compose.parallel: DEBUG: Finished processing: <Container: appsearch_7918a246f6f8_appsearch_1 (c65324)>
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: <Container: appsearch_7918a246f6f8_elasticsearch_1 (b5666c)>
compose.parallel: DEBUG: Pending: set()
compose.network: INFO: Removing network appsearch_7918a246f6f8_default
compose.config.config: DEBUG: Using configuration files: /go/src/github.com/elastic/beats/x-pack/metricbeat/module/appsearch/docker-compose.yml
compose.config.environment: WARNING: The PWD variable is not set. Defaulting to a blank string.
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
compose.service: INFO: Pulling appsearch (docker.elastic.co/integrations-ci/beats-appsearch:7.5.0-1)...
docker.auth: DEBUG: Looking for auth config
docker.auth: DEBUG: No auth config in memory - loading from filesystem
docker.utils.config: DEBUG: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
docker.utils.config: DEBUG: No config file found
docker.auth: DEBUG: Looking for auth entry for 'docker.elastic.co'
docker.auth: DEBUG: No entry found
docker.auth: DEBUG: No auth config found
compose.service: ERROR: 404 Client Error: Not Found ("manifest for docker.elastic.co/integrations-ci/beats-appsearch:7.5.0-1 not found")
compose.network: INFO: Creating network "appsearch_7918a246f6f8_default" with the default driver
compose.project: DEBUG: appsearch has upstream changes (elasticsearch)
compose.parallel: DEBUG: Pending: {<Service: elasticsearch>, <Service: appsearch>}
compose.parallel: DEBUG: Starting producer thread for <Service: elasticsearch>
compose.parallel: DEBUG: Pending: {ServiceName(project='appsearch_7918a246f6f8', service='elasticsearch', number=1)}
compose.parallel: DEBUG: Starting producer thread for ServiceName(project='appsearch_7918a246f6f8', service='elasticsearch', number=1)
compose.service: DEBUG: Added config hash: 488f6d9399e49f9be0f25c812c7041537a3e8aa7262ebe9848f33f94f0dd6736
compose.parallel: DEBUG: Pending: {<Service: appsearch>}
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: {<Service: appsearch>}
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: ServiceName(project='appsearch_7918a246f6f8', service='elasticsearch', number=1)
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: <Service: elasticsearch>
compose.parallel: DEBUG: Pending: {<Service: appsearch>}
compose.parallel: DEBUG: Starting producer thread for <Service: appsearch>
compose.parallel: DEBUG: Pending: {ServiceName(project='appsearch_7918a246f6f8', service='appsearch', number=1)}
compose.parallel: DEBUG: Starting producer thread for ServiceName(project='appsearch_7918a246f6f8', service='appsearch', number=1)
compose.service: DEBUG: Added config hash: 77be3b874a22f03f207e5dd4b69059f10a106d3b585a92d695131311b9280243
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: ServiceName(project='appsearch_7918a246f6f8', service='appsearch', number=1)
compose.parallel: DEBUG: Pending: set()
compose.parallel: DEBUG: Finished processing: <Service: appsearch>
compose.parallel: DEBUG: Pending: set()
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: /go/src/github.com/elastic/beats/x-pack/metricbeat/build/TEST-python-integration.xml
[error] 52.79% <nose.suite.ContextSuite context=Test>:setup: 137.1124s
@ycombinator ycombinator added the flaky-test Unstable or unreliable test cases. label Jul 8, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 8, 2020
@ycombinator ycombinator added the Team:Services (Deprecated) Label for the former Integrations-Services team label Jul 8, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 8, 2020
@ycombinator ycombinator mentioned this issue Jul 8, 2020
Closed
@ycombinator ycombinator changed the title Flaky test: appsearch/stats metricset golang integration test Flaky test: appsearch/stats metricset integration and system tests Jul 8, 2020
@ycombinator
Copy link
Contributor Author

Indeed, I tried pulling the docker image used by the tests from the integrations-ci docker registry locally and it failed with the same error as reported in the issue description.

» docker pull docker.elastic.co/integrations-ci/beats-appsearch:7.5.0-1
Error response from daemon: manifest for docker.elastic.co/integrations-ci/beats-appsearch:7.5.0-1 not found: manifest unknown: manifest unknown

If I try to build the docker image for the appsearch service locally, it succeeds.

» docker-compose build appsearch
Building appsearch
Step 1/5 : ARG APPSEARCH_VERSION
Step 2/5 : FROM docker.elastic.co/app-search/app-search:${APPSEARCH_VERSION}
7.5.0: Pulling from app-search/app-search
c808caf183b6: Pull complete
ec90c1d07b58: Pull complete
0df893e87de7: Pull complete
4032a87eaf94: Pull complete
9177b9312f7c: Pull complete
48989071d63b: Pull complete
Digest: sha256:922e56ef1534dcfb85d422159c61fa913456b71c33eaa521a1decaf05b4e21c9
Status: Downloaded newer image for docker.elastic.co/app-search/app-search:7.5.0
 ---> 9d25727461da
Step 3/5 : COPY docker-entrypoint-dependencies.sh /usr/local/bin/
 ---> 1e5a8fb515f4
Step 4/5 : ENTRYPOINT /usr/local/bin/docker-entrypoint-dependencies.sh
 ---> Running in 69edd56485f2
Removing intermediate container 69edd56485f2
 ---> c57ac1740342
Step 5/5 : HEALTHCHECK --interval=1s --retries=300 --start-period=60s CMD python -c 'import urllib, json; response = urllib.urlopen("http://myelastic:changeme@localhost:3002/api/as/v1/internal/health"); response.getcode() == 200 or exit(1); data = json.loads(response.read()); exit(0);'
 ---> Running in 968ac571fecf
Removing intermediate container 968ac571fecf
 ---> 6db3d6587160

Successfully built 6db3d6587160
Successfully tagged docker.elastic.co/integrations-ci/beats-appsearch:7.5.0-1

@jsoriano any ideas what might be going on? Are you the person to ping about the integrations-ci docker registry or should I be pinging someone else in the future?

@jsoriano
Copy link
Member

jsoriano commented Jul 8, 2020

@ycombinator it is ok if you ping me about these problems 🙂

compose logging is a bit too verbose and confusing.

We always try to pull the image before trying to build it, so if it is available in the registry we use it, and if not, we build it. For all the images we don't have available, the error about not found manifest is going to appear, but it is ignored, and then the image is built.

Actually, it did build the image at the end:

compose.service: WARNING: Image for service appsearch was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.

This error is more significative:

Exception: Container appsearch_1 unexpectedly finished on startup

Container finished on startup. This uses to mean that the container was actually started (thus the image was there), but stopped before reaching a healthy state. This may be caused by some lack of resources or some other source of flakiness in the image or service startup.

@ycombinator
Copy link
Contributor Author

ycombinator commented Jul 8, 2020 •
edited
Loading

Ah, thanks @jsoriano I didn't realize I could ignore the error from the registry so I didn't look further — thanks for clarifying this!

Yes, we have had issues with this container taking too long to start up in the past. That's why we added a very long timeout here:

beats/x-pack/metricbeat/module/appsearch/stats/stats_integration_test.go

Line 19 in a00eaba

service := compose.EnsureUpWithTimeout(t, 570, "appsearch")

But evidently that is not enough time in CI or something else is more fundamentally broken when the container tries to start up. So now we need to figure out what's going on when the container tries to start up.

@jsoriano
Copy link
Member

jsoriano commented Jul 8, 2020

Yes, this service uses to take some time to start. But take into account that the error in this case means that the container stopped, so it wouldn't have gone to a healthy state no matter how long we would have waited.

@ycombinator
Copy link
Contributor Author

Good point. So something is more fundamentally broken during this container's start up then. It's not just taking too long to start up. Makes sense, thanks.

@jsoriano
Copy link
Member

jsoriano commented Jul 8, 2020

Take into account that the container may be "logically" fine, but get killed by OOM for example.

@ycombinator
Copy link
Contributor Author

Here is the issue, from the logs of the appsearch container:

appsearch_1      | App Search is starting...
appsearch_1      |
appsearch_1      | scripting container class loader urls: [file:/tmp/jruby9107565378189885387extract/lib/jruby-stdlib-9.2.9.0.jar, file:/tmp/jruby9107565378189885387extract/lib/jetty-util.jar, file:/tmp/jruby9107565378189885387extract/lib/jruby-rack-1.1.21.jar, file:/tmp/jruby9107565378189885387extract/lib/jetty-servlets.jar, file:/tmp/jruby9107565378189885387extract/lib/jruby-core-9.2.9.0-complete.jar, file:/tmp/jruby9107565378189885387extract/lib/jruby-rack-worker_0.14.1.jar]
appsearch_1      | setting GEM_HOME to /tmp/jruby9107565378189885387extract/gems
appsearch_1      | ... and BUNDLE_GEMFILE to /tmp/jruby9107565378189885387extract/Gemfile
appsearch_1      | loading resource: /tmp/jruby9107565378189885387extract/./META-INF/rails.rb
appsearch_1      | invoking /tmp/jruby9107565378189885387extract/./META-INF/rails.rb with: [runner, LocoTogo.start_app_server!]
appsearch_1      | [2020-07-08T15:24:30.423+00:00][228][2002][app-server][ERROR]:
appsearch_1      | --------------------------------------------------------------------------------
appsearch_1      |
appsearch_1      | Unsupported Elasticsearch version: 7.7.0. App Search requires Elasticsearch >= 7.2 and < 7.6
appsearch_1      |
appsearch_1      | --------------------------------------------------------------------------------
appsearch_1      |
appsearch_1      |
appsearch_1      | ERROR: org.jruby.exceptions.SystemExit: (SystemExit) exit
appsearch_1      | org.jruby.exceptions.SystemExit: (SystemExit) exit
appsearch_1      |      at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:751)
appsearch_1      |      at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:714)
appsearch_1      |      at RUBY.fatal_error(/tmp/jruby9107565378189885387extract/shared_togo/lib/shared_togo.class:738)
appsearch_1      |      at RUBY.configure!(/tmp/jruby9107565378189885387extract/shared_togo/lib/shared_togo.class:165)
appsearch_1      |      at RUBY.configure!(/tmp/jruby9107565378189885387extract/shared_togo/lib/shared_togo.class:32)
appsearch_1      |      at RUBY.<main>(/tmp/jruby9107565378189885387extract/config/application.class:24)
appsearch_1      |      at org.jruby.RubyKernel.load(org/jruby/RubyKernel.java:1013)
appsearch_1      |      at RUBY.<main>(/tmp/jruby9107565378189885387extract/config/application.rb:1)
appsearch_1      |      at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:978)
appsearch_1      |      at RUBY.<main>(/tmp/jruby9107565378189885387extract/gems/gems/railties-4.2.11.1/lib/rails/commands/runner.rb:51)
appsearch_1      |      at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:978)
appsearch_1      |      at RUBY.require_command!(/tmp/jruby9107565378189885387extract/gems/gems/railties-4.2.11.1/lib/rails/commands/commands_tasks.rb:123)
appsearch_1      |      at RUBY.runner(/tmp/jruby9107565378189885387extract/gems/gems/railties-4.2.11.1/lib/rails/commands/commands_tasks.rb:90)
appsearch_1      |      at RUBY.run_command!(/tmp/jruby9107565378189885387extract/gems/gems/railties-4.2.11.1/lib/rails/commands/commands_tasks.rb:39)
appsearch_1      |      at RUBY.<main>(/tmp/jruby9107565378189885387extract/gems/gems/railties-4.2.11.1/lib/rails/commands.rb:17)
appsearch_1      |      at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:978)
appsearch_1      |      at tmp.jruby9107565378189885387extract.$_dot_.META_minus_INF.rails.<main>(/tmp/jruby9107565378189885387extract/./META-INF/rails.rb:7)
appsearch_appsearch_1 exited with code 1

@ycombinator
Copy link
Contributor Author

ycombinator commented Jul 8, 2020 •
edited
Loading

Bumping up the appsearch image version to 7.7.0 (same as the Elasticsearch one) did not work.

docker-compose up
Creating network "appsearch_default" with the default driver
Building elasticsearch
Step 1/3 : ARG ELASTICSEARCH_VERSION
Step 2/3 : FROM docker.elastic.co/elasticsearch/elasticsearch:${ELASTICSEARCH_VERSION}
7.7.0: Pulling from elasticsearch/elasticsearch
86dbb57a3083: Pull complete
2b5d7fba0568: Pull complete
441e4ff55d6b: Pull complete
d5bae169186d: Pull complete
0bd94a0e499c: Pull complete
0459ad1587bf: Pull complete
1ccf419e016d: Pull complete
4cc72f3ea33a: Pull complete
74d1ead8433b: Pull complete
Digest: sha256:b4ec018c974d23a796b1d17ddec3863e4e6deff76d25f58027c45b7a697089dc
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.7.0
 ---> 7ec4f35ab452
Step 3/3 : HEALTHCHECK --interval=1s --retries=300 CMD curl -f http://localhost:9200/_license
 ---> Running in 19e5dfbef4c3
Removing intermediate container 19e5dfbef4c3
 ---> 1b8aa8a31070

Successfully built 1b8aa8a31070
Successfully tagged docker.elastic.co/integrations-ci/beats-elasticsearch:7.7.0-1
WARNING: Image for service elasticsearch was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Building appsearch
Step 1/5 : ARG APPSEARCH_VERSION
Step 2/5 : FROM docker.elastic.co/app-search/app-search:${APPSEARCH_VERSION}
ERROR: Service 'appsearch' failed to build: manifest for docker.elastic.co/app-search/app-search:7.7.0 not found: manifest unknown: manifest unknown

However, bumping it up to 7.6.2 worked, so I'll put up a PR with this change.

@ioanatia any ideas why there is no 7.7.0 Docker image for app search (docker.elastic.co/app-search/app-search:7.7.0 )? Maybe we have to pull a different image now?

@ycombinator ycombinator mentioned this issue Jul 8, 2020
Merged
This was referenced Aug 3, 2020
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
flaky-test Unstable or unreliable test cases. Team:Services (Deprecated) Label for the former Integrations-Services team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Bumping up app search image version to 7.6.2 ycombinator/beats
3 participants
@jsoriano @ycombinator @elasticmachine