Debug: get agent data #464

mtojek · 2021-08-04T06:50:53Z

This PR adds more logging to get additional agent data.

Introduced to help debugging the issue: https://github.com/elastic/kibana/issues/107514

elasticmachine · 2021-08-04T07:02:58Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-04T06:51:07.218+0000
Duration: 12 min 37 sec
Commit: 734ee73

Trends 🧪

Steps errors

Expand to view the steps failures

`Check`

Took 9 min 11 sec . View more details on here
Description: make check

Log output

Expand to view the last 100 lines of log output

[2021-08-04T07:02:15.560Z] 2021/08/04 07:02:15 DEBUG reassigning original policy back to agent...
[2021-08-04T07:02:15.560Z] 2021/08/04 07:02:15 DEBUG PUT http://127.0.0.1:5601/api/fleet/agents/96974b38-2cf9-46c7-ac92-124883ca54d2/reassign
[2021-08-04T07:02:16.940Z] Error: error running package system tests: could not complete test run: could not assign policy to agent: error occurred while waiting for the policy to be assigned to all agents: timeout: policy hasn't been assigned in time
[2021-08-04T07:02:16.941Z] + cleanup
[2021-08-04T07:02:16.941Z] + r=1
[2021-08-04T07:02:16.941Z] + elastic-package stack dump -v --output build/elastic-stack-dump/check
[2021-08-04T07:02:16.941Z] 2021/08/04 07:02:16 DEBUG Enable verbose logging
[2021-08-04T07:02:16.941Z] 2021/08/04 07:02:16 DEBUG Distribution built without a version tag, can't determine release chronology. Please consider using official releases at https://github.com/elastic/elastic-package/releases
[2021-08-04T07:02:16.941Z] 2021/08/04 07:02:16 DEBUG Dump Elastic stack data
[2021-08-04T07:02:16.941Z] 2021/08/04 07:02:16 DEBUG Dump stack logs
[2021-08-04T07:02:16.941Z] 2021/08/04 07:02:16 DEBUG Recreate the output location (path: build/elastic-stack-dump/check)
[2021-08-04T07:02:16.941Z] 2021/08/04 07:02:16 DEBUG Dump stack logs for elasticsearch
[2021-08-04T07:02:16.941Z] 2021/08/04 07:02:16 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/t-manager_elastic-package_PR-464/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs elasticsearch
[2021-08-04T07:02:17.880Z] 2021/08/04 07:02:17 DEBUG Dump stack logs for elastic-agent
[2021-08-04T07:02:17.880Z] 2021/08/04 07:02:17 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/t-manager_elastic-package_PR-464/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs elastic-agent
[2021-08-04T07:02:18.818Z] 2021/08/04 07:02:18 DEBUG Dump stack logs for fleet-server
[2021-08-04T07:02:18.818Z] 2021/08/04 07:02:18 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/t-manager_elastic-package_PR-464/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs fleet-server
[2021-08-04T07:02:19.758Z] 2021/08/04 07:02:19 DEBUG Dump stack logs for kibana
[2021-08-04T07:02:19.759Z] 2021/08/04 07:02:19 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/t-manager_elastic-package_PR-464/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs kibana
[2021-08-04T07:02:20.698Z] 2021/08/04 07:02:20 DEBUG Dump stack logs for package-registry
[2021-08-04T07:02:20.698Z] 2021/08/04 07:02:20 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/t-manager_elastic-package_PR-464/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs package-registry
[2021-08-04T07:02:21.636Z] Path to stack dump: build/elastic-stack-dump/check
[2021-08-04T07:02:21.636Z] Done
[2021-08-04T07:02:21.636Z] + kubectl describe pods --all-namespaces
[2021-08-04T07:02:21.895Z] + kubectl logs -l app=kind-fleet-agent-clusterscope -n kube-system
[2021-08-04T07:02:21.895Z] + kind delete cluster
[2021-08-04T07:02:21.895Z] Deleting cluster "kind" ...
[2021-08-04T07:02:25.189Z] + elastic-package stack down -v
[2021-08-04T07:02:25.189Z] 2021/08/04 07:02:24 DEBUG Enable verbose logging
[2021-08-04T07:02:25.189Z] 2021/08/04 07:02:24 DEBUG Distribution built without a version tag, can't determine release chronology. Please consider using official releases at https://github.com/elastic/elastic-package/releases
[2021-08-04T07:02:25.189Z] Take down the Elastic stack
[2021-08-04T07:02:25.189Z] 2021/08/04 07:02:24 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/t-manager_elastic-package_PR-464/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack down --volumes
[2021-08-04T07:02:25.449Z] Stopping elastic-package-stack_elastic-agent_1    ... 
[2021-08-04T07:02:25.449Z] Stopping elastic-package-stack_fleet-server_1     ... 
[2021-08-04T07:02:25.449Z] Stopping elastic-package-stack_kibana_1           ... 
[2021-08-04T07:02:25.449Z] Stopping elastic-package-stack_package-registry_1 ... 
[2021-08-04T07:02:25.449Z] Stopping elastic-package-stack_elasticsearch_1    ... 
[2021-08-04T07:02:42.047Z] 
Stopping elastic-package-stack_elastic-agent_1    ... done

Stopping elastic-package-stack_fleet-server_1     ... done

Stopping elastic-package-stack_kibana_1           ... done

Stopping elastic-package-stack_package-registry_1 ... done

Stopping elastic-package-stack_elasticsearch_1    ... done
Removing elastic-package-stack_elastic-agent_is_ready_1    ... 
[2021-08-04T07:02:42.047Z] Removing elastic-package-stack_fleet-server_is_ready_1     ... 
[2021-08-04T07:02:42.047Z] Removing elastic-package-stack_elastic-agent_1             ... 
[2021-08-04T07:02:42.047Z] Removing elastic-package-stack_kibana_is_ready_1           ... 
[2021-08-04T07:02:42.047Z] Removing elastic-package-stack_fleet-server_1              ... 
[2021-08-04T07:02:42.047Z] Removing elastic-package-stack_kibana_1                    ... 
[2021-08-04T07:02:42.047Z] Removing elastic-package-stack_elasticsearch_is_ready_1    ... 
[2021-08-04T07:02:42.047Z] Removing elastic-package-stack_package-registry_is_ready_1 ... 
[2021-08-04T07:02:42.047Z] Removing elastic-package-stack_package-registry_1          ... 
[2021-08-04T07:02:42.047Z] Removing elastic-package-stack_elasticsearch_1             ... 
[2021-08-04T07:02:42.047Z] 
Removing elastic-package-stack_kibana_is_ready_1           ... done

Removing elastic-package-stack_kibana_1                    ... done

Removing elastic-package-stack_fleet-server_is_ready_1     ... done

Removing elastic-package-stack_elastic-agent_is_ready_1    ... done

Removing elastic-package-stack_package-registry_is_ready_1 ... done

Removing elastic-package-stack_package-registry_1          ... done

Removing elastic-package-stack_elasticsearch_is_ready_1    ... done

Removing elastic-package-stack_elasticsearch_1             ... done

Removing elastic-package-stack_elastic-agent_1             ... done

Removing elastic-package-stack_fleet-server_1              ... done
Removing network elastic-package-stack_default
[2021-08-04T07:02:42.048Z] error while removing network: network elastic-package-stack_default id 55c6ca09102b8bf6afe051b18cc556da8baaf6f19e08da121dfff84b4da75f52 has active endpoints
[2021-08-04T07:02:42.307Z] Error: tearing down the stack failed: stopping docker containers failed: running command failed: running Docker Compose down command failed: exit status 1
[2021-08-04T07:02:42.307Z] Makefile:37: recipe for target 'test-check-packages' failed
[2021-08-04T07:02:42.307Z] make: *** [test-check-packages] Error 1
[2021-08-04T07:02:42.481Z] Post stage
[2021-08-04T07:02:42.508Z] Running in /var/lib/jenkins/workspace/t-manager_elastic-package_PR-464/src/github.com/elastic/elastic-package
[2021-08-04T07:02:42.547Z] Archiving artifacts
[2021-08-04T07:02:42.589Z] Archiving artifacts
[2021-08-04T07:02:42.630Z] Archiving artifacts
[2021-08-04T07:02:42.684Z] Archiving artifacts
[2021-08-04T07:02:42.728Z] Recording test results
[2021-08-04T07:02:42.835Z] No test report files were found. Configuration error?
[2021-08-04T07:02:42.883Z] Error when executing always post condition:
[2021-08-04T07:02:42.883Z] Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from beats-ci-immutable-ubuntu-1804-1628059880534258049.c.elastic-ci-prod.internal/10.224.0.117:57418
[2021-08-04T07:02:42.883Z] 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
[2021-08-04T07:02:42.883Z] 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
[2021-08-04T07:02:42.883Z] 		at hudson.remoting.Channel.call(Channel.java:1001)
[2021-08-04T07:02:42.883Z] 		at hudson.FilePath.act(FilePath.java:1159)
[2021-08-04T07:02:42.883Z] 		at hudson.FilePath.act(FilePath.java:1148)
[2021-08-04T07:02:42.883Z] 		at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:107)
[2021-08-04T07:02:42.883Z] 		at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:153)
[2021-08-04T07:02:42.883Z] 		at hudson.tasks.junit.JUnitResultArchiver.parseAndSummarize(JUnitResultArchiver.java:247)
[2021-08-04T07:02:42.883Z] 		at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:63)
[2021-08-04T07:02:42.883Z] 		at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:29)
[2021-08-04T07:02:42.883Z] 		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2021-08-04T07:02:42.883Z] 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2021-08-04T07:02:42.883Z] 		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2021-08-04T07:02:42.883Z] 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2021-08-04T07:02:42.883Z] 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2021-08-04T07:02:42.883Z] hudson.AbortException: No test report files were found. Configuration error?
[2021-08-04T07:02:42.883Z] 	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:152)
[2021-08-04T07:02:42.883Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3317)
[2021-08-04T07:02:42.883Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
[2021-08-04T07:02:42.883Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2021-08-04T07:02:42.883Z] 	at hudson.remoting.Request$2.run(Request.java:376)
[2021-08-04T07:02:42.883Z] 	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
[2021-08-04T07:02:42.883Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2021-08-04T07:02:42.883Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2021-08-04T07:02:42.883Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2021-08-04T07:02:42.883Z] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:122)
[2021-08-04T07:02:42.884Z] 	at java.lang.Thread.run(Thread.java:748)
[2021-08-04T07:02:42.884Z] 
[2021-08-04T07:02:42.956Z] Stage "Release" skipped due to earlier failure(s)
[2021-08-04T07:02:43.283Z] Running on Jenkins in /var/lib/jenkins/workspace/t-manager_elastic-package_PR-464
[2021-08-04T07:02:43.367Z] [INFO] getVaultSecret: Getting secrets
[2021-08-04T07:02:43.416Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-08-04T07:02:44.283Z] + chmod 755 generate-build-data.sh
[2021-08-04T07:02:44.283Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/elastic-package/PR-464/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/elastic-package/PR-464/runs/1 FAILURE 696788
[2021-08-04T07:02:44.283Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/elastic-package/PR-464/runs/1/steps/?limit=10000 -o steps-info.json
[2021-08-04T07:02:44.833Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/elastic-package/PR-464/runs/1/tests/?status=FAILED -o tests-errors.json
[2021-08-04T07:02:44.833Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-08-04T07:02:45.745Z] Retry 2/3 exited 22, retrying in 2 seconds...

mtojek · 2021-08-04T07:25:45Z

emergency fix, will merge it

elasticmachine · 2021-08-04T07:47:46Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-04T07:08:04.015+0000
Duration: 40 min 39 sec
Commit: eae2f5e

Test stats 🧪

Test	Results
Failed	0
Passed	425
Skipped	4
Total	429

Trends 🧪

Debug: get agent data

734ee73

mtojek self-assigned this Aug 4, 2021

Fix

eae2f5e

mtojek merged commit 470fc80 into elastic:master Aug 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Debug: get agent data #464

Debug: get agent data #464

Uh oh!

mtojek commented Aug 4, 2021

Uh oh!

elasticmachine commented Aug 4, 2021

Build stats

Trends 🧪

`Check`

Uh oh!

mtojek commented Aug 4, 2021

Uh oh!

elasticmachine commented Aug 4, 2021

Build stats

Test stats 🧪

Trends 🧪

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Debug: get agent data #464

Debug: get agent data #464

Uh oh!

Conversation

mtojek commented Aug 4, 2021

Uh oh!

elasticmachine commented Aug 4, 2021

💔 Build Failed

Build stats

Trends 🧪

Steps errors

Check

Log output

Uh oh!

mtojek commented Aug 4, 2021

Uh oh!

elasticmachine commented Aug 4, 2021

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

`Check`