transient failures during lxc test during shutdown

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1783198

Launchpad details

affected_projects = []
assignee = smoser
assignee_name = Scott Moser
date_closed = 2022-11-02T09:34:22.785306+00:00
date_created = 2018-07-23T20:04:48.742985+00:00
date_fix_committed = None
date_fix_released = None
id = 1783198
importance = medium
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1783198
milestone = None
owner = smoser
owner_name = Scott Moser
private = False
status = wont_fix
submitter = smoser
submitter_name = Scott Moser
tags = []
duplicates = []

Launchpad user Scott Moser(smoser) wrote on 2018-07-23T20:04:48.742985+00:00

We have been seeing a lot of transient failures

https://jenkins.ubuntu.com/server/job/cloud-init-integration-lxd-c/72/consoleFull
with a stack trace that looks like below.

I think that we might be attempting to delete an instance twice or shutting it down twice. not sure.

2018-07-20 12:20:30,781 - tests.cloud_tests - DEBUG - executing "collect: instance-id"
2018-07-20 12:20:46,612 - tests.cloud_tests - ERROR - stage: collect test data for cosmic encountered error: not found
2018-07-20 12:20:46,614 - tests.cloud_tests - ERROR - traceback:
File "/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-integration-lxd-c/cloud-init/tests/cloud_tests/stage.py", line 97, in run_stage
(call_res, call_failed) = call()
File "/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-integration-lxd-c/cloud-init/tests/cloud_tests/collect.py", line 111, in collect_test_data
instance.shutdown()
File "/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-integration-lxd-c/cloud-init/tests/cloud_tests/platforms/lxd/instance.py", line 171, in shutdown
self.pylxd_container.stop(wait=wait)
File "/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-integration-lxd-c/cloud-init/.tox/citest/lib/python3.5/site-packages/pylxd/models/container.py", line 316, in stop
wait=wait)
File "/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-integration-lxd-c/cloud-init/.tox/citest/lib/python3.5/site-packages/pylxd/models/container.py", line 291, in _set_state
response.json()['operation'])
File "/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-integration-lxd-c/cloud-init/.tox/citest/lib/python3.5/site-packages/pylxd/models/operation.py", line 33, in wait_for_operation
return cls.get(client, operation.id)
File "/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-integration-lxd-c/cloud-init/.tox/citest/lib/python3.5/site-packages/pylxd/models/operation.py", line 40, in get
response = client.api.operations[operation_id].get()
File "/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-integration-lxd-c/cloud-init/.tox/citest/lib/python3.5/site-packages/pylxd/client.py", line 148, in get
is_api=is_api)
File "/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-integration-lxd-c/cloud-init/.tox/citest/lib/python3.5/site-packages/pylxd/client.py", line 103, in _assert_response
raise exceptions.NotFound(response)

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2018-07-27T15:38:43.951180+00:00

I'm attaching all console logs that were available of lxd runs of integration
tests. The interesting thing is that most of the time when we see pylxd
related trace backs it is in shutdown (there was one in start). Also
interesting is that most of the time the traceback occurs after collection
of the last file, and about 25-30 seconds later.

So for shutdown specifically, it could be a result of the system failing
to shutdown.

Example:
2018-07-15 12:20:11,234 - tests.cloud_tests - DEBUG - executing "collect: result.json"
2018-07-15 12:20:28,612 - tests.cloud_tests - ERROR - stage: collect test data for bionic encountered error: not found
2018-07-15 12:20:28,615 - tests.cloud_tests - ERROR - traceback:

Launchpad attachments: console logs of lxd integration tests

[ubuntu-server-builder]

Launchpad user James Falcon(falcojr) wrote on 2022-11-02T09:34:15.366925+00:00

These tests no longer exist.