Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI Failure (Segments were not removed) in TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test #8046

Closed
mmaslankaprv opened this issue Jan 5, 2023 · 5 comments · Fixed by #8090 or #8214
Closed

CI Failure (Segments were not removed) in TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test #8046

mmaslankaprv opened this issue Jan 5, 2023 · 5 comments · Fixed by #8090 or #8214
Assignees
@jcsp
Labels
area/cloud-storage Shadow indexing subsystem ci-disabled-test ci-failure kind/bug Something isn't working sev/low Bugs which are non-functional paper cuts, e.g. typos, issues in log messages

Comments

@mmaslankaprv
Copy link
Member

mmaslankaprv commented Jan 5, 2023 •
edited by rystsov

https://buildkite.com/redpanda/redpanda/builds/20646#018581fc-219d-4e84-8d5d-53de79924b41

Module: rptest.tests.topic_delete_test
Class:  TopicDeleteCloudStorageTest
Method: topic_delete_cloud_storage_test
Arguments:
{
  "disable_delete": true
}

A failure in:

rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test.disable_delete=True

Test failed with a message:

[INFO  - 2023-01-05 13:29:12,740 - runner_client - log - lineno:278]: RunnerClient: rptest.tests.topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test.disable_delete=True: FAIL: TimeoutError('Segments were not removed')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/topic_delete_test.py", line 305, in topic_delete_cloud_storage_test
    self._populate_topic(self.topic)
  File "/root/tests/rptest/tests/topic_delete_test.py", line 207, in _populate_topic
    wait_for_segments_removal(self.redpanda, topic_name, i, 5)
  File "/root/tests/rptest/util.py", line 256, in wait_for_segments_removal
    wait_until(done,
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Segments were not removed
@mmaslankaprv mmaslankaprv added area/cloud-storage Shadow indexing subsystem ci-failure labels Jan 5, 2023
@mmaslankaprv mmaslankaprv mentioned this issue Jan 5, 2023
Merged
6 tasks
@r-vasquez r-vasquez mentioned this issue Jan 5, 2023
Merged
1 task
@r-vasquez
Copy link
Contributor

Same here: https://buildkite.com/redpanda/redpanda/builds/20665#018582ee-91a6-4831-a7fa-9fcadb93a0eb/6-2646

@BenPope
Copy link
Member

BenPope commented Jan 6, 2023

I think the same here: https://buildkite.com/redpanda/vtools/builds/5050#01858470-138f-4407-ac29-ee10b89b5f8f

@dlex
Copy link
Contributor

dlex commented Jan 6, 2023

https://buildkite.com/redpanda/redpanda/builds/20621#01858070-ed5f-403c-a233-f087f17889e4

@NyaliaLui NyaliaLui mentioned this issue Jan 6, 2023
Merged
6 tasks
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 6, 2023
@jcsp
tests: mark unstable topic deletion tests ok_to_fail
94a19c8 
Related: redpanda-data#8071
Related: redpanda-data#8046
@jcsp jcsp mentioned this issue Jan 6, 2023
Merged
6 tasks
@jcsp jcsp self-assigned this Jan 6, 2023
@jcsp
Copy link
Contributor

jcsp commented Jan 6, 2023

A leadership transfer happened partway through the test, which bumped up the segment count.

I'm fixing the test to use a byte count rather than a segment count in its condition.

@jcsp jcsp added kind/bug Something isn't working sev/low Bugs which are non-functional paper cuts, e.g. typos, issues in log messages labels Jan 6, 2023
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 6, 2023
@jcsp
tests: improve topic_delete_test
73ec061 
* Use a disk space condition instead of a segment count
  condition in _populate_topic
* Log remainig contents in `topic_storage_purged`, so that
  we have some idea what is going wrong if purging doesn't
  complete.

Related: redpanda-data#8046
@dlex
Copy link
Contributor

dlex commented Jan 6, 2023

on (amd64, container) https://buildkite.com/redpanda/redpanda/builds/20745#018586b8-db96-4fb0-8d3d-8d366b985f7c
on (amd64, container) https://buildkite.com/redpanda/redpanda/builds/20760#0185879f-94e5-4bdf-9cbc-3d1f9ba56a02

jcsp added a commit to jcsp/redpanda that referenced this issue Jan 9, 2023
@jcsp
tests: improve topic_delete_test
35ff8d9 
* Use a disk space condition instead of a segment count
  condition in _populate_topic
* Log remainig contents in `topic_storage_purged`, so that
  we have some idea what is going wrong if purging doesn't
  complete.

Related: redpanda-data#8046
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 9, 2023
@jcsp
tests: improve topic_delete_test
31f0ef1 
* Use a disk space condition instead of a segment count
  condition in _populate_topic
* Log remainig contents in `topic_storage_purged`, so that
  we have some idea what is going wrong if purging doesn't
  complete.

Related: redpanda-data#8046
@jcsp jcsp mentioned this issue Jan 9, 2023
Merged
6 tasks
@rystsov rystsov changed the title Failure in topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test CI Failure (Segments were not removed) in topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test Jan 9, 2023
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 12, 2023
@jcsp
tests: improve topic_delete_test
9a54206 
* Use a disk space condition instead of a segment count
  condition in _populate_topic
* Log remainig contents in `topic_storage_purged`, so that
  we have some idea what is going wrong if purging doesn't
  complete.

Related: redpanda-data#8046
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 13, 2023
@jcsp
tests: improve topic_delete_test
5231c13 
* Use a disk space condition instead of a segment count
  condition in _populate_topic
* Log remainig contents in `topic_storage_purged`, so that
  we have some idea what is going wrong if purging doesn't
  complete.

Related: redpanda-data#8046
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 13, 2023
@jcsp
tests: improve topic_delete_test
ffbb279 
* Use a disk space condition instead of a segment count
  condition in _populate_topic
* Log remainig contents in `topic_storage_purged`, so that
  we have some idea what is going wrong if purging doesn't
  complete.

Fixes: redpanda-data#8046
Fixes: redpanda-data#8084
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 13, 2023
@jcsp
tests: re-enable topic_delete_cloud_storage_test
0348119 
Related: redpanda-data#8046
@jcsp jcsp mentioned this issue Jan 13, 2023
Merged
6 tasks
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 13, 2023
@jcsp
tests: improve topic_delete_test
fbddf01 
* Use a disk space condition instead of a segment count
  condition in _populate_topic
* Log remainig contents in `topic_storage_purged`, so that
  we have some idea what is going wrong if purging doesn't
  complete.

Related: redpanda-data#8046
@rystsov rystsov changed the title CI Failure (Segments were not removed) in topic_delete_test.TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test CI Failure (Segments were not removed) in TopicDeleteCloudStorageTest.topic_delete_cloud_storage_test Jan 13, 2023
jcsp added a commit that referenced this issue Jan 13, 2023
@jcsp
Merge pull request #8214 from jcsp/issue-8084
d91039d 
tests: fix cloud storage deletion issues #8046, #8084
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jcsp jcsp

Labels
area/cloud-storage Shadow indexing subsystem ci-disabled-test ci-failure kind/bug Something isn't working sev/low Bugs which are non-functional paper cuts, e.g. typos, issues in log messages
Projects
None yet
Milestone
No milestone
5 participants
@BenPope @jcsp @dlex @mmaslankaprv @r-vasquez