Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failure in StorageResourceRestartTest.test_recovery_reads on ARM #7256

Closed
NyaliaLui opened this issue Nov 11, 2022 · 9 comments · Fixed by #8433
Closed

Failure in StorageResourceRestartTest.test_recovery_reads on ARM #7256

NyaliaLui opened this issue Nov 11, 2022 · 9 comments · Fixed by #8433
Assignees
@VladLazar
Labels
area/storage ci-failure kind/bug Something isn't working

Comments

@NyaliaLui
Copy link
Contributor

Version & Environment

Redpanda version: dev CDT nightly ARM:

https://buildkite.com/redpanda/vtools/builds/4201#018463f2-bed8-437b-af13-fa9cc677ecaf/6-7175
https://buildkite.com/redpanda/vtools/builds/4196#01846250-d53f-473f-83be-eb6536cfa36f/6-7170

    AssertionError()
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/storage_resources_test.py", line 279, in test_recovery_reads
    assert self._sum_metrics(read_bytes_metric) < total_bytes
AssertionError
@NyaliaLui NyaliaLui added kind/bug Something isn't working area/tests ci-failure labels Nov 11, 2022
@ztlpn
Copy link
Contributor

ztlpn commented Dec 20, 2022

https://buildkite.com/redpanda/vtools/builds/4744#01852bff-dff6-454f-9d4f-2b414eb221c7

@BenPope
Copy link
Member

BenPope commented Dec 30, 2022

FAIL test: StorageResourceRestartTest.test_recovery_reads.clean_shutdown=True (7/153 runs)
failure at 2022-12-30T04:28:17.969Z: AssertionError()
on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4928#01855f7c-4001-4ed2-90cc-e7a1398b565a
failure at 2022-12-29T04:31:38.661Z: AssertionError()
on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4923#01855a56-3e54-4997-9a52-fa50cb839888
failure at 2022-12-28T04:28:12.819Z: AssertionError()
on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4918#0185552f-c829-43a8-9d3b-b74db0421f55
failure at 2022-12-27T04:39:59.859Z: AssertionError()
on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4913#0185500b-bdaa-4cb0-b640-31259ecac3be
failure at 2022-12-26T04:32:21.108Z: AssertionError()
on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4908#01854ae2-7717-429d-85ac-87d26f4e421a
failure at 2022-12-24T04:13:23.294Z: AssertionError()
on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4891#01854096-20d5-401c-b8b1-94e4d21e8e45
failure at 2022-12-25T04:20:18.252Z: AssertionError()
on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/4898#018545bd-095d-4b37-b549-0173b1d272c1

@dlex
Copy link
Contributor

dlex commented Jan 6, 2023

on (arm64, VM) https://buildkite.com/redpanda/vtools/builds/5007#01857e64-9b47-44f1-ade7-92fcfff06d73

@dlex
Copy link
Contributor

dlex commented Jan 6, 2023 •
edited

on (arm64, VM) https://buildkite.com/redpanda/vtools/builds/5047#0185838c-0b8d-4784-abe4-d77cf7b61b3e
on (arm64, VM) https://buildkite.com/redpanda/vtools/builds/5007#01858500-14b5-4276-87c4-449e1f6c4880

@andijcr
Copy link
Contributor

andijcr commented Jan 23, 2023 

FAIL test: StorageResourceRestartTest.test_recovery_reads.clean_shutdown=True (1/39 runs)
  failure at 2023-01-23T04:13:45.139Z: AssertionError()
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/5396#0185db17-163b-49ec-ace4-b4238606be02

https://buildkite.com/redpanda/vtools/builds/5396#0185db17-163b-49ec-ace4-b4238606be02

@rystsov
Copy link
Contributor

rystsov commented Jan 24, 2023

https://buildkite.com/redpanda/vtools/builds/5426#0185e03d-d646-4dc3-b49b-3ff4cfe3c97a

FAIL test: StorageResourceRestartTest.test_recovery_reads.clean_shutdown=True (1/30 runs)
  failure at 2023-01-24T04:13:12.241Z: AssertionError()

@andijcr
Copy link
Contributor

andijcr commented Jan 25, 2023

https://buildkite.com/redpanda/vtools/builds/5455#0185e565-386c-42c5-824d-8cb81a46d627

FAIL test: StorageResourceRestartTest.test_recovery_reads.clean_shutdown=True (1/35 runs)
  failure at 2023-01-25T03:40:52.893Z: AssertionError()
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/5455#0185e565-386c-42c5-824d-8cb81a46d627

@rystsov
Copy link
Contributor

rystsov commented Jan 26, 2023

https://buildkite.com/redpanda/vtools/builds/5484#0185ea8f-6d91-4f53-94ca-348ecb773302

FAIL test: StorageResourceRestartTest.test_recovery_reads.clean_shutdown=True (1/39 runs)
  failure at 2023-01-26T03:42:54.398Z: AssertionError()
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/5484#0185ea8f-6d91-4f53-94ca-348ecb773302

VladLazar pushed a commit to VladLazar/redpanda that referenced this issue Jan 26, 2023
storage: share target replay bytes amongst shards
9491250 
'storage_target_replay_bytes' is a configuration option that represents
a soft limit of how many dirty bytes the state machines on a node can
have. It is meant to be shared between the shards.

This commit fixes a bug due to which this target was not being shared
between shards. The impact is that we were replaying more data (up to
the 64MiB hard-coded limit) from the log on restarts as the latest
snapshot was further in the past.

Fixes redpanda-data#7256
Related redpanda-data#6854
@VladLazar VladLazar mentioned this issue Jan 26, 2023
Merged
6 tasks
@andijcr
Copy link
Contributor

andijcr commented Jan 27, 2023

https://buildkite.com/redpanda/vtools/builds/5500#0185efb8-35af-4ec9-8e11-2d6d69c3d310

FAIL test: StorageResourceRestartTest.test_recovery_reads.clean_shutdown=True (1/43 runs)
  failure at 2023-01-27T03:54:39.906Z: AssertionError()
      on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/5500#0185efb8-35af-4ec9-8e11-2d6d69c3d310

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VladLazar VladLazar

Labels
area/storage ci-failure kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

storage: share storage_target_replay_bytes amongst all shards VladLazar/redpanda
8 participants
@dotnwat @andijcr @BenPope @rystsov @ztlpn @NyaliaLui @dlex @VladLazar