Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

arm64 robustness test failure #17593

Open
serathius opened this issue Mar 17, 2024 · 3 comments
Open

arm64 robustness test failure #17593

serathius opened this issue Mar 17, 2024 · 3 comments
Assignees
Labels
area/robustness-testing area/testing priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. type/flake

Comments

@serathius
Copy link
Member

Which Github Action / Prow Jobs are flaking?

Robustness nightly on arm

Which tests are flaking?

TestRobustnessExploratory/Etcd/HighTraffic/ClusterOfSize3 x2
TestRobustnessExploratory/Kubernetes/LowTraffic/ClusterOfSize3
TestRobustnessExploratory/Etcd/LowTraffic/ClusterOfSize3

  • Timeouts

Github Action / Prow Job link

https://github.com/etcd-io/etcd/actions/runs/8153962092/job/22286380964

Reason for failure (if possible)

Selection_013

7/18 flake ratio, all on arm64.

In most cases logs imply too low qps, which might be sign of lower performance, or issue with recovering from failure injection.

I have seen previously failure cases with logs reporting broken snapshots sent between members. So there might be an underlying correctness issue on arm64.

Anything else we need to know?

If arm64 tests are not stabilized, I will consider disabling it due to lack of support.

@ah8ad3
Copy link
Contributor

ah8ad3 commented Mar 18, 2024

If it's ok i can take a look at them. Maybe i can find the issue.

@jmhbnz
Copy link
Member

jmhbnz commented Mar 18, 2024

Hey @ah8ad3 - Thanks for your interest, I will take this one. I believe it's a performance related challenge with our arm64 CI infrastructure.

Please feel free to look into any failures on arm64 that don't relate to qps to low, these will be more interesting flakes to explore.

/assign

@ah8ad3
Copy link
Contributor

ah8ad3 commented Mar 18, 2024

Thanks for the info @jmhbnz . OK i'll do that. GL on this one.

@jmhbnz jmhbnz added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Mar 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/robustness-testing area/testing priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. type/flake
Development

No branches or pull requests

3 participants