Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky tests should be disabled on BazelCI #1631

Closed
joeleba opened this issue May 5, 2023 · 3 comments
Closed

Flaky tests should be disabled on BazelCI #1631

joeleba opened this issue May 5, 2023 · 3 comments
Assignees
@meteorcloudy

Comments

@joeleba
Copy link
Member

joeleba commented May 5, 2023

Test Example flake
src/test/java/com/google/devtools/build/lib/skyframe/LocalDiffAwarenessIntegrationTest buildkite
src/test/shell/bazel/bazel_sandboxing_networking_test buildkite
Windows tests buildkite
src/test/java/com/google/devtools/build/lib/remote/remote buildkite

IMHO these tests add quite a bit of inertia to submitting changes, especially when we have a chain of small CLs to submit.

From my own experience: the 3 automatic trials almost never resolves the flakes. It's only when we manually click the retry button from BuildKite does the result differ.
@meteorcloudy meteorcloudy self-assigned this May 5, 2023
@meteorcloudy meteorcloudy removed their assignment May 16, 2023
copybara-service bot pushed a commit to bazelbuild/bazel that referenced this issue May 23, 2023
@meteorcloudy
Disable //src/test/shell/bazel/bazel_sandboxing_networking_test on rb…
67530f2 
…e_ubuntu1804 in Bazel presubmit

Related bazelbuild/continuous-integration#1631

PiperOrigin-RevId: 534424531
Change-Id: I42c225687cd0a3522e5a3fd880173286c5252ce2
@sgowroji
Copy link
Member

sgowroji commented May 24, 2023 •
edited
Loading

Adding few more flaky tests from downstream CI.

Test Example flake
src/test/shell/bazel:bazel_coverage_java_jdk11_toolchain_head_test buildkite
src/test/shell/bazel:bazel_hermetic_sandboxing_test buildkite
src/test/shell/bazel:bazel_proto_library_test buildkite
src/test/shell/bazel:bazel_workspaces_test buildkite

@meteorcloudy
Copy link
Member

https://buildkite.com/organizations/bazel/analytics has just been available, those tests will probably show up on the dashboard.

copybara-service bot pushed a commit to bazelbuild/bazel that referenced this issue May 24, 2023
@meteorcloudy
Bump test size for bazel_coverage_java_jdk_* tests and bazel_hermetic…
3f0b223 
…_sandboxing_test

Addressing bazelbuild/continuous-integration#1631 (comment)

PiperOrigin-RevId: 534827179
Change-Id: I2a3656e4c19cab0fa51ab642b62e84237d2b03c5
fweikert pushed a commit to fweikert/bazel that referenced this issue May 25, 2023
@meteorcloudy @fweikert
Disable //src/test/shell/bazel/bazel_sandboxing_networking_test on rb…
40b66df 
…e_ubuntu1804 in Bazel presubmit

Related bazelbuild/continuous-integration#1631

PiperOrigin-RevId: 534424531
Change-Id: I42c225687cd0a3522e5a3fd880173286c5252ce2
fweikert pushed a commit to fweikert/bazel that referenced this issue May 25, 2023
@meteorcloudy @fweikert
Bump test size for bazel_coverage_java_jdk_* tests and bazel_hermetic…
7f60ef9 
…_sandboxing_test

Addressing bazelbuild/continuous-integration#1631 (comment)

PiperOrigin-RevId: 534827179
Change-Id: I2a3656e4c19cab0fa51ab642b62e84237d2b03c5
@meteorcloudy meteorcloudy self-assigned this May 30, 2023
@meteorcloudy
Copy link
Member

I'm closing this one since all flaky tests are tracked by https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3A%22flaky+test%22 and gTech team is helping reporting new flaky tests based on https://buildkite.com/organizations/bazel/analytics .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@meteorcloudy meteorcloudy

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@meteorcloudy @joeleba @sgowroji