[test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2004. #32748

jtattermusch · 2023-03-29T16:25:37Z

Add a new docker image "rbe_ubuntu2004" that is built in a way that's analogous to how our other testing docker images are built (this gives us control over what exactly is contained in the docker image and ability to fine-tune our RBE configuration)
Switch RBE on linux to the new image (which gives us ubuntu20.04-based builds)

For some reason, RBE seems to have trouble pulling the docker image from Google Artifact Registry (GAR), which is where our public testing images normally live, so for now, I used a workaround and I upload a copy of the rbe_ubuntu2004 docker image to GCR as well, and that makes RBE works just fine (see comment in the renerate_linux_rbe_configs.sh script).

More followup items (config cleanup, getting local sanitizer builds working etc.) are in go/grpc-rbe-tech-debt-2023

jtattermusch · 2023-04-14T08:06:43Z

tools/remote_build/linux.bazelrc

+# use MSAN-instrumented version of libc++
+# TODO(jtattermusch): include the modified LD_LIBRARY_PATH in an msan-specific
+# toolchain configuration, instead of needing to explicitly set it here.
+build:msan --action_env=LD_LIBRARY_PATH=/usr/local/libcxx-msan/lib/x86_64-unknown-linux-gnu


qq: until now, we haven't been setting any special flavor of libc++ for TSAN. Since the rbe_ubuntu2004 image now has also the TSAN-instrumented version of libstdc++ installed, it might be a good idea to use it (in a similar way we're setting MSAN-instrumented version of libstdc++ here). I'm leaving that as a possible followup (I don't want to change more stuff than necessary in this PR).

jtattermusch · 2023-04-14T08:33:19Z

Issues to resolve:
The examine_stack_test is failing with ASAN (for unknown reason, perhaps something missing in the RBE docker image?)

[symbolize_elf.inc : 1403] RAW: /b/f/w/bazel-out/k8-fastbuild/bin/test/core/gprpp/examine_stack_test.runfiles/com_github_grpc_grpc/_solib_k8/libsrc_Score_Slibexamine_Ustack.so: unable to find LOAD segment for pc: 0x7fc55d6b8ecb, start_addr: 7fc55d6b7000
test/core/gprpp/examine_stack_test.cc:71: Failure
Value of: stack_trace->find("GetCurrentStackTrace") != std::string::npos
  Actual: false
Expected: true
[  FAILED  ] ExamineStackTest.AbseilStackProvider (15 ms)

https://source.cloud.google.com/results/invocations/b07c5649-3972-476d-a581-06d76b5d67d1/targets;collapsed=/%2F%2Ftest%2Fcore%2Fgprpp:examine_stack_test/tests

jtattermusch · 2023-04-14T13:58:56Z

@veblush @ctiller I'd be grateful for tips how to solve the examine_stack_test failure on ASAN (since it seems to be only thing that prevents merging).

In order to help #32748, change the test so that it tells us what the problem is in the logs.

Try a different approach to this test and check some non-leaf functions in the returned text - looks like we're running into problems getting the leaf function out of the stack trace on that platform (which is probably fine): https://source.cloud.google.com/results/invocations/09e8e1ea-df48-4fdb-96dd-916bd5014f90/targets/%2F%2Ftest%2Fcore%2Fgprpp:examine_stack_test/tests Needed to unblock #32748

veblush · 2023-05-01T17:05:43Z

It looks like it's now passing all Bazel RBE tests so it's worth being merged.

In order to help grpc#32748, change the test so that it tells us what the problem is in the logs.

jtattermusch · 2023-05-03T09:38:26Z

Thanks @ctiller for unblocking this!

Try a different approach to this test and check some non-leaf functions in the returned text - looks like we're running into problems getting the leaf function out of the stack trace on that platform (which is probably fine): https://source.cloud.google.com/results/invocations/09e8e1ea-df48-4fdb-96dd-916bd5014f90/targets/%2F%2Ftest%2Fcore%2Fgprpp:examine_stack_test/tests Needed to unblock grpc#32748

…004. (grpc#32748) - Add a new docker image "rbe_ubuntu2004" that is built in a way that's analogous to how our other testing docker images are built (this gives us control over what exactly is contained in the docker image and ability to fine-tune our RBE configuration) - Switch RBE on linux to the new image (which gives us ubuntu20.04-based builds) For some reason, RBE seems to have trouble pulling the docker image from Google Artifact Registry (GAR), which is where our public testing images normally live, so for now, I used a workaround and I upload a copy of the rbe_ubuntu2004 docker image to GCR as well, and that makes RBE works just fine (see comment in the `renerate_linux_rbe_configs.sh` script). More followup items (config cleanup, getting local sanitizer builds working etc.) are in go/grpc-rbe-tech-debt-2023

In order to help #32748, change the test so that it tells us what the problem is in the logs.

Try a different approach to this test and check some non-leaf functions in the returned text - looks like we're running into problems getting the leaf function out of the stack trace on that platform (which is probably fine): https://source.cloud.google.com/results/invocations/09e8e1ea-df48-4fdb-96dd-916bd5014f90/targets/%2F%2Ftest%2Fcore%2Fgprpp:examine_stack_test/tests Needed to unblock #32748

…004. (#32748) - Add a new docker image "rbe_ubuntu2004" that is built in a way that's analogous to how our other testing docker images are built (this gives us control over what exactly is contained in the docker image and ability to fine-tune our RBE configuration) - Switch RBE on linux to the new image (which gives us ubuntu20.04-based builds) For some reason, RBE seems to have trouble pulling the docker image from Google Artifact Registry (GAR), which is where our public testing images normally live, so for now, I used a workaround and I upload a copy of the rbe_ubuntu2004 docker image to GCR as well, and that makes RBE works just fine (see comment in the `renerate_linux_rbe_configs.sh` script). More followup items (config cleanup, getting local sanitizer builds working etc.) are in go/grpc-rbe-tech-debt-2023

grpc-checks bot added per-call-memory/neutral per-channel-memory/neutral bloat/none labels Mar 29, 2023

github-actions bot added title needs formatting lang/Python labels Mar 29, 2023

jtattermusch force-pushed the rbe2004 branch from f080f92 to 5d10cd8 Compare April 13, 2023 12:46

add rbe_ubuntu2004 docker image for running linux RBE tests

a41137c

jtattermusch force-pushed the rbe2004 branch from 5d10cd8 to 7dcd862 Compare April 14, 2023 07:52

jtattermusch changed the title ~~Rbe2004~~ Switch RBE linux build to a new custom image rbe_ubuntu2004 Apr 14, 2023

github-actions bot added the PR title format label Apr 14, 2023

jtattermusch changed the title ~~Switch RBE linux build to a new custom image rbe_ubuntu2004~~ [test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2004 Apr 14, 2023

jtattermusch marked this pull request as ready for review April 14, 2023 08:01

jtattermusch requested a review from apolcyn as a code owner April 14, 2023 08:01

github-actions bot removed the PR title format label Apr 14, 2023

jtattermusch added lang/c++ release notes: no Indicates if PR should not be in release notes and removed lang/Python labels Apr 14, 2023

jtattermusch requested review from ctiller and veblush April 14, 2023 08:03

jtattermusch assigned veblush Apr 14, 2023

jtattermusch commented Apr 14, 2023

View reviewed changes

jtattermusch added 6 commits April 14, 2023 10:35

use MSAN-instrumented libc++ on RBE

a16cb2d

use image mirror in gcr.io to make RBE work

7aa1d06

regenerate dockerfile

9250293

regenerate toolchain

2ae68e8

rename toolchain rbe_ubuntu1804->rbe_ubuntu2004

be1c2c9

rename toolchain config to rbe_ubuntu2004

6e8ce11

jtattermusch force-pushed the rbe2004 branch from 7dcd862 to 6e8ce11 Compare April 14, 2023 08:40

jtattermusch requested a review from gnossen April 14, 2023 13:57

ctiller mentioned this pull request Apr 19, 2023

[gprpp] Better test for examine_stack #32897

Merged

ctiller added the kokoro:force-run label Apr 30, 2023

grpc-kokoro removed the kokoro:force-run label Apr 30, 2023

ctiller mentioned this pull request Apr 30, 2023

[examine-stack] Try to unblock Ubuntu 20.04 #32975

Merged

ctiller added the kokoro:force-run label May 1, 2023

grpc-kokoro removed the kokoro:force-run label May 1, 2023

veblush approved these changes May 1, 2023

View reviewed changes

ctiller removed their request for review May 1, 2023 17:24

ctiller changed the title ~~[test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2004~~ [test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2004. May 1, 2023

veblush merged commit 30b3d50 into grpc:master May 1, 2023
62 checks passed

copybara-service bot added the imported Specifies if the PR has been imported to the internal repository label May 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2004. #32748

[test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2004. #32748

jtattermusch commented Mar 29, 2023 •

edited

jtattermusch Apr 14, 2023 •

edited

jtattermusch commented Apr 14, 2023 •

edited

jtattermusch commented Apr 14, 2023

veblush commented May 1, 2023

jtattermusch commented May 3, 2023

[test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2004. #32748

[test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2004. #32748

Conversation

jtattermusch commented Mar 29, 2023 • edited

jtattermusch Apr 14, 2023 • edited

Choose a reason for hiding this comment

jtattermusch commented Apr 14, 2023 • edited

jtattermusch commented Apr 14, 2023

veblush commented May 1, 2023

jtattermusch commented May 3, 2023

jtattermusch commented Mar 29, 2023 •

edited

jtattermusch Apr 14, 2023 •

edited

jtattermusch commented Apr 14, 2023 •

edited