[C++] Linux Flight/FlightSQL test randomly segfaults after gRPC version bump

[alinaliBQ]

Describe the bug, including details regarding any error messages, version, and platform.

After the following PR was merged:

• GH-48926: [C++] Upgrade Abseil/Protobuf/GRPC/Google-Cloud-CPP bundled versions #48964

Non-deterministic crash at Flight/Flight SQL/ODBC tests is introduced:

SIGSEGV in /lib64/ld-linux-x86-64.so.2

The crash can happen at arrow-flight-test, arrow-flight-sql-test, or arrow-odbc-spi-impl-test. They all use gRPC and protobuf.

arrow-flight-test failure: https://github.com/apache/arrow/actions/runs/24410900735/job/71307679982#step:6:10272
arrow-flight-sql-test failure: https://github.com/apache/arrow/actions/runs/24410900735/job/71318452958#step:6:10207
arrow-odbc-spi-impl-test failure: https://github.com/apache/arrow/actions/runs/24410900735/job/71307679982#step:6:10330

The error log looks like:

    Test #68: arrow-flight-test .....................***Failed    0.54 sec
Running arrow-flight-test, redirecting output into /build/cpp/build/test-logs/arrow-flight-test.txt (attempt 1/1)
/arrow/cpp/build-support/run-test.sh: line 88: 26750 Segmentation fault      (core dumped) $TEST_EXECUTABLE "$@" > $LOGFILE.raw 2>&1
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Running '/build/cpp/release/arrow-flight-test' produced core dump at '/tmp/core.arrow-flight-te.26750', printing backtrace:
[New LWP 26750]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/build/cpp/release/arrow-flight-test'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f643d998128 in ?? () from /lib64/ld-linux-x86-64.so.2

Thread 1 (Thread 0x7f643c215700 (LWP 26750)):
#0  0x00007f643d998128 in ?? () from /lib64/ld-linux-x86-64.so.2
#1  0x00007f643d9aa789 in ?? () from /lib64/ld-linux-x86-64.so.2
#2  0x00007f643d9a6f46 in ?? () from /lib64/ld-linux-x86-64.so.2
#3  0x00007f643d9a875e in ?? () from /lib64/ld-linux-x86-64.so.2
#4  0x00007f643d9a7548 in ?? () from /lib64/ld-linux-x86-64.so.2
#5  0x0000000000000001 in ?? ()
#6  0x00007fffa0cad7f7 in ?? ()
#7  0x0000000000000000 in ?? ()
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/build/cpp/src/arrow/flight

Component(s)

FlightRPC, C++

[alinaliBQ]

cc @raulcd @lidavidm @kou
If you could have a look at this, it would be great. Thanks

[kou]

Can we use debug build not release build in CI for better backtrace?

[raulcd]

@alinaliBQ apart from @kou 's suggestion so we can better diagnose, have you been able to reproduce locally?

[alinaliBQ]

@kou I have a debug workflow run here: https://github.com/apache/arrow/actions/runs/24532172035/job/71718034555?pr=49777#step:6:8685. Here is a draft PR that adds a debug workflow: #49777)
The backtrace seems similar as release. I think it's because the loading failure occurs before the test cases run.

@raulcd I could not reproduce this issue locally. I have only seen it on CI with cpp_test.sh.

[raulcd]

I've been able to reproduce pretty consistently locally (5 failures out of 6) with:

UBUNTU=24.04 archery docker run -e ARROW_BUILD_TYPE=DEBUG ubuntu-cpp-odbc

I've also reproduced with the underlying docker command:

UBUNTU=24.04 docker compose run --rm -e ARROW_BUILD_TYPE=DEBUG ubuntu-cpp-odbc

But when I try to run an interactive shell in the container with:

$ UBUNTU=24.04 docker compose run -it --rm -e ARROW_BUILD_TYPE=DEBUG ubuntu-cpp-odbc /bin/bash

and execute the build, install driver and test using the same command specified on the compose.yaml file:

root@3dad19853bc0:/# /arrow/ci/scripts/cpp_build.sh /arrow /build
...
root@3dad19853bc0:/# sudo /arrow/cpp/src/arrow/flight/sql/odbc/install/unix/install_odbc.sh /usr/local/lib/libarrow_flight_sql_odbc.so
Adding [Apache Arrow Flight SQL ODBC Driver] to odbcinst.ini...
root@3dad19853bc0:/# /arrow/ci/scripts/cpp_test.sh /arrow /build
...
100% tests passed, 0 tests failed out of 56

Label Time Summary:
arrow-compute-tests    = 102.32 sec*proc (15 tests)
arrow-tests            =  82.14 sec*proc (38 tests)
arrow_flight           =   7.66 sec*proc (2 tests)
arrow_flight_sql       =   1.44 sec*proc (1 test)
filesystem             =   1.90 sec*proc (1 test)
unittest               = 193.55 sec*proc (56 tests)

I can't seem to reproduce no matter how many times I try.

I have tried to debug and understand the crash but I haven't been able to advance much on it.

[alinaliBQ]

Thanks for looking into this Raul.

I was able to reproduce the error one time by running the docker compose run ... command you posted. Before I was trying to reproduce with the interactive shell as well and couldn't get the error to show no matter how many times I run the arrow-odbc-spi-impl-test test. Since the error can't be reproduced on the bash shell, I am not sure if this is an real issue or the tests react differently with ctest.

I think for now we could re-run the CI to get the workflow to pass. We will note this for future ODBC Linux CI runs.

cc @lidavidm @kou