[C++][Flight] arrow-flight-sql-test fails with ASAN UBSAN

[asfimport]

The CI job for ASAN UBSAN is based on Ubuntu 20.04: C++ / AMD64 Ubuntu 20.04 C++ ASAN UBSAN 

Trying to build Flight and Flight SQL on Ubuntu 20.04 the job for ASAN UBSAN will also build with Flight and Flight SQL. This triggers some arrow-flight-sql-test failures like:

  [ RUN      ] TestFlightSqlClient.TestGetDbSchemas
unknown file: Failure
Unexpected mock function call - taking default action specified at:
/arrow/cpp/src/arrow/flight/sql/client_test.cc:151:
    Function call: GetFlightInfo(@0x61500007d948 184-byte object <00-00 00-00 00-00 F0-BF 40-00 00-00 00-00 00-00 80-4C 06-49 CF-7F 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 01-01 00-00 00-00 00-00 00-20 00-00 00-00 00-00 ... 01-00 00-04 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>, @0x7fff35794e80 64-byte object <02-00 00-00 00-00 00-00 C0-45 08-00 B0-60 00-00 65-00 00-00 00-00 00-00 65-00 00-00 00-00 00-00 C4-A9 AE-66 00-10 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>)
          Returns: (nullptr)
Google Mock tried the following 1 expectation, but it didn't match:
/arrow/cpp/src/arrow/flight/sql/client_test.cc:152: EXPECT_CALL(sql_client_, GetFlightInfo(Ref(call_options_), descriptor))...
  Expected arg #1: is equal to 64-byte object <02-00 00-00 BE-BE BE-BE C0-6B 05-00 C0-60 00-00 73-00 00-00 00-00 00-00 73-00 00-00 00-00 00-00 BE-BE BE-BE BE-BE BE-BE 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>
           Actual: 64-byte object <02-00 00-00 00-00 00-00 C0-45 08-00 B0-60 00-00 65-00 00-00 00-00 00-00 65-00 00-00 00-00 00-00 C4-A9 AE-66 00-10 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>
         Expected: to be called once
           Actual: never called - unsatisfied and active
/arrow/cpp/src/arrow/flight/sql/client_test.cc:152: Failure
Actual function call count doesn't match EXPECT_CALL(sql_client_, GetFlightInfo(Ref(call_options_), descriptor))...
         Expected: to be called once
           Actual: never called - unsatisfied and active
[  FAILED  ] TestFlightSqlClient.TestGetDbSchemas (1 ms)

The error can be seen here: https://github.com/apache/arrow/runs/7297442828?check_suite_focus=true

This is the initial PR that triggered it:

#13548

 

Reporter: Raúl Cumplido / @raulcd
Assignee: David Li / @lidavidm

Related issues:

• [CI] Enable Flight and Flight SQL on ubuntu cpp 20.04 builds (blocks)
• [C++] Flight integration tests fail on verify rc nightly on linux amd64 (supercedes)

PRs and other links:

• GitHub Pull Request #13599

_{Note: This issue was originally created as ARROW-17051. Please see the migration documentation for further details.}

[asfimport]

Raúl Cumplido / @raulcd:
cc @lidavidm  

[asfimport]

David Li / @lidavidm:
Thanks for the help Raúl. I have it on my list to take a look this week

[asfimport]

David Li / @lidavidm:
Ok, it only occurs with bundled (static) Protobuf/gRPC. It's not related to ASAN/UBSAN, this will do it:

-DARROW_FLIGHT=ON -DARROW_FLIGHT_SQL=ON -DARROW_BUILD_TESTS=ON -DProtobuf_SOURCE=BUNDLED -DgRPC_SOURCE=BUNDLED -DGTest_SOURCE=BUNDLED -DARROW_BUILD_SHARED=ON -DARROW_BUILD_STATIC=OFF 

It also fails differently when only a single test is run.

I suspect that gRPC/Protobuf is getting linked twice, which is a common issue. Both libarrow_flight and libarrow_flight_sql contain Protobuf symbols. env LD_DEBUG=all shows the dynamic linker is not resolving any Protobuf symbols - so presumably each library is using its own copy of Protobuf. But Protobuf has global state.

To wit, it passes if we set -DARROW_BUILD_SHARED=OFF -DARROW_BUILD_STATIC=ON instead.

So I think the solution here is: change this job to link statically instead of dynamically, and prevent Flight from building shared libraries if Protobuf/gRPC are static dependencies.

[asfimport]

Kouhei Sutou / @kou:
Issue resolved by pull request 13599
#13599