[CI][Release] Integration tests on source verification for almalinux and Ubuntu fail

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

Some integration tests have been failing for the last three months on Ubuntu and Almalinux:

verify-rc-source-integration-linux-almalinux-8-amd64
verify-rc-source-integration-linux-ubuntu-20.04-amd64

################# FAILURES #################
FAILED TEST: session_options C++ producing,  C++ consuming
<class 'RuntimeError'>: Command failed: /tmp/arrow-HEAD.dnpwB/cpp-build/release/flight-test-integration-client -host localhost -port=46559 -scenario session_options
With output:
--------------
-- Arrow Fatal Error --
Invalid: res4 incorrect: <GetSessionOptionsResult session_options={[bardouble]: 456, [big_ol_string_list]: ["a", "b", "sea", "dee", " ", "  ", "geee", "(づ｡◕‿‿◕｡)づ"]}>

--------------


FAILED TEST: session_options C++ producing,  Java consuming
<class 'RuntimeError'>: Command failed: java -Dio.netty.tryReflectionSetAccessible=true -Darrow.struct.conflict.policy=CONFLICT_APPEND -XX:-UsePerfData --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED --add-reads=org.apache.arrow.flight.core=ALL-UNNAMED -cp /arrow/java/flight/flight-integration-tests/target/flight-integration-tests-16.0.0-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.flight.integration.tests.IntegrationTestClient -port 36453 -scenario session_options
With output:
--------------
WARNING: Unknown module: org.apache.arrow.flight.core specified to --add-reads
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
Exception in thread "main" java.lang.AssertionError: Expected:
{lol_invalid=org.apache.arrow.flight.SetSessionOptionsResult$Error@21a21c64, key_with_invalid_value=org.apache.arrow.flight.SetSessionOptionsResult$Error@7803bfd}
but got:
{lol_invalid=org.apache.arrow.flight.SetSessionOptionsResult$Error@21a21c64}
	at org.apache.arrow.flight.integration.tests.IntegrationAssertions.assertEquals(IntegrationAssertions.java:59)
	at org.apache.arrow.flight.integration.tests.SessionOptionsScenario.client(SessionOptionsScenario.java:75)
	at org.apache.arrow.flight.integration.tests.IntegrationTestClient.run(IntegrationTestClient.java:98)
	at org.apache.arrow.flight.integration.tests.IntegrationTestClient.main(IntegrationTestClient.java:69)

--------------


FAILED TEST: session_options C++ producing,  Go consuming
<class 'RuntimeError'>: Command failed: /tmp/arrow-HEAD.dnpwB/go/gopath/bin/arrow-flight-integration-client -host localhost -port=41835 -scenario session_options
With output:
--------------
panic: errors expected: 2, got: 1

goroutine 1 [running]:
main.main()
	/arrow/go/arrow/internal/flight_integration/cmd/arrow-flight-integration-client/main.go:52 +0x28b

--------------


FAILED TEST: session_options Java producing,  C++ consuming
<class 'RuntimeError'>: Command failed: /tmp/arrow-HEAD.dnpwB/cpp-build/release/flight-test-integration-client -host localhost -port=44073 -scenario session_options
With output:
--------------
-- Arrow Fatal Error --
Invalid: res1 incorrect: <SetSessionOptionsResult errors={[lol_invalid]: {InvalidName}}>

--------------


FAILED TEST: session_options Go producing,  C++ consuming
<class 'RuntimeError'>: Command failed: /tmp/arrow-HEAD.dnpwB/cpp-build/release/flight-test-integration-client -host localhost -port=33683 -scenario session_options
With output:
--------------
-- Arrow Fatal Error --
Invalid: res1 incorrect: <SetSessionOptionsResult errors={[lol_invalid]: {InvalidName}}>

--------------

I am not entirely sure if this warning is the relevant bit:

WARNING: Unknown module: org.apache.arrow.flight.core specified to --add-reads
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens

This has been failing since those commits were introduced (55afcf0...05b8f36).

Component(s)

Continuous Integration, Release

[raulcd]

I am having maven issues on my docker containers reproducing locally. I will keep investigating.
CC @lidavidm @kou @zeroshade this is the only issue currently blocking me from creating a RC. Even though there are still some bugs and issues that are tagged as 16.0.0 this is the current real release blocker as far as I understand.

[raulcd]

CC @pitrou

[lidavidm]

This only fails on this platform?

[lidavidm]

It seems like the C++ side is the common issue here

[raulcd]

In the maintenance branch it fails on Ubuntu 20.04, Ubuntu 22.04 and Almalinux 8, see: #41118 (comment)
It does pass on conda and macOS.

edit Ubuntu version

[lidavidm]

we appear to have been bitten by https://stackoverflow.com/questions/44086269/why-does-my-variant-convert-a-stdstring-to-a-bool though I wonder why some platforms appear to do it "correctly" (incorrectly?)

[lidavidm]

wow, I'm not sure how these pass in the first place

[raulcd]

Issue resolved by pull request 41202
#41202