Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-15987: [C++][FlightRPC] Work around arrow-flight-test crash on AppVeyor #12683

Closed
wants to merge 1 commit into from

Conversation

lidavidm
Copy link
Member

@lidavidm lidavidm commented Mar 21, 2022

Unable to root cause. Poking around with a debugger, MSVC appears to generate wild code that calls into seemingly random functions, causing a crash. Holding back grpc-cpp does avoid the crash, so it seems to be something about the new gRPC version.

@lidavidm
Copy link
Member Author

lidavidm commented Mar 21, 2022

What I found with grpc-cpp=1.45.0:
VirtualBox_WinDev2202Eval_21_03_2022_18_17_53
VirtualBox_WinDev2202Eval_21_03_2022_18_18_25

@github-actions
Copy link

@lidavidm lidavidm changed the title ARROW-15987: [C++][FlightRPC] Attempt to work around AppVeyor failure ARROW-15987: [C++][FlightRPC] Work around arrow-flight-test crash on AppVeyor Mar 22, 2022
@lidavidm
Copy link
Member Author

The difference in packages installed is basically only grpc-cpp:

--- bad.txt	2022-03-22 07:56:02.589401042 -0400
+++ good.txt	2022-03-22 07:55:51.369332834 -0400
@@ -43,8 +43,8 @@
   + gflags                                2.2.2  ha925a31_1004       conda-forge/win-64       80 KB
   + glog                                  0.5.0  h4797de2_0          conda-forge/win-64       90 KB
   + gmock                                1.11.0  h2d74725_0          conda-forge/win-64        8 MB
-  + google-cloud-cpp                     1.38.0  h178cbb0_1          conda-forge/win-64       63 MB
-  + grpc-cpp                             1.45.0  h7b4b439_0          conda-forge/win-64       26 MB
+  + google-cloud-cpp                     1.38.0  haf58ea7_0          conda-forge/win-64       63 MB
+  + grpc-cpp                             1.44.0  h7b4b439_1          conda-forge/win-64       24 MB
   + gtest                                1.11.0  h2d74725_0          conda-forge/win-64        5 MB
   + hypothesis                           6.39.4  pyhd8ed1ab_0        conda-forge/noarch      256 KB
   + idna                                    3.3  pyhd8ed1ab_0        conda-forge/noarch       55 KB
@@ -61,7 +61,7 @@
   + libcurl                              7.82.0  h789b8ee_0          conda-forge/win-64      301 KB
   + libffi                                3.4.2  h8ffe710_5          conda-forge/win-64       41 KB
   + libglib                              2.70.2  h3be07f2_4          conda-forge/win-64        3 MB
-  + libgoogle-cloud                      1.38.0  h178cbb0_1          conda-forge/win-64       10 KB
+  + libgoogle-cloud                      1.38.0  haf58ea7_0          conda-forge/win-64       10 KB
   + libiconv                               1.16  he774522_0          conda-forge/win-64      Cached
   + liblapack                             3.9.0  5_hd5c7e75_netlib   conda-forge/win-64        3 MB
   + libllvm8                              8.0.0  2                   conda-forge/win-64       22 KB

@pitrou
Copy link
Member

pitrou commented Mar 22, 2022

Perhaps you can rebuild in RelWithDebInfo mode to get better diagnostics?

@lidavidm
Copy link
Member Author

I've done some more digging in RelWithDebInfo (see Jira) and it seems this is a regression with gRPC 1.45, that said, minimizing this into a reproducible example may take a while, this should get CI working for now

@lidavidm
Copy link
Member Author

Reported upstream at grpc/grpc#29185

Copy link
Member

@pitrou pitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, let's merge to get passing CI and avoid skipping the Python tests

@pitrou pitrou closed this in 6921741 Mar 23, 2022
@ursabot
Copy link

ursabot commented Mar 23, 2022

Benchmark runs are scheduled for baseline = 5073d63 and contender = 6921741. 6921741 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.17% ⬆️0.04%] test-mac-arm
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.26% ⬆️0.0%] ursa-thinkcentre-m75q
Buildkite builds:
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/358| 6921741d ec2-t3-xlarge-us-east-2>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/345| 6921741d test-mac-arm>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/337| 6921741d ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/355| 6921741d ursa-thinkcentre-m75q>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/357| 5073d636 ec2-t3-xlarge-us-east-2>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/344| 5073d636 test-mac-arm>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/336| 5073d636 ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/354| 5073d636 ursa-thinkcentre-m75q>
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@lidavidm lidavidm deleted the arrow-15987 branch June 14, 2022 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants