Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-14554: [C++][CI] Fix OSS-Fuzz build failure #11583

Closed
wants to merge 3 commits into from
Closed

ARROW-14554: [C++][CI] Fix OSS-Fuzz build failure #11583

wants to merge 3 commits into from

Conversation

pitrou
Copy link
Member

@pitrou pitrou commented Nov 1, 2021 •
edited
Loading

ARROW-14393 downgraded the bundled GTest version to a version that doesn't build on the OSS-Fuzz setup
(and there doesn't seem to be an obvious fix or workaround around those build errors).

Re-bump the bundled GTest version, and try to fix the issue in the Windows verification script differently.

Also, add a Crossbow task for the OSS-Fuzz build check, since the Github Actions cron job failure went unnoticed.

@pitrou
Copy link
Member Author

pitrou commented Nov 1, 2021

@github-actions crossbow submit -g cpp

@github-actions
Copy link

github-actions bot commented Nov 1, 2021

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW

Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

@pitrou pitrou changed the title [CI] [WIP] Always compile GTest from source [CI] Always compile GTest from source Nov 1, 2021
@pitrou pitrou marked this pull request as draft November 1, 2021 15:39
@github-actions
Copy link

github-actions bot commented Nov 1, 2021

Revision: 209b1e6

Submitted crossbow builds: ursacomputing/crossbow @ actions-1063

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-debian-10-cpp Github Actions
test-debian-11-cpp Github Actions
test-fedora-33-cpp Github Actions
test-ubuntu-18.04-cpp Github Actions
test-ubuntu-18.04-cpp-release Github Actions
test-ubuntu-18.04-cpp-static Github Actions
test-ubuntu-20.04-cpp Github Actions
test-ubuntu-20.04-cpp-14 Github Actions
test-ubuntu-20.04-cpp-17 Github Actions
test-ubuntu-20.04-cpp-bundled Github Actions
test-ubuntu-20.04-cpp-thread-sanitizer Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 1, 2021

@github-actions crossbow submit test-conda-*

@github-actions
Copy link

github-actions bot commented Nov 1, 2021

Revision: 4da0482

Submitted crossbow builds: ursacomputing/crossbow @ actions-1064

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 1, 2021

@github-actions crossbow submit test-conda-*

@pitrou
Copy link
Member Author

pitrou commented Nov 1, 2021

@github-actions crossbow submit -g verify-rc-source

@github-actions
Copy link

github-actions bot commented Nov 1, 2021

Revision: 70a5ce2

Submitted crossbow builds: ursacomputing/crossbow @ actions-1065

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 1, 2021

@github-actions crossbow submit vcpkg

@github-actions
Copy link

github-actions bot commented Nov 1, 2021

Revision: 70a5ce2

Submitted crossbow builds: ursacomputing/crossbow @ actions-1066

Task Status
verify-rc-source-cpp-linux-amd64 Github Actions
verify-rc-source-cpp-macos-amd64 Github Actions
verify-rc-source-cpp-macos-arm64 Github Actions
verify-rc-source-csharp-linux-amd64 Github Actions
verify-rc-source-csharp-macos-amd64 Github Actions
verify-rc-source-csharp-macos-arm64 Github Actions
verify-rc-source-go-linux-amd64 Github Actions
verify-rc-source-go-macos-amd64 Github Actions
verify-rc-source-go-macos-arm64 Github Actions
verify-rc-source-integration-linux-amd64 Github Actions
verify-rc-source-integration-macos-amd64 Github Actions
verify-rc-source-integration-macos-arm64 Github Actions
verify-rc-source-java-linux-amd64 Github Actions
verify-rc-source-java-macos-amd64 Github Actions
verify-rc-source-js-linux-amd64 Github Actions
verify-rc-source-js-macos-amd64 Github Actions
verify-rc-source-js-macos-arm64 Github Actions
verify-rc-source-python-linux-amd64 Github Actions
verify-rc-source-python-macos-amd64 Github Actions
verify-rc-source-python-macos-arm64 Github Actions
verify-rc-source-ruby-linux-amd64 Github Actions
verify-rc-source-ruby-macos-amd64 Github Actions
verify-rc-source-ruby-macos-arm64 Github Actions
verify-rc-source-windows Github Actions

@github-actions
Copy link

github-actions bot commented Nov 1, 2021

Revision: 70a5ce2

Submitted crossbow builds: ursacomputing/crossbow @ actions-1067

Task Status
test-build-vcpkg-win Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 1, 2021

Hmm, it appears we cannot exercise the release verification script using crossbow (it tries to fetch Arrow 1.0.0)? @kszucs

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit test-conda-*

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

Revision: a9cd09f

Submitted crossbow builds: ursacomputing/crossbow @ actions-1068

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit -g fuzz

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit -g fuzz

@pitrou pitrou changed the title [CI] Always compile GTest from source ARROW-14554: [C++][CI] Fix OSS-Fuzz build failure Nov 2, 2021
@github-actions
Copy link

github-actions bot commented Nov 2, 2021

https://issues.apache.org/jira/browse/ARROW-14554

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit -g fuzz

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

Revision: 7cf7cd6

Submitted crossbow builds: ursacomputing/crossbow @ actions-1069

Task Status
test-build-cpp-fuzz Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit -g fuzz

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit -g fuzz

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

Revision: 860317f

Submitted crossbow builds: ursacomputing/crossbow @ actions-1072

Task Status
test-build-cpp-fuzz Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit -g fuzz

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

Revision: 9097177

Submitted crossbow builds: ursacomputing/crossbow @ actions-1073

Task Status
test-build-cpp-fuzz Github Actions

@pitrou
ARROW-14554: [C++][CI] Fix OSS-Fuzz build failure
f356fd1 
ARROW-14393 downgraded the bundled GTest version to a version that doesn't build on the OSS-Fuzz setup
(and there doesn't seem to be an obvious fix or workaround around those build errors).

Re-bump the bundled GTest version, and try to fix the issue in the Windows verification script differently.

Also, add a Crossbow task for the OSS-Fuzz build check, since the Github Actions cron job failure went unnoticed.
@pitrou pitrou marked this pull request as ready for review November 2, 2021 15:52
@pitrou pitrou requested a review from kszucs November 2, 2021 15:52
@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit test-conda-*

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

Revision: 0bc448e

Submitted crossbow builds: ursacomputing/crossbow @ actions-1074

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit test-conda-*

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit test-conda-cpp-valgrind

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

Revision: e035ed5

Submitted crossbow builds: ursacomputing/crossbow @ actions-1075

Task Status
test-conda-cpp-valgrind Azure

@pitrou
Copy link
Member Author

pitrou commented Nov 3, 2021

@github-actions crossbow submit test-conda-cpp-valgrind

@github-actions
Copy link

github-actions bot commented Nov 3, 2021

Revision: e035ed5

Submitted crossbow builds: ursacomputing/crossbow @ actions-1077

Task Status
test-conda-cpp-valgrind Azure

@pitrou
Copy link
Member Author

pitrou commented Nov 3, 2021

@kszucs Can we merge this? I'd like to unblock the OSS-Fuzz builds.

@kszucs
Copy link
Member

kszucs commented Nov 3, 2021

Hmm, it appears we cannot exercise the release verification script using crossbow (it tries to fetch Arrow 1.0.0)? @kszucs

We can only build releases with those tasks by providing additional parameters: #11511 (comment)

@kszucs
Copy link
Member

kszucs commented Nov 3, 2021

Re-bump the bundled GTest version, and try to fix the issue in the Windows verification script differently.

Do we have a JIRA for that?

@pitrou
Copy link
Member Author

pitrou commented Nov 3, 2021

I don't understand the question. What do you want a JIRA for?

@kszucs kszucs closed this in 64b80dd Nov 3, 2021
@pitrou pitrou deleted the oss-fuzz-gtest branch November 3, 2021 15:48
@ursabot
Copy link

ursabot commented Nov 3, 2021 •
edited
Loading

Benchmark runs are scheduled for baseline = 09b79a1 and contender = 64b80dd. 64b80dd is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️1.03% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.36% ⬆️0.04%] ursa-thinkcentre-m75q
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True

@kszucs
Copy link
Member

kszucs commented Nov 3, 2021

I don't understand the question. What do you want a JIRA for?

Does this fix the windows verification issue as well?

@pitrou
Copy link
Member Author

pitrou commented Nov 3, 2021 •
edited
Loading

Does this fix the windows verification issue as well?

Hopefully, but I have no way to verify it (as explained above :-)).

@pitrou
Copy link
Member Author

pitrou commented Nov 3, 2021

(see diff in https://github.com/apache/arrow/pull/11583/files#diff-6c1e203cf55c63a1f177b4b0c97dc1aa62ea5eed5bd8e017e70703ddf119e343L90-L92)

@asfimport asfimport mentioned this pull request Nov 4, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kszucs kszucs kszucs approved these changes

Assignees
No one assigned
Labels
Component: C++
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@pitrou @kszucs @ursabot