Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-17545: [C++][CI] Mandate C++17 instead of C++11 #13991

Merged
merged 2 commits into from
Sep 13, 2022

Conversation

pitrou
Copy link
Member

@pitrou pitrou commented Aug 29, 2022

This PR switches our build system to require C++17 instead of C++11.

Because the conda packaging jobs are out of sync with the conda-forge files, the Windows conda packaging jobs are broken with this change. The related task (sync conda packaging files with conda-forge) is tracked in ARROW-17635.

@github-actions
Copy link

@github-actions
Copy link

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

@pitrou

This comment was marked as outdated.

@pitrou
Copy link
Member Author

pitrou commented Aug 29, 2022

@kou It seems the centos-7-amd64 job uses gcc 4.8.5, should we switch to the devtoolset instead?

@pitrou
Copy link
Member Author

pitrou commented Aug 29, 2022

The packaging jobs fail at the "upload artifacts" step with a weird error:

archery.crossbow.core.CrossbowError: Unable to parse the github owner and repository from the repository's remote url 'https://github.com/ursacomputing/crossbow/'

https://github.com/ursacomputing/crossbow/runs/8075587016?check_suite_focus=true#step:11:47

@raulcd Do you know where that might come from?

@raulcd
Copy link
Member

raulcd commented Aug 29, 2022

archery.crossbow.core.CrossbowError: Unable to parse the github owner and repository from the repository's remote url 'https://github.com/ursacomputing/crossbow/'

https://github.com/ursacomputing/crossbow/runs/8075587016?check_suite_focus=true#step:11:47

@raulcd Do you know where that might come from?

I am not entirely sure why the remote is different but on a successful nightly job https://github.com/ursacomputing/crossbow/runs/8066802928?check_suite_focus=true#step:11:4 the archery command uses: --queue-remote https://github.com/ursacomputing/crossbow but on your job --queue-remote https://github.com/ursacomputing/crossbow/ making the regex to select the user repo fail:

>>> remote_url = "https://github.com/ursacomputing/crossbow"
>>> m = re.match(r'.*\/([^\/]+)\/([^\/\.]+)(\.git)?$', remote_url)
>>> m
<re.Match object; span=(0, 41), match='https://github.com/ursacomputing/crossbow'>
>>> remote_url = "https://github.com/ursacomputing/crossbow/"
>>> m = re.match(r'.*\/([^\/]+)\/([^\/\.]+)(\.git)?$', remote_url)
>>> m

Did you use https://github.com/ursacomputing/crossbow/ on the --queue-remote when submitting the jobs? We probably should improve the regex

@pitrou
Copy link
Member Author

pitrou commented Aug 29, 2022

Did you use https://github.com/ursacomputing/crossbow/ on the --queue-remote when submitting the jobs?

I did not use anything, I let archery crossbow submit use its own defaults.

@pitrou

This comment was marked as outdated.

@pitrou

This comment was marked as outdated.

@kou
Copy link
Member

kou commented Aug 29, 2022

@kou It seems the centos-7-amd64 job uses gcc 4.8.5, should we switch to the devtoolset instead?

Sure. The following patch will work:

diff --git a/dev/tasks/linux-packages/apache-arrow/yum/centos-7/Dockerfile b/dev/tasks/linux-packages/apache-arrow/yum/centos-7/Dockerfile
index 04e74012f9..1da8e0fb79 100644
--- a/dev/tasks/linux-packages/apache-arrow/yum/centos-7/Dockerfile
+++ b/dev/tasks/linux-packages/apache-arrow/yum/centos-7/Dockerfile
@@ -18,13 +18,20 @@
 ARG FROM=centos:7
 FROM ${FROM}
 
+ENV \
+  SCL=devtoolset-11
+
 ARG DEBUG
 
 RUN \
   quiet=$([ "${DEBUG}" = "yes" ] || echo "--quiet") && \
   yum update -y ${quiet} && \
-  yum install -y ${quiet} epel-release && \
   yum install -y ${quiet} \
+    centos-release-scl-rh \
+    epel-release && \
+  yum install -y ${quiet} \
+    ${SCL}-gcc-c++ \
+    ${SCL}-make \
     bison \
     boost169-devel \
     brotli-devel \
@@ -33,7 +40,6 @@ RUN \
     cmake3 \
     curl-devel \
     flex \
-    gcc-c++ \
     gflags-devel \
     git \
     glog-devel \
@@ -42,7 +48,6 @@ RUN \
     json-devel \
     libzstd-devel \
     lz4-devel \
-    make \
     ninja-build \
     openssl-devel \
     pkg-config \

@pitrou

This comment was marked as outdated.

@pitrou
Copy link
Member Author

pitrou commented Aug 30, 2022

@kou The CentOS 7 patched build now fails with:

error: Failed build dependencies:
	gcc-c++ is needed by arrow-10.0.0.dev145-1.el7.x86_64

https://github.com/ursacomputing/crossbow/runs/8088068865?check_suite_focus=true#step:6:2097

Would you like to suggest or push a fix?

@kou
Copy link
Member

kou commented Aug 30, 2022

Oh, sorry. I'll push a fix to this branch later.

@pitrou

This comment was marked as outdated.

@pitrou pitrou marked this pull request as ready for review September 7, 2022 13:50
@h-vetinari
Copy link
Contributor

@h-vetinari As I expected, your suggestion failed:

Weeeeeeell... Clang wasn't in the pinning I commented on - and there's no incompatibility from the POV of conda(-forge). In other words, this is not going to be fixed by synching the pinnings. It's due to the combination of:

  • requiring clang on windows (I don't really understand this need; at least it's very uncommon...)
  • pinning to clang 10 (which is 2.5 years old and thus apparently incompatible with vs2019)

@pitrou
Copy link
Member Author

pitrou commented Sep 7, 2022

It's due to the combination of:

* requiring clang on windows (I don't really understand this need; at least it's very uncommon...)

* pinning to clang 10 (which is 2.5 years old and thus apparently incompatible with vs2019)

This is probably due to Gandiva which compiles some files to LLVM bitcode. In any case, I won't debug this any further. @xhochy is the primary maintainer of conda files and he can fix them when/if desired :-)

@pitrou

This comment was marked as outdated.

@pitrou

This comment was marked as outdated.

@pitrou
Copy link
Member Author

pitrou commented Sep 8, 2022

(rebased)

@pitrou
Copy link
Member Author

pitrou commented Sep 8, 2022

Hmm, I think I need to investigate the "Sphinx & Numpydoc" CI failure to see if it's caused by these changes.

Copy link
Member

@kou kou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

.github/workflows/r.yml Outdated Show resolved Hide resolved
cpp/src/gandiva/CMakeLists.txt Outdated Show resolved Hide resolved
@pitrou

This comment was marked as outdated.

@pitrou
Copy link
Member Author

pitrou commented Sep 12, 2022

Revision: fb53ff7

Submitted crossbow builds: ursacomputing/crossbow @ cxx17-20

Task Status
almalinux-8-amd64 Github Actions
almalinux-8-arm64 TravisCI
almalinux-9-amd64 Github Actions
almalinux-9-arm64 TravisCI
amazon-linux-2-amd64 Github Actions
amazon-linux-2-arm64 TravisCI
centos-7-amd64 Github Actions
centos-8-stream-amd64 Github Actions
centos-8-stream-arm64 TravisCI
centos-9-stream-amd64 Github Actions
centos-9-stream-arm64 TravisCI
conan-maximum Github Actions
conan-minimum Github Actions
conda-clean Azure
conda-linux-gcc-py310-arm64 Azure
conda-linux-gcc-py310-cpu Azure
conda-linux-gcc-py310-cuda Azure
conda-linux-gcc-py310-ppc64le Azure
conda-linux-gcc-py37-arm64 Azure
conda-linux-gcc-py37-cpu-r40 Azure
conda-linux-gcc-py37-cpu-r41 Azure
conda-linux-gcc-py37-cuda Azure
conda-linux-gcc-py37-ppc64le Azure
conda-linux-gcc-py38-arm64 Azure
conda-linux-gcc-py38-cpu Azure
conda-linux-gcc-py38-cuda Azure
conda-linux-gcc-py38-ppc64le Azure
conda-linux-gcc-py39-arm64 Azure
conda-linux-gcc-py39-cpu Azure
conda-linux-gcc-py39-cuda Azure
conda-linux-gcc-py39-ppc64le Azure
conda-osx-arm64-clang-py310 Azure
conda-osx-arm64-clang-py38 Azure
conda-osx-arm64-clang-py39 Azure
conda-osx-clang-py310 Azure
conda-osx-clang-py37-r40 Azure
conda-osx-clang-py37-r41 Azure
conda-osx-clang-py38 Azure
conda-osx-clang-py39 Azure
conda-win-vs2019-py310 Azure
conda-win-vs2019-py37-r40 Azure
conda-win-vs2019-py37-r41 Azure
conda-win-vs2019-py38 Azure
conda-win-vs2019-py39 Azure
debian-bookworm-amd64 Github Actions
debian-bookworm-arm64 TravisCI
debian-bullseye-amd64 Github Actions
debian-bullseye-arm64 TravisCI
example-cpp-minimal-build-static Github Actions
example-cpp-minimal-build-static-system-dependency Github Actions
example-python-minimal-build-fedora-conda Github Actions
example-python-minimal-build-ubuntu-venv Github Actions
homebrew-cpp Github Actions
homebrew-r-autobrew Github Actions
homebrew-r-brew Github Actions
java-jars Github Actions
nuget Github Actions
python-sdist Github Actions
r-binary-packages Github Actions
test-alpine-linux-cpp Github Actions
test-build-cpp-fuzz Github Actions
test-build-vcpkg-win Github Actions
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions
test-debian-10-cpp-amd64 Github Actions
test-debian-10-cpp-i386 Github Actions
test-debian-11-cpp-amd64 Github Actions
test-debian-11-cpp-i386 Github Actions
test-debian-11-go-1.17 Azure
test-debian-11-python-3 Azure
test-debian-c-glib Github Actions
test-debian-ruby Github Actions
test-fedora-35-cpp Github Actions
test-fedora-35-python-3 Azure
test-fedora-r-clang-sanitizer Azure
test-r-arrow-backwards-compatibility Github Actions
test-r-depsource-bundled Azure
test-r-depsource-system Github Actions
test-r-dev-duckdb Github Actions
test-r-devdocs Github Actions
test-r-gcc-11 Github Actions
test-r-gcc-12 Github Actions
test-r-install-local Github Actions
test-r-linux-as-cran Github Actions
test-r-linux-rchk Github Actions
test-r-linux-valgrind Azure
test-r-minimal-build Azure
test-r-offline-maximal Github Actions
test-r-offline-minimal Azure
test-r-rhub-debian-gcc-devel-lto-latest Azure
test-r-rhub-debian-gcc-release-custom-ccache Azure
test-r-rhub-ubuntu-gcc-release-latest Azure
test-r-rocker-r-base-latest Azure
test-r-rstudio-r-base-4.1-opensuse153 Azure
test-r-rstudio-r-base-4.2-centos7-devtoolset-8 Azure
test-r-rstudio-r-base-4.2-focal Azure
test-r-ubuntu-22.04 Github Actions
test-r-versions Github Actions
test-skyhook-integration Github Actions
test-ubuntu-18.04-cpp Github Actions
test-ubuntu-18.04-cpp-release Github Actions
test-ubuntu-18.04-cpp-static Github Actions
test-ubuntu-18.04-r-sanitizer Azure
test-ubuntu-20.04-cpp Github Actions
test-ubuntu-20.04-cpp-17 Github Actions
test-ubuntu-20.04-cpp-bundled Github Actions
test-ubuntu-20.04-cpp-thread-sanitizer Github Actions
test-ubuntu-20.04-python-3 Azure
test-ubuntu-22.04-cpp Github Actions
test-ubuntu-c-glib Github Actions
test-ubuntu-default-docs Azure
test-ubuntu-ruby Github Actions
ubuntu-bionic-amd64 Github Actions
ubuntu-bionic-arm64 TravisCI
ubuntu-focal-amd64 Github Actions
ubuntu-focal-arm64 TravisCI
ubuntu-jammy-amd64 Github Actions
ubuntu-jammy-arm64 TravisCI
wheel-macos-big-sur-cp310-arm64 Github Actions
wheel-macos-big-sur-cp310-universal2 Github Actions
wheel-macos-big-sur-cp38-arm64 Github Actions
wheel-macos-big-sur-cp39-arm64 Github Actions
wheel-macos-big-sur-cp39-universal2 Github Actions
wheel-macos-high-sierra-cp310-amd64 Github Actions
wheel-macos-high-sierra-cp37-amd64 Github Actions
wheel-macos-high-sierra-cp38-amd64 Github Actions
wheel-macos-high-sierra-cp39-amd64 Github Actions
wheel-manylinux2014-cp310-amd64 Github Actions
wheel-manylinux2014-cp310-arm64 TravisCI
wheel-manylinux2014-cp37-amd64 Github Actions
wheel-manylinux2014-cp37-arm64 TravisCI
wheel-manylinux2014-cp38-amd64 Github Actions
wheel-manylinux2014-cp38-arm64 TravisCI
wheel-manylinux2014-cp39-amd64 Github Actions
wheel-manylinux2014-cp39-arm64 TravisCI
wheel-windows-cp310-amd64 Github Actions
wheel-windows-cp37-amd64 Github Actions
wheel-windows-cp38-amd64 Github Actions
wheel-windows-cp39-amd64 Github Actions

@pitrou
Copy link
Member Author

pitrou commented Sep 12, 2022

Hmm, I think I need to investigate the "Sphinx & Numpydoc" CI failure to see if it's caused by these changes.

It seems like this was fixed in the meantime (on git master?).

@pitrou
Copy link
Member Author

pitrou commented Sep 12, 2022

The Travis-CI s390x build times out. I don't think there's anything we can do against it, though (short of making the build optional).

@pitrou
Copy link
Member Author

pitrou commented Sep 12, 2022

Revision: 24a9099

Submitted crossbow builds: ursacomputing/crossbow @ cxx17-21

Task Status
conda-linux-gcc-py37-cpu-r40 Azure
conda-linux-gcc-py37-cpu-r41 Azure
conda-osx-clang-py37-r40 Azure
conda-osx-clang-py37-r41 Azure
conda-win-vs2019-py37-r40 Azure
conda-win-vs2019-py37-r41 Azure
homebrew-r-autobrew Github Actions
homebrew-r-brew Github Actions
r-binary-packages Github Actions
test-fedora-r-clang-sanitizer Azure
test-r-arrow-backwards-compatibility Github Actions
test-r-depsource-bundled Azure
test-r-depsource-system Github Actions
test-r-dev-duckdb Github Actions
test-r-devdocs Github Actions
test-r-gcc-11 Github Actions
test-r-gcc-12 Github Actions
test-r-install-local Github Actions
test-r-linux-as-cran Github Actions
test-r-linux-rchk Github Actions
test-r-linux-valgrind Azure
test-r-minimal-build Azure
test-r-offline-maximal Github Actions
test-r-offline-minimal Azure
test-r-rhub-debian-gcc-devel-lto-latest Azure
test-r-rhub-debian-gcc-release-custom-ccache Azure
test-r-rhub-ubuntu-gcc-release-latest Azure
test-r-rocker-r-base-latest Azure
test-r-rstudio-r-base-4.1-opensuse153 Azure
test-r-rstudio-r-base-4.2-centos7-devtoolset-8 Azure
test-r-rstudio-r-base-4.2-focal Azure
test-r-ubuntu-22.04 Github Actions
test-r-versions Github Actions
test-ubuntu-18.04-r-sanitizer Azure

@pitrou
Copy link
Member Author

pitrou commented Sep 12, 2022

I will merge if CI turns out as green as can be hoped given the current failures on git master.

@h-vetinari
Copy link
Contributor

@pitrou, just FYI, you might be interested in #14102

@pitrou pitrou merged commit 9d65981 into apache:master Sep 13, 2022
@pitrou pitrou deleted the ARROW-17545-cpp17 branch September 13, 2022 06:40
@ursabot
Copy link

ursabot commented Sep 13, 2022

Benchmark runs are scheduled for baseline = c9844f0 and contender = 9d65981. 9d65981 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.58% ⬆️0.54%] test-mac-arm
[Failed ⬇️0.85% ⬆️0.28%] ursa-i9-9960x
[Finished ⬇️4.12% ⬆️2.27%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 9d659810 ec2-t3-xlarge-us-east-2
[Failed] 9d659810 test-mac-arm
[Failed] 9d659810 ursa-i9-9960x
[Finished] 9d659810 ursa-thinkcentre-m75q
[Finished] c9844f05 ec2-t3-xlarge-us-east-2
[Failed] c9844f05 test-mac-arm
[Failed] c9844f05 ursa-i9-9960x
[Finished] c9844f05 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@ursabot
Copy link

ursabot commented Sep 13, 2022

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

zagto pushed a commit to zagto/arrow that referenced this pull request Oct 7, 2022
This PR switches our build system to require C++17 instead of C++11.

Because the conda packaging jobs are out of sync with the conda-forge files, the Windows conda packaging jobs are broken with this change. The related task (sync conda packaging files with conda-forge) is tracked in ARROW-17635.


Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
fatemehp pushed a commit to fatemehp/arrow that referenced this pull request Oct 17, 2022
This PR switches our build system to require C++17 instead of C++11.

Because the conda packaging jobs are out of sync with the conda-forge files, the Windows conda packaging jobs are broken with this change. The related task (sync conda packaging files with conda-forge) is tracked in ARROW-17635.


Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants