Skip to content

Commit

Permalink
Remove centos7 and devtoolset hacks for R
Browse files Browse the repository at this point in the history
  • Loading branch information
jonkeane committed Apr 29, 2024
1 parent 6536162 commit 020f8e1
Show file tree
Hide file tree
Showing 10 changed files with 4 additions and 81 deletions.
3 changes: 0 additions & 3 deletions .env
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,6 @@ ARROW_R_DEV=TRUE
R_PRUNE_DEPS=FALSE
TZ=UTC

# Any non-empty string will install devtoolset-${DEVTOOLSET_VERSION}
DEVTOOLSET_VERSION=

# Used through docker-compose.yml and serves as the default version for the
# ci/scripts/install_vcpkg.sh script. Prefer to use short SHAs to keep the
# docker tags more readable.
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/r.yml
Original file line number Diff line number Diff line change
Expand Up @@ -192,12 +192,11 @@ jobs:
fail-fast: false
matrix:
config:
- { org: "rhub", image: "ubuntu-gcc12", tag: "latest", devtoolset: "" }
- { org: "rhub", image: "ubuntu-gcc12", tag: "latest" }
env:
R_ORG: ${{ matrix.config.org }}
R_IMAGE: ${{ matrix.config.image }}
R_TAG: ${{ matrix.config.tag }}
DEVTOOLSET_VERSION: ${{ matrix.config.devtoolset }}
steps:
- name: Checkout Arrow
uses: actions/checkout@3df4ab11eba7bda6032a0b82a6bb43b11571feac # v4.0.0
Expand Down
3 changes: 0 additions & 3 deletions ci/docker/linux-r.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,6 @@ ENV R_BIN=${r_bin}
ARG r_dev=FALSE
ENV ARROW_R_DEV=${r_dev}

ARG devtoolset_version=
ENV DEVTOOLSET_VERSION=${devtoolset_version}

ARG r_prune_deps=FALSE
ENV R_PRUNE_DEPS=${r_prune_deps}

Expand Down
3 changes: 0 additions & 3 deletions ci/scripts/java_jni_manylinux_build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,6 @@ echo "=== Clear output directories and leftovers ==="
rm -rf ${build_dir}

echo "=== Building Arrow C++ libraries ==="
devtoolset_version=$(rpm -qa "devtoolset-*-gcc" --queryformat %{VERSION} | \
grep -o "^[0-9]*")
devtoolset_include_cpp="/opt/rh/devtoolset-${devtoolset_version}/root/usr/include/c++/${devtoolset_version}"
: ${ARROW_ACERO:=ON}
export ARROW_ACERO
: ${ARROW_BUILD_TESTS:=ON}
Expand Down
20 changes: 0 additions & 20 deletions ci/scripts/r_docker_configure.sh
Original file line number Diff line number Diff line change
Expand Up @@ -67,26 +67,6 @@ sloppiness = include_file_ctime
hash_dir = false" >> ~/.ccache/ccache.conf
fi

# Special hacking to try to reproduce quirks on centos using non-default build
# tooling.
if [[ -n "$DEVTOOLSET_VERSION" ]]; then
$PACKAGE_MANAGER install -y centos-release-scl
$PACKAGE_MANAGER install -y "devtoolset-$DEVTOOLSET_VERSION"

# Enable devtoolset here so that `which gcc` finds the right compiler below
source /opt/rh/devtoolset-${DEVTOOLSET_VERSION}/enable

# Build images which require the devtoolset don't have CXX17 variables
# set as the system compiler doesn't support C++17
if [ ! "`{R_BIN} CMD config CXX17`" ]; then
mkdir -p ~/.R
echo "CC = $(which gcc) -fPIC" >> ~/.R/Makevars
echo "CXX17 = $(which g++) -fPIC" >> ~/.R/Makevars
echo "CXX17STD = -std=c++17" >> ~/.R/Makevars
echo "CXX17FLAGS = ${CXX11FLAGS}" >> ~/.R/Makevars
fi
fi

if [ -f "${ARROW_SOURCE_HOME}/ci/scripts/r_install_system_dependencies.sh" ]; then
"${ARROW_SOURCE_HOME}/ci/scripts/r_install_system_dependencies.sh"
fi
Expand Down
1 change: 0 additions & 1 deletion dev/tasks/r/azure.linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ jobs:
export R_ORG={{ r_org }}
export R_IMAGE={{ r_image }}
export R_TAG={{ r_tag }}
export DEVTOOLSET_VERSION={{ devtoolset_version|default("") }}
export R_CUSTOM_CCACHE={{ r_custom_ccache|default("false") }}
docker-compose pull --ignore-pull-failures r
docker-compose build r
Expand Down
7 changes: 3 additions & 4 deletions dev/tasks/r/github.packages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -299,14 +299,14 @@ jobs:
# choosing a binary on this OS. If libarrow_binary is TRUE, we're on
# an OS that is not in the allowlist, so we have to opt-in to use the
# binary. Other env vars used in r_docker_configure.sh can be added
# here (like devtoolset) and wired up in the later steps.
# here and wired up in the later steps.
- {image: "rhub/ubuntu-clang", libarrow_binary: "TRUE"}
# fedora-clang-devel cannot use binaries bc of libc++ (uncomment to see the error)
# - {image: "rhub/fedora-clang-devel", libarrow_binary: "TRUE"}
- {image: "rhub/ubuntu-release"} # currently ubuntu-22.04
- {image: "rocker/r-ver:4.0.0"} # ubuntu-20.04
- {image: "rstudio/r-base:4.1-focal"} # ubuntu-20.04
- {image: "rstudio/r-base:4.2-centos7", devtoolset: "8"}
- {image: "rstudio/r-base:4.1-focal"}
- {image: "rstudio/r-base:4.2-jammy"}
- {image: "rstudio/r-base:4.3-noble"}
steps:
# Get the arrow checkout just for the docker config scripts
Expand All @@ -317,7 +317,6 @@ jobs:
- name: Install system requirements
env:
ARROW_R_DEV: "TRUE" # To install curl/openssl in r_docker_configure.sh
DEVTOOLSET_VERSION: {{ '${{ matrix.config.devtoolset }}' }}
shell: bash
run: |
# Make sure R is on the path for the R-hub devel versions (where RPREFIX is set in its dockerfile)
Expand Down
9 changes: 0 additions & 9 deletions dev/tasks/tasks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1411,15 +1411,6 @@ tasks:
GCC_VERSION: 12
image: ubuntu-r-only-r

test-r-rstudio-r-base-4.2-centos7-devtoolset-8:
ci: azure
template: r/azure.linux.yml
params:
r_org: rstudio
r_image: r-base
r_tag: 4.2-centos7
devtoolset_version: 8

test-r-minimal-build:
ci: azure
template: r/azure.linux.yml
Expand Down
3 changes: 0 additions & 3 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1470,7 +1470,6 @@ services:
args:
base: ${R_ORG}/${R_IMAGE}:${R_TAG}
r_dev: ${ARROW_R_DEV}
devtoolset_version: ${DEVTOOLSET_VERSION}
tz: ${TZ}
r_prune_deps: ${R_PRUNE_DEPS}
r_custom_ccache: ${R_CUSTOM_CCACHE}
Expand All @@ -1482,7 +1481,6 @@ services:
ARROW_R_DEV: ${ARROW_R_DEV}
# To test for CRAN release, delete ^^ these two env vars so we download the Apache release
ARROW_USE_PKG_CONFIG: "false"
devtoolset_version: ${DEVTOOLSET_VERSION}
volumes:
- .:/arrow:delegated
command: >
Expand Down Expand Up @@ -1526,7 +1524,6 @@ services:
# TODO: change this to rhub/clang-asan
base: rhub/fedora-clang-devel-san
r_dev: ${ARROW_R_DEV}
devtoolset_version: ${DEVTOOLSET_VERSION}
r_bin: RDsan
tz: ${TZ}
r_prune_deps: ${R_PRUNE_DEPS}
Expand Down
33 changes: 0 additions & 33 deletions r/vignettes/install.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -28,35 +28,6 @@ For `gcc`, this generally means version 7 or newer. Most contemporary Linux
distributions have a new enough compiler; however, CentOS 7 is a notable
exception, as it ships with gcc 4.8.

If you are on CentOS 7, to build arrow you will need to install a newer `devtoolset`, and you'll need to update R's Makevars to define the `CXX17` variables. This script installs `devtoolset-8` and configures R to be able to use C++17:

```
#!/usr/bin/env bash
yum install -y centos-release-scl
yum install -y devtoolset-8
# Optional: also install cloud storage dependencies, as described below
yum install -y libcurl-devel openssl-devel
source /opt/rh/devtoolset-8/enable
if [ ! `R CMD config CXX17` ]; then
mkdir -p ~/.R
echo "CC = $(which gcc) -fPIC" >> ~/.R/Makevars
echo "CXX17 = $(which g++) -fPIC" >> ~/.R/Makevars
echo "CXX17STD = -std=c++17" >> ~/.R/Makevars
echo "CXX17FLAGS = ${CXX11FLAGS}" >> ~/.R/Makevars
fi
```

Note that the C++17 compiler is only required at *build* time. You don't need
to enable the devtoolset every time you load the package. What's more, if you
install a binary package from RStudio Package Manager (see method 1a below), you
do not need to set up any of this. Likewise, if you `R CMD INSTALL --build`
arrow on a CentOS machine with the newer compilers, you can take the binary
package it produces and install it on any other CentOS machine without those
compilers.

### Libraries

Optional support for reading from cloud storage--AWS S3 and
Expand Down Expand Up @@ -517,10 +488,6 @@ The install script should work everywhere, so if libarrow fails to compile,
please [report an issue](https://issues.apache.org/jira/projects/ARROW/issues)
so that we can improve the script.

### Known installation issues

* On CentOS, building the package requires a more modern `devtoolset` than the default system compilers. See "System dependencies" above.

## Contributing

We are constantly working to make the installation process as painless as
Expand Down

0 comments on commit 020f8e1

Please sign in to comment.