GH-45505: [CI][R] Use Ubuntu 22.04 instead of 20.04 as much as possible for nightly jobs #45507

kou · 2025-02-12T06:49:11Z

Rationale for this change

Ubuntu 20.04 will reach EOL on 2025-05.

What changes are included in this PR?

Use Ubuntu 22.04 instead of Ubuntu 20.04 for Apache Arrow C++ 4.0.0 or later.
Keep using Ubuntu 20.04 for Apache Arrow C++ 3.0.0 or earlier because we can't build Apache Arrow C++ 3.0.0 or earlier on Ubuntu 22.04. We can use pre-built binaries forApache Arrow C++ 3.0.0 or earlier on Ubuntu 20.04.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

GitHub Issue: [CI][R] Use Ubuntu 22.04 instead of 20.04 as much as possible for nightly jobs #45505

kou · 2025-02-12T06:49:21Z

@github-actions crossbow submit -g r

github-actions · 2025-02-12T06:49:35Z

⚠️ GitHub issue #45505 has been automatically assigned in GitHub to PR creator.

kou · 2025-02-12T06:49:59Z

dev/tasks/r/github.packages.yml

@@ -313,8 +313,8 @@ jobs:
          # fedora-clang-devel cannot use binaries bc of libc++ (uncomment to see the error)
          # - {image: "rhub/fedora-clang-devel", libarrow_binary: "TRUE"}
          - {image: "rhub/ubuntu-release"} # currently ubuntu-22.04
-          - {image: "rocker/r-ver:4.0.0"} # ubuntu-20.04
-          - {image: "rstudio/r-base:4.1-focal"}
+          - {image: "rstudio/r-base:4.0-jammy"}


Is it OK?

I assume that R 4.0 (not Ubuntu 20.04) is important here.

@jonkeane Do you have any opinion for this?

Yup, you're absolutely right that 4.0 is important and not the version — that comment was mostly to call attention to the fact that rocker/r-ver would end up being ubuntu 20.04

Thanks for confirming this!

github-actions · 2025-02-12T06:51:57Z

Revision: be959e5

Submitted crossbow builds: ursacomputing/crossbow @ actions-b1cf7ac8c9

Task	Status
r-binary-packages
r-recheck-most
test-r-arrow-backwards-compatibility
test-r-clang-sanitizer
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-macos-as-cran
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-debian-gcc-release-custom-ccache
test-r-rhub-ubuntu-release-latest
test-r-rocker-r-ver-latest
test-r-rstudio-r-base-4.1-opensuse155
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-r-sanitizer

kou · 2025-02-13T00:56:31Z

dev/tasks/r/github.linux.arrow.version.back.compat.yml

@@ -89,7 +89,7 @@ jobs:
        - { old_arrow_version: '1.0.1', r: '4.0' }


Can we remove 1.0.1 - 5.0.0?
It seems that we don't have Arrow C++ binaries for them on Ubuntu 22.04.

https://github.com/ursacomputing/crossbow/actions/runs/13279720095/job/37133898415#step:7:21

Failure ('test-read-files.R:25:3'): Can read the file (parquet) `df <- read_parquet(pq_file)` threw an error. Message: Cannot call io___MemoryMappedFile__Open(). See https://arrow.apache.org/docs/r/articles/install.html for help installing Arrow C++ libraries. Class: simpleError/error/condition Backtrace: ▆ 1. ├─testthat::expect_error(df <- read_parquet(pq_file), NA) at test-read-files.R:25:3 2. │ └─testthat:::quasi_capture(...) 3. │ ├─testthat (local) .capture(...) 4. │ │ └─base::withCallingHandlers(...) 5. │ └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo)) 6. └─arrow::read_parquet(pq_file) 7. └─arrow:::make_readable_file(file) 8. └─arrow::mmap_open(file) 9. └─arrow:::io___MemoryMappedFile__Open(path, mode)

Hmmmm, we could, but would it be possible to install them from source instead? Or is that not possible at this point?

It would be great to be able to assert that we are still backwards compatible, though in practice going back this far is probably not super important

Hmm. It seems that binary "arrow" package (not source "arrow" package) is used:

https://github.com/ursacomputing/crossbow/actions/runs/13279720095/job/37133898415#step:4:275

* installing *binary* package ‘arrow’ ...

Can we install Apache Arrow C++ from source when we use binary "arrow" package...?

jonkeane · 2025-02-14T01:02:40Z

dev/tasks/r/github.linux.arrow.version.back.compat.yml

@@ -89,7 +89,7 @@ jobs:
        - { old_arrow_version: '1.0.1', r: '4.0' }
    env:
      ARROW_R_DEV: "TRUE"
-      RSPM: "https://packagemanager.rstudio.com/cran/__linux__/focal/latest"
+      RSPM: "https://packagemanager.rstudio.com/cran/__linux__/jammy/latest"


What if we moved this later and did something like:

- name: Set RSPM if: ${{ matrix.config.use_rspm }} run: echo "RSPM =https://packagemanager.rstudio.com/cran/__linux__/jammy/latest" >> $GITHUB_ENV

And then set use_rspm in the matrix:

- { old_arrow_version: '14.0.2.1', r: '4.3', use_rspm = true } - { old_arrow_version: '13.0.0.1', r: '4.3', use_rspm = true } - { old_arrow_version: '12.0.1.1', r: '4.3', use_rspm = true } - { old_arrow_version: '11.0.0.3', r: '4.2', use_rspm = true } - { old_arrow_version: '10.0.1', r: '4.2', use_rspm = true } - { old_arrow_version: '9.0.0', r: '4.2', use_rspm = true } - { old_arrow_version: '8.0.0', r: '4.2', use_rspm = true } - { old_arrow_version: '7.0.0', r: '4.1', use_rspm = true } - { old_arrow_version: '6.0.1', r: '4.1', use_rspm = true } - { old_arrow_version: '5.0.0', r: '4.1', use_rspm = false } - { old_arrow_version: '4.0.0', r: '4.0', use_rspm = false } - { old_arrow_version: '3.0.0', r: '4.0', use_rspm = false } - { old_arrow_version: '2.0.0', r: '4.0', use_rspm = false } - { old_arrow_version: '1.0.1', r: '4.0', use_rspm = false }

That should prevent setting RSPM and then it will use source installs when that's not set. This might not work / have other issues, but it's worth trying IMHO

OK. Let's try it!

BTW, it may be better that we do this backward compatibility test in C++ not R...

it may be better that we do this backward compatibility test in C++ not R...

Yeah, totally. There are some R-specific pieces of metadata that it's nice to test, but if we did have c++ compatibility testing for the underlying parquet implementation, the decision to drop 1.0.1 - 5.0.0 testing for R does become less of a worry (yes, the R-specific metadata might not work, but that's also not so catastrophic)

Ah, I see. I didn't know that these tests have some R-specific checks.

kou · 2025-02-14T01:30:31Z

@github-actions crossbow submit test-r-arrow-backwards-compatibility

kou · 2025-02-14T05:17:36Z

@github-actions crossbow submit test-r-arrow-backwards-compatibility

github-actions · 2025-02-14T05:19:43Z

Revision: 3636c5c

Submitted crossbow builds: ursacomputing/crossbow @ actions-4f1a2ba4a1

Task	Status
test-r-arrow-backwards-compatibility

jonkeane · 2025-02-14T14:15:45Z

We got a few more versions with that! 1.0.1–3.0.0 look like they fail when trying to build some of our dependencies. That's not totally surprising, I guess. Arrow 3.0.0 was just over 4 years ago, so maybe we do drop those from the matrix. And especially if we had C++ tests for this kind of backwards compatibility like you mention @kou that would give us more confidence too.

One other alternative (though I think it's more work than it's worth right here / right now) would be to refactor this action so that the old version of arrow is run on an older OS that it is compatible with.

kou · 2025-02-16T00:41:18Z

We can use Ubuntu 20.04 for 1.0.0-3.0.0 but we need to remove them eventually because Ubuntu 20.04 will reach EOL.

Should we keep 1.0.0-3.0.0 while we don't get any error with Ubuntu 20.04? Or can we drop 1.0.0-3.0.0 in this PR?

jonkeane · 2025-02-17T14:11:37Z

Should we keep 1.0.0-3.0.0 while we don't get any error with Ubuntu 20.04? Or can we drop 1.0.0-3.0.0 in this PR?

Let's keep them for now (with 20.04, and we might / will eventually get an error) and I can look into what we want to do going forward

… jobs

kou · 2025-02-18T00:24:26Z

@github-actions crossbow submit test-r-arrow-backwards-compatibility

github-actions · 2025-02-18T00:26:42Z

Revision: 83c378f

Submitted crossbow builds: ursacomputing/crossbow @ actions-288e5d5ed0

Task	Status
test-r-arrow-backwards-compatibility

kou · 2025-02-18T00:51:57Z

Let's keep them for now (with 20.04, and we might / will eventually get an error) and I can look into what we want to do going forward

OK! Done!

jonkeane

Thank you for all the work on this ❤️

jonkeane · 2025-02-18T13:52:18Z

dev/tasks/r/github.linux.arrow.version.back.compat.yml

+    # condition && true-case || false-case
+    # ==
+    # condition ? true-case : false-case
+    #
+    # We use Ubuntu 20.04 for 3.0.0 or earlier because 3.0.0 or
+    # earlier can't be built on Ubuntu 22.04. We will drop 3.0.0 or
+    # earlier when Ubuntu 20.04 is unavailable on GitHub Actions.


Thanks for this comment + the helpful hint about how these work — I always have to go and look these up when I edit them.

conbench-apache-arrow · 2025-02-19T08:54:17Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 15f1d94.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

…possible for nightly jobs (apache#45507) ### Rationale for this change Ubuntu 20.04 will reach EOL on 2025-05. ### What changes are included in this PR? * Use Ubuntu 22.04 instead of Ubuntu 20.04 for Apache Arrow C++ 4.0.0 or later. * Keep using Ubuntu 20.04 for Apache Arrow C++ 3.0.0 or earlier because we can't build Apache Arrow C++ 3.0.0 or earlier on Ubuntu 22.04. We can use pre-built binaries forApache Arrow C++ 3.0.0 or earlier on Ubuntu 20.04. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: apache#45505 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

assignUser · 2025-03-04T19:59:08Z

@jonkeane fyi they moved total deprecation of 20.04 up a month to beginning of april

kou requested review from assignUser, jonkeane and raulcd as code owners February 12, 2025 06:49

github-actions bot added the awaiting committer review label Feb 12, 2025

kou commented Feb 12, 2025

View reviewed changes

github-actions bot added awaiting changes and removed awaiting committer review labels Feb 12, 2025

kou commented Feb 13, 2025

View reviewed changes

jonkeane reviewed Feb 14, 2025

View reviewed changes

github-actions bot added awaiting change review and removed awaiting changes labels Feb 14, 2025

This comment was marked as outdated.

Sign in to view

github-actions bot added awaiting changes and removed awaiting change review labels Feb 14, 2025

kou force-pushed the ci-r-ubuntu-20.04 branch from 8bdd4ef to 3636c5c Compare February 14, 2025 05:17

github-actions bot added awaiting change review and removed awaiting changes labels Feb 14, 2025

github-actions bot added awaiting changes and removed awaiting change review labels Feb 14, 2025

kou added 2 commits February 18, 2025 09:03

apacheGH-45505: [CI][R] Use Ubuntu 22.04 instead of 20.04 for nightly…

57052fa

… jobs

Don't use binary package for old Arrow

28dd526

Use Ubuntu 20.04 for 3.0.0 or earlier

eb57ce6

kou force-pushed the ci-r-ubuntu-20.04 branch from 3636c5c to eb57ce6 Compare February 18, 2025 00:03

github-actions bot added awaiting change review and removed awaiting changes labels Feb 18, 2025

kou added 2 commits February 18, 2025 09:08

Add missing escape

4f4f47d

Use double quote

83c378f

kou changed the title ~~GH-45505: [CI][R] Use Ubuntu 22.04 instead of 20.04 for nightly jobs~~ GH-45505: [CI][R] Use Ubuntu 22.04 instead of 20.04 as much as possible for nightly jobs Feb 18, 2025

jonkeane approved these changes Feb 18, 2025

View reviewed changes

github-actions bot added awaiting merge and removed awaiting change review labels Feb 18, 2025

kou merged commit 15f1d94 into apache:main Feb 19, 2025
10 checks passed

kou deleted the ci-r-ubuntu-20.04 branch February 19, 2025 01:56

kou removed the awaiting merge label Feb 19, 2025

kou mentioned this pull request Feb 19, 2025

[CI][R] Use Ubuntu 22.04 instead of 20.04 as much as possible for nightly jobs #45505

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-45505: [CI][R] Use Ubuntu 22.04 instead of 20.04 as much as possible for nightly jobs #45507

GH-45505: [CI][R] Use Ubuntu 22.04 instead of 20.04 as much as possible for nightly jobs #45507

kou commented Feb 12, 2025 •

edited

Loading

kou commented Feb 12, 2025

github-actions bot commented Feb 12, 2025

kou Feb 12, 2025

kou Feb 13, 2025

jonkeane Feb 14, 2025

kou Feb 14, 2025

github-actions bot commented Feb 12, 2025

kou Feb 13, 2025

jonkeane Feb 14, 2025

kou Feb 14, 2025

jonkeane Feb 14, 2025

kou Feb 14, 2025

kou Feb 14, 2025

jonkeane Feb 14, 2025

kou Feb 16, 2025

kou commented Feb 14, 2025

This comment was marked as outdated.

kou commented Feb 14, 2025

github-actions bot commented Feb 14, 2025

jonkeane commented Feb 14, 2025

kou commented Feb 16, 2025

jonkeane commented Feb 17, 2025

kou commented Feb 18, 2025

github-actions bot commented Feb 18, 2025

kou commented Feb 18, 2025

jonkeane left a comment

jonkeane Feb 18, 2025

conbench-apache-arrow bot commented Feb 19, 2025

assignUser commented Mar 4, 2025

		@@ -89,7 +89,7 @@ jobs:
		- { old_arrow_version: '1.0.1', r: '4.0' }

GH-45505: [CI][R] Use Ubuntu 22.04 instead of 20.04 as much as possible for nightly jobs #45507

GH-45505: [CI][R] Use Ubuntu 22.04 instead of 20.04 as much as possible for nightly jobs #45507

Conversation

kou commented Feb 12, 2025 • edited Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

kou commented Feb 12, 2025

github-actions bot commented Feb 12, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 12, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kou commented Feb 14, 2025

This comment was marked as outdated.

kou commented Feb 14, 2025

github-actions bot commented Feb 14, 2025

jonkeane commented Feb 14, 2025

kou commented Feb 16, 2025

jonkeane commented Feb 17, 2025

kou commented Feb 18, 2025

github-actions bot commented Feb 18, 2025

kou commented Feb 18, 2025

jonkeane left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

conbench-apache-arrow bot commented Feb 19, 2025

assignUser commented Mar 4, 2025

kou commented Feb 12, 2025 •

edited

Loading