GH-27839: [R] Fetch latest nightly binary for arrow R dev versions. #38236

assignUser · 2023-10-12T04:39:13Z

Rationale for this change

We currently need to manually download the latest nightly build as the ".9000' dev version never matches any nightly builds.

What changes are included in this PR?

Check the nightly repo html listing for the latest version matching the local major version.
Refactor nixlibs.R and integrate the functionality of winlibs.R into it to simplify maintenance and prepare for the future of in-line windows libarrow builds.

These changes are build on #38195 so once that is merged I will rebase this.

Are these changes tested?

Crossbow

Closes: [R] [Packaging] Developing + {LIBARROW_BINARY=true} #27839

assignUser · 2023-10-12T04:39:42Z

@github-actions crossbow submit -g r

github-actions · 2023-10-12T04:39:42Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

assignUser · 2023-10-12T04:53:42Z

@github-actions crossbow submit r-binary-packages

github-actions · 2023-10-12T04:56:16Z

Revision: 183f365

Submitted crossbow builds: ursacomputing/crossbow @ actions-007951fa57

Task	Status
r-binary-packages

github-actions · 2023-10-12T06:06:10Z

Revision: dd2a5f3

Submitted crossbow builds: ursacomputing/crossbow @ actions-fefb754618

Task	Status
r-binary-packages

r/tools/nixlibs.R

### Rationale for this change Several PRs over the last few months have update the build system to be more friendly for developers. During this process it has also come to light that we haven't supported the Windows development setup documented here since R 4.1 (released in spring 2021). I had to remove Windows from the test-r-devdocs job because the approach used there was not compatible with the `setup-r@ v2` action, and the job was failing with the `@ v1` action. ### What changes are included in this PR? - Updated the sections on using pre-built static libraries and bundled builds - Removed the Windows section regarding the bundled build. This section would need rewriting to support the last two minor releases of R but in the meantime I think it is mostly confusing. ### Are these changes tested? They are documentation changes. They are also slightly optimisitc: we can fix problems with the developer setup incrementally between releases, but it's more difficult to update our documentation. This PR documents the intended behaviour after #38236 . ### Are there any user-facing changes? No. * Closes: #37945 Lead-authored-by: Dewey Dunnington <dewey@voltrondata.com> Co-authored-by: Dewey Dunnington <dewey@fishandwhistle.net> Co-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>

main use case automatic download of nightly versions

assignUser · 2023-10-23T15:56:43Z

@paleolimbot ah sorry I got sidetracked and was fiddling around on windows^^ I'll update this today, would gladly accept some help with adding tests for the new functionality!

assignUser · 2023-10-24T02:22:49Z

With the release over I think it makes sense to merge this an refine in follow up PRs (e.g. #38430, #38429) so we can all benefit from the new functionality (and find out the kinks that need ironing out^^).

paleolimbot

Pending the rename of del perhaps followed by a crossbow submit -g r...I'll take on #38430 after this is merged. Thanks!

r/tools/nixlibs.R

paleolimbot · 2023-10-24T18:47:46Z

r/tools/nixlibs.R

+      matching_major <- versions[versions$X1 == description_version[1, 1], ]
+      latest <- matching_major[which.max(matching_major$X4), ]
+      package_version(paste0(latest, collapse = "."))


Lets punt this to #38430

assignUser · 2023-10-25T00:54:02Z

@github-actions crossbow submit r-binary-packages

github-actions · 2023-10-25T00:56:34Z

Revision: d219ba4

Submitted crossbow builds: ursacomputing/crossbow @ actions-c07979cf0d

Task	Status
r-binary-packages

…ons. (apache#38236) ### Rationale for this change We currently need to manually download the latest nightly build as the ".9000' dev version never matches any nightly builds. ### What changes are included in this PR? Check the nightly repo html listing for the latest version matching the local major version. Refactor nixlibs.R and integrate the functionality of winlibs.R into it to simplify maintenance and prepare for the future of in-line windows libarrow builds. These changes are build on apache#38195 so once that is merged I will rebase this. ### Are these changes tested? Crossbow * Closes: apache#27839 Lead-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Co-authored-by: Dewey Dunnington <dewey@dunnington.ca> Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>

conbench-apache-arrow · 2023-10-26T12:56:15Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 2a4e0db.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.

…38534) ### Rationale for this change A few rough edges exist after #38236 including: - When zero or 1 nightly with the matching major version exist, detection of the latest nightly might fail - At least one CI job is pulling nightlies instead of using the version from the current commit ### What changes are included in this PR? - Clean up `find_latest_nightly()` + add test - Ensure all CI jobs are not using `find_latest_nightly()` ### Are these changes tested? Yes (test added) ### Are there any user-facing changes? No * Closes: #38430 Lead-authored-by: Dewey Dunnington <dewey@fishandwhistle.net> Co-authored-by: Jonathan Keane <jkeane@gmail.com> Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>

…ons. (apache#38236) ### Rationale for this change We currently need to manually download the latest nightly build as the ".9000' dev version never matches any nightly builds. ### What changes are included in this PR? Check the nightly repo html listing for the latest version matching the local major version. Refactor nixlibs.R and integrate the functionality of winlibs.R into it to simplify maintenance and prepare for the future of in-line windows libarrow builds. These changes are build on apache#38195 so once that is merged I will rebase this. ### Are these changes tested? Crossbow * Closes: apache#27839 Lead-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Co-authored-by: Dewey Dunnington <dewey@dunnington.ca> Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>

### Rationale for this change Several PRs over the last few months have update the build system to be more friendly for developers. During this process it has also come to light that we haven't supported the Windows development setup documented here since R 4.1 (released in spring 2021). I had to remove Windows from the test-r-devdocs job because the approach used there was not compatible with the `setup-r@ v2` action, and the job was failing with the `@ v1` action. ### What changes are included in this PR? - Updated the sections on using pre-built static libraries and bundled builds - Removed the Windows section regarding the bundled build. This section would need rewriting to support the last two minor releases of R but in the meantime I think it is mostly confusing. ### Are these changes tested? They are documentation changes. They are also slightly optimisitc: we can fix problems with the developer setup incrementally between releases, but it's more difficult to update our documentation. This PR documents the intended behaviour after apache#38236 . ### Are there any user-facing changes? No. * Closes: apache#37945 Lead-authored-by: Dewey Dunnington <dewey@voltrondata.com> Co-authored-by: Dewey Dunnington <dewey@fishandwhistle.net> Co-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>

…he#38303) ### Rationale for this change `expr` was printing the number of matching chars which showed up as noise in the log (which we want to avoid as much as possible to avoid any false positive checks) See apache#38236 (comment) for @ jonkeane's investigation. ### What changes are included in this PR? Replace use of expr with test. ### Are these changes tested? Crossbow Lead-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Co-authored-by: Jonathan Keane <jkeane@gmail.com> Signed-off-by: Jonathan Keane <jkeane@gmail.com>

…ons. (apache#38236) ### Rationale for this change We currently need to manually download the latest nightly build as the ".9000' dev version never matches any nightly builds. ### What changes are included in this PR? Check the nightly repo html listing for the latest version matching the local major version. Refactor nixlibs.R and integrate the functionality of winlibs.R into it to simplify maintenance and prepare for the future of in-line windows libarrow builds. These changes are build on apache#38195 so once that is merged I will rebase this. ### Are these changes tested? Crossbow * Closes: apache#27839 Lead-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Co-authored-by: Dewey Dunnington <dewey@dunnington.ca> Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>

…tor (apache#38534) ### Rationale for this change A few rough edges exist after apache#38236 including: - When zero or 1 nightly with the matching major version exist, detection of the latest nightly might fail - At least one CI job is pulling nightlies instead of using the version from the current commit ### What changes are included in this PR? - Clean up `find_latest_nightly()` + add test - Ensure all CI jobs are not using `find_latest_nightly()` ### Are these changes tested? Yes (test added) ### Are there any user-facing changes? No * Closes: apache#38430 Lead-authored-by: Dewey Dunnington <dewey@fishandwhistle.net> Co-authored-by: Jonathan Keane <jkeane@gmail.com> Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>

…38236) ### Rationale for this change We currently need to manually download the latest nightly build as the ".9000' dev version never matches any nightly builds. ### What changes are included in this PR? Check the nightly repo html listing for the latest version matching the local major version. Refactor nixlibs.R and integrate the functionality of winlibs.R into it to simplify maintenance and prepare for the future of in-line windows libarrow builds. These changes are build on #38195 so once that is merged I will rebase this. ### Are these changes tested? Crossbow * Closes: #27839 Lead-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Co-authored-by: Dewey Dunnington <dewey@dunnington.ca> Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>

…38534) ### Rationale for this change A few rough edges exist after #38236 including: - When zero or 1 nightly with the matching major version exist, detection of the latest nightly might fail - At least one CI job is pulling nightlies instead of using the version from the current commit ### What changes are included in this PR? - Clean up `find_latest_nightly()` + add test - Ensure all CI jobs are not using `find_latest_nightly()` ### Are these changes tested? Yes (test added) ### Are there any user-facing changes? No * Closes: #38430 Lead-authored-by: Dewey Dunnington <dewey@fishandwhistle.net> Co-authored-by: Jonathan Keane <jkeane@gmail.com> Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>

…tor (apache#38534) ### Rationale for this change A few rough edges exist after apache#38236 including: - When zero or 1 nightly with the matching major version exist, detection of the latest nightly might fail - At least one CI job is pulling nightlies instead of using the version from the current commit ### What changes are included in this PR? - Clean up `find_latest_nightly()` + add test - Ensure all CI jobs are not using `find_latest_nightly()` ### Are these changes tested? Yes (test added) ### Are there any user-facing changes? No * Closes: apache#38430 Lead-authored-by: Dewey Dunnington <dewey@fishandwhistle.net> Co-authored-by: Jonathan Keane <jkeane@gmail.com> Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>

### Rationale for this change Several PRs over the last few months have update the build system to be more friendly for developers. During this process it has also come to light that we haven't supported the Windows development setup documented here since R 4.1 (released in spring 2021). I had to remove Windows from the test-r-devdocs job because the approach used there was not compatible with the `setup-r@ v2` action, and the job was failing with the `@ v1` action. ### What changes are included in this PR? - Updated the sections on using pre-built static libraries and bundled builds - Removed the Windows section regarding the bundled build. This section would need rewriting to support the last two minor releases of R but in the meantime I think it is mostly confusing. ### Are these changes tested? They are documentation changes. They are also slightly optimisitc: we can fix problems with the developer setup incrementally between releases, but it's more difficult to update our documentation. This PR documents the intended behaviour after apache#38236 . ### Are there any user-facing changes? No. * Closes: apache#37945 Lead-authored-by: Dewey Dunnington <dewey@voltrondata.com> Co-authored-by: Dewey Dunnington <dewey@fishandwhistle.net> Co-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>

…he#38303) ### Rationale for this change `expr` was printing the number of matching chars which showed up as noise in the log (which we want to avoid as much as possible to avoid any false positive checks) See apache#38236 (comment) for @ jonkeane's investigation. ### What changes are included in this PR? Replace use of expr with test. ### Are these changes tested? Crossbow Lead-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Co-authored-by: Jonathan Keane <jkeane@gmail.com> Signed-off-by: Jonathan Keane <jkeane@gmail.com>

…ons. (apache#38236) ### Rationale for this change We currently need to manually download the latest nightly build as the ".9000' dev version never matches any nightly builds. ### What changes are included in this PR? Check the nightly repo html listing for the latest version matching the local major version. Refactor nixlibs.R and integrate the functionality of winlibs.R into it to simplify maintenance and prepare for the future of in-line windows libarrow builds. These changes are build on apache#38195 so once that is merged I will rebase this. ### Are these changes tested? Crossbow * Closes: apache#27839 Lead-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de> Co-authored-by: Dewey Dunnington <dewey@dunnington.ca> Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>

…tor (apache#38534) ### Rationale for this change A few rough edges exist after apache#38236 including: - When zero or 1 nightly with the matching major version exist, detection of the latest nightly might fail - At least one CI job is pulling nightlies instead of using the version from the current commit ### What changes are included in this PR? - Clean up `find_latest_nightly()` + add test - Ensure all CI jobs are not using `find_latest_nightly()` ### Are these changes tested? Yes (test added) ### Are there any user-facing changes? No * Closes: apache#38430 Lead-authored-by: Dewey Dunnington <dewey@fishandwhistle.net> Co-authored-by: Jonathan Keane <jkeane@gmail.com> Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>

github-actions bot added Component: R Component: C++ awaiting review Awaiting review labels Oct 12, 2023

assignUser mentioned this pull request Oct 12, 2023

GH-37945: [R] Update developer documentation #38220

Merged

This comment was marked as outdated.

Sign in to view

jonkeane reviewed Oct 12, 2023

View reviewed changes

r/tools/nixlibs.R Outdated Show resolved Hide resolved

assignUser mentioned this pull request Oct 12, 2023

GH-38043: [R] Enable all features by default on macOS #38195

Merged

jonkeane reviewed Oct 12, 2023

View reviewed changes

r/tools/nixlibs.R Show resolved Hide resolved

jonkeane reviewed Oct 12, 2023

View reviewed changes

r/tools/nixlibs.R Outdated Show resolved Hide resolved

jonkeane reviewed Oct 12, 2023

View reviewed changes

r/tools/nixlibs.R Outdated Show resolved Hide resolved

assignUser mentioned this pull request Oct 12, 2023

GH-37941: [R][CI][Release] Add checksum verification for pre-compiled binaries #38115

Merged

assignUser added 11 commits October 13, 2023 07:22

use logging function

ee16a16

add latests nightly check to nixlibs

9a7b8ab

seperate vars and functions

81fa870

add early exit on nixlib error

13e2698

integrate windows into nixlibs.R

45e90c1

fix version for testing

7a90498

skip search if dev repo is not nightlies.a.o

8d9ed40

fix arrow repo

f8bbfa4

set not_cran when dev version

a759f70

automatically handle version mismatch between package and libarrow

2d5d66e

main use case automatic download of nightly versions

fix libarrow dir on windows

32a06e4

assignUser force-pushed the refactor-and-latest branch from 0b174d7 to 32a06e4 Compare October 13, 2023 05:28

github-actions bot removed the Component: C++ label Oct 13, 2023

assignUser mentioned this pull request Oct 24, 2023

[R] Add test for new nixlibs.R #38430

Closed

assignUser marked this pull request as ready for review October 24, 2023 02:21

assignUser requested a review from thisisnic as a code owner October 24, 2023 02:21

assignUser requested a review from paleolimbot October 24, 2023 02:23

paleolimbot approved these changes Oct 24, 2023

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels Oct 24, 2023

rename del to cleanup

d219ba4

assignUser merged commit 2a4e0db into apache:main Oct 25, 2023
10 checks passed

assignUser removed the awaiting merge Awaiting merge label Oct 25, 2023

paleolimbot mentioned this pull request Oct 31, 2023

GH-38430: [R] Add test + fix corner cases after nixlibs.R refactor #38534

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-27839: [R] Fetch latest nightly binary for arrow R dev versions. #38236

GH-27839: [R] Fetch latest nightly binary for arrow R dev versions. #38236

assignUser commented Oct 12, 2023 •

edited by github-actions bot

assignUser commented Oct 12, 2023

github-actions bot commented Oct 12, 2023

This comment was marked as outdated.

assignUser commented Oct 12, 2023 •

edited

github-actions bot commented Oct 12, 2023

github-actions bot commented Oct 12, 2023

assignUser commented Oct 23, 2023

assignUser commented Oct 24, 2023 •

edited

paleolimbot left a comment

paleolimbot Oct 24, 2023

assignUser commented Oct 25, 2023

github-actions bot commented Oct 25, 2023

conbench-apache-arrow bot commented Oct 26, 2023

GH-27839: [R] Fetch latest nightly binary for arrow R dev versions. #38236

GH-27839: [R] Fetch latest nightly binary for arrow R dev versions. #38236

Conversation

assignUser commented Oct 12, 2023 • edited by github-actions bot

Rationale for this change

What changes are included in this PR?

Are these changes tested?

assignUser commented Oct 12, 2023

github-actions bot commented Oct 12, 2023

This comment was marked as outdated.

assignUser commented Oct 12, 2023 • edited

github-actions bot commented Oct 12, 2023

github-actions bot commented Oct 12, 2023

assignUser commented Oct 23, 2023

assignUser commented Oct 24, 2023 • edited

paleolimbot left a comment

Choose a reason for hiding this comment

paleolimbot Oct 24, 2023

Choose a reason for hiding this comment

assignUser commented Oct 25, 2023

github-actions bot commented Oct 25, 2023

conbench-apache-arrow bot commented Oct 26, 2023

assignUser commented Oct 12, 2023 •

edited by github-actions bot

assignUser commented Oct 12, 2023 •

edited

assignUser commented Oct 24, 2023 •

edited