GH-36456: [R] Link to correct version of OpenSSL when using autobrew #36551

paleolimbot · 2023-07-07T14:43:16Z

Rationale for this change

The r-binary-packages job (which uses autobrew) and the autobrew nightly jobs are failing because they are linking to a different version of OpenSSL than the package was built against. I believe this occurred because Arrow and its dependencies are built against the autobrew headers which included openssl. The ssl and crypto libraries weren't explicitly linked, so I think whatever LibreSSL fork MacOS installs by default was getting linked. This was perhaps compatible using the version of autobrew for High Sierra/the version of LibreSSL on High Sierra but was not compatible with the version of autobrew for Big Sur/the version of LibreSSL on Big Sur.

What changes are included in this PR?

This PR explicitly adds OpenSSL 1.1 to the autobrew formulas and explicitly adds -lssl -lcrypto to the PKG_LIBS (1.1 because that's what was in the corresponding homebrew formula).

Are these changes tested?

Existing nightly tests cover these changes.

Are there any user-facing changes?

No.

Closes: [R] R package is linking to the wrong OpenSSL on MacOS BigSur? #36456

paleolimbot · 2023-07-07T14:44:06Z

@github-actions crossbow submit r-binary-packages homebrew-r-autobrew

github-actions · 2023-07-07T14:46:41Z

Revision: b896259

Submitted crossbow builds: ursacomputing/crossbow @ actions-3d21794472

Task	Status
homebrew-r-autobrew
r-binary-packages

paleolimbot · 2023-07-07T16:06:05Z

@github-actions crossbow submit r-binary-packages homebrew-r-autobrew

github-actions · 2023-07-07T16:08:40Z

Revision: 15700d4

Submitted crossbow builds: ursacomputing/crossbow @ actions-007d7f7198

Task	Status
homebrew-r-autobrew
r-binary-packages

paleolimbot · 2023-07-10T14:44:35Z

@github-actions crossbow submit r-binary-packages homebrew-r-autobrew

github-actions · 2023-07-10T14:47:09Z

Revision: 6a538f4

Submitted crossbow builds: ursacomputing/crossbow @ actions-0bc60af847

Task	Status
homebrew-r-autobrew
r-binary-packages

paleolimbot · 2023-07-11T15:24:18Z

@github-actions crossbow submit r-binary-packages homebrew-r-autobrew

github-actions · 2023-07-11T15:26:58Z

Revision: 047bc6a

Submitted crossbow builds: ursacomputing/crossbow @ actions-bd672e75e9

Task	Status
homebrew-r-autobrew
r-binary-packages

paleolimbot · 2023-07-11T16:54:10Z

@github-actions crossbow submit r-binary-packages homebrew-r-autobrew

github-actions · 2023-07-11T16:56:48Z

Revision: 047bc6a

Submitted crossbow builds: ursacomputing/crossbow @ actions-47a81f0980

Task	Status
homebrew-r-autobrew
r-binary-packages

paleolimbot · 2023-07-11T19:32:13Z

@nealrichardson If my reading of all this is correct, I think this will change the OpenSSL that is linked by default in the CRAN package that everybody uses. Is there a reason that this wasn't done before that I'm not aware of?

nealrichardson · 2023-07-11T20:06:11Z

No reason I recall that it wasn't done before. How does upstream autobrew handle this?

If we're pinning a version, why not pin 3 instead of 1.1? (Edit: oh I see you mentioned that in the description)

assignUser

Nice! I don't think it should be an issue for CRAN as we modify/don't use system libs explicitly in autobrew#L68-L75

paleolimbot · 2023-07-12T01:50:16Z

@nealrichardson Thanks!

How does upstream autobrew handle this?

I am not sure what this means! I assumed the formula in the autobrew repos is modified to reflect the one I modified here before each release? My understanding of autobrew is admittedly limited.

kou

+1

Great!

(But I don't know why openssl@1.1 is chosen instead of openssl@3 by Apache Arrow C++ when both of them are installed...)

r/tools/autobrew

nealrichardson · 2023-07-12T04:00:37Z

How does upstream autobrew handle this?

I am not sure what this means! I assumed the formula in the autobrew repos is modified to reflect the one I modified here before each release? My understanding of autobrew is admittedly limited.

We do, but @jeroen maintains the actual autobrew formulae and scripts (here, here, and here, specifically), of which ours are copies that we modify to test on every commit. When changes to Homebrew happen or CRAN mandates new things, he handles it there, and we should watch (at release time, or when things break) for changes we need to pull in accordingly to stay in sync.

I don't see openssl pinned there, so I'm not sure what's different such that we need it--something with the self-hosted runner setup, perhaps? If so, that's probably ok, though might be worth adding comments that that's the case.

Re: openssl version, Homebrew actually has openssl@3: https://github.com/Homebrew/homebrew-core/blob/master/Formula/apache-arrow.rb#L30 If our copy is pinned on 1.1, we should update it, and pull in any other changes from upstream, same as autobrew.

jeroen · 2023-07-12T09:07:33Z

You already explicitly depend on openssl via thrift (and aws-sdk maybe)?

## brew deps --tree apache-arrow-static
autobrew/cran/apache-arrow-static
├── aws-sdk-cpp-static
├── brotli
├── lz4
├── snappy
├── thrift
│   └── openssl@3
│       └── ca-certificates
└── zstd
    ├── lz4
    └── xz

Upstream homebrew bumped thrift and aws-sdk to openssl@3 a few weeks ago, so the next release of the apache-arrow-static bundle will automatically include openssl@3.

The current PR will probably create a conflict because you end up depending on both openssl@11 and openssl@3. This may work in a dynamic linked stack, but once you try to statically link the entire stack in the R package this will probably create symbol conflicts.

paleolimbot · 2023-07-12T12:32:30Z

@github-actions crossbow submit r-binary-packages homebrew-r-autobrew

github-actions · 2023-07-12T12:34:56Z

Revision: 018ca98

Submitted crossbow builds: ursacomputing/crossbow @ actions-225ba98cc6

Task	Status
homebrew-r-autobrew
r-binary-packages

paleolimbot · 2023-07-12T13:20:53Z

@github-actions crossbow submit r-binary-packages homebrew-r-autobrew

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

paleolimbot · 2023-07-16T16:06:14Z

@github-actions crossbow submit r-binary-packages homebrew-r-autobrew

github-actions · 2023-07-16T16:08:47Z

Revision: ae451ee

Submitted crossbow builds: ursacomputing/crossbow @ actions-0fd077276c

Task	Status
homebrew-r-autobrew
r-binary-packages

paleolimbot · 2023-07-16T17:32:01Z

@github-actions crossbow submit r-binary-packages homebrew-r-autobrew

github-actions · 2023-07-16T17:34:30Z

Revision: 0043b1c

Submitted crossbow builds: ursacomputing/crossbow @ actions-1602f52d96

Task	Status
homebrew-r-autobrew
r-binary-packages

paleolimbot · 2023-07-16T21:35:23Z

Ok: I opened #36707 for the CI failure and for the r-binary-packages failure; also, I removed any unnecessary changes to the formulae.

kou

+1

kou · 2023-07-16T21:58:39Z

r/configure

Could you revert changes in this file?

…36551) ### Rationale for this change The r-binary-packages job (which uses autobrew) and the autobrew nightly jobs are failing because they are linking to a different version of OpenSSL than the package was built against. I believe this occurred because Arrow and its dependencies are built against the autobrew headers which included openssl. The `ssl` and `crypto` libraries weren't explicitly linked, so I think whatever LibreSSL fork MacOS installs by default was getting linked. This was perhaps compatible using the version of autobrew for High Sierra/the version of LibreSSL on High Sierra but was not compatible with the version of autobrew for Big Sur/the version of LibreSSL on Big Sur. ### What changes are included in this PR? This PR explicitly adds OpenSSL 1.1 to the autobrew formulas and explicitly adds `-lssl -lcrypto` to the PKG_LIBS (1.1 because that's what was in the corresponding homebrew formula). ### Are these changes tested? Existing nightly tests cover these changes. ### Are there any user-facing changes? No. * Closes: #36456 Lead-authored-by: Dewey Dunnington <dewey@voltrondata.com> Co-authored-by: Dewey Dunnington <dewey@fishandwhistle.net> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>

…obrew (apache#36551) ### Rationale for this change The r-binary-packages job (which uses autobrew) and the autobrew nightly jobs are failing because they are linking to a different version of OpenSSL than the package was built against. I believe this occurred because Arrow and its dependencies are built against the autobrew headers which included openssl. The `ssl` and `crypto` libraries weren't explicitly linked, so I think whatever LibreSSL fork MacOS installs by default was getting linked. This was perhaps compatible using the version of autobrew for High Sierra/the version of LibreSSL on High Sierra but was not compatible with the version of autobrew for Big Sur/the version of LibreSSL on Big Sur. ### What changes are included in this PR? This PR explicitly adds OpenSSL 1.1 to the autobrew formulas and explicitly adds `-lssl -lcrypto` to the PKG_LIBS (1.1 because that's what was in the corresponding homebrew formula). ### Are these changes tested? Existing nightly tests cover these changes. ### Are there any user-facing changes? No. * Closes: apache#36456 Lead-authored-by: Dewey Dunnington <dewey@voltrondata.com> Co-authored-by: Dewey Dunnington <dewey@fishandwhistle.net> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>

conbench-apache-arrow · 2023-07-25T10:56:15Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit ed87a5b.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.

…obrew (apache#36551) ### Rationale for this change The r-binary-packages job (which uses autobrew) and the autobrew nightly jobs are failing because they are linking to a different version of OpenSSL than the package was built against. I believe this occurred because Arrow and its dependencies are built against the autobrew headers which included openssl. The `ssl` and `crypto` libraries weren't explicitly linked, so I think whatever LibreSSL fork MacOS installs by default was getting linked. This was perhaps compatible using the version of autobrew for High Sierra/the version of LibreSSL on High Sierra but was not compatible with the version of autobrew for Big Sur/the version of LibreSSL on Big Sur. ### What changes are included in this PR? This PR explicitly adds OpenSSL 1.1 to the autobrew formulas and explicitly adds `-lssl -lcrypto` to the PKG_LIBS (1.1 because that's what was in the corresponding homebrew formula). ### Are these changes tested? Existing nightly tests cover these changes. ### Are there any user-facing changes? No. * Closes: apache#36456 Lead-authored-by: Dewey Dunnington <dewey@voltrondata.com> Co-authored-by: Dewey Dunnington <dewey@fishandwhistle.net> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>

github-actions bot added Component: R awaiting committer review Awaiting committer review labels Jul 7, 2023

paleolimbot marked this pull request as ready for review July 11, 2023 19:29

paleolimbot requested review from assignUser, kou and raulcd as code owners July 11, 2023 19:29

assignUser mentioned this pull request Jul 12, 2023

GH-36456: [CI][R] Unlink system OpenSSL to avoid mixing OpenSSL versions #36522

Closed

assignUser approved these changes Jul 12, 2023

View reviewed changes

kou approved these changes Jul 12, 2023

View reviewed changes

r/tools/autobrew Outdated Show resolved Hide resolved

github-actions bot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Jul 12, 2023

paleolimbot force-pushed the r-openssl branch from 81aa77e to fba6479 Compare July 12, 2023 12:38

paleolimbot and others added 12 commits July 16, 2023 13:05

try setting BREW env var

2fc75c6

undo BREW env var

8c58e0c

maybe use openssl from autobrew

053b57b

Update r/tools/autobrew

4f5b662

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

don't modify formula

2778993

maybe better configure fix

3ffbc73

maybe link -lcurl to the right SSL

5e08d6b

attempt pin to openssl3

c0fe845

maybe back to working state

e93b4ab

Use openssl3 in apache-arrow-static

3913e0b

try curl in apache-arrow-static

a0232bf

revert curl

ae451ee

paleolimbot force-pushed the r-openssl branch from 9634560 to ae451ee Compare July 16, 2023 16:05

remove explicit dependency

0043b1c

kou approved these changes Jul 16, 2023

View reviewed changes

r/configure Outdated

Copy link

Member

kou Jul 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you revert changes in this file?

paleolimbot reacted with thumbs up emoji

github-actions bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Jul 16, 2023

fix trailing whitespace

73850ff

paleolimbot merged commit ed87a5b into apache:main Jul 16, 2023
9 checks passed

paleolimbot removed the awaiting merge Awaiting merge label Jul 16, 2023

thisisnic mentioned this pull request Jul 21, 2023

[R] CRAN packaging checklist for version 13.0.0 #36802

Closed

44 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-36456: [R] Link to correct version of OpenSSL when using autobrew #36551

GH-36456: [R] Link to correct version of OpenSSL when using autobrew #36551

paleolimbot commented Jul 7, 2023 •

edited

paleolimbot commented Jul 7, 2023

github-actions bot commented Jul 7, 2023

paleolimbot commented Jul 7, 2023

github-actions bot commented Jul 7, 2023

paleolimbot commented Jul 10, 2023

github-actions bot commented Jul 10, 2023

paleolimbot commented Jul 11, 2023

github-actions bot commented Jul 11, 2023

paleolimbot commented Jul 11, 2023

github-actions bot commented Jul 11, 2023

paleolimbot commented Jul 11, 2023

nealrichardson commented Jul 11, 2023 •

edited

assignUser left a comment

paleolimbot commented Jul 12, 2023

kou left a comment •

edited

nealrichardson commented Jul 12, 2023

jeroen commented Jul 12, 2023 •

edited

paleolimbot commented Jul 12, 2023

github-actions bot commented Jul 12, 2023

paleolimbot commented Jul 12, 2023

paleolimbot commented Jul 16, 2023

github-actions bot commented Jul 16, 2023

paleolimbot commented Jul 16, 2023

github-actions bot commented Jul 16, 2023

paleolimbot commented Jul 16, 2023

kou left a comment

kou Jul 16, 2023

conbench-apache-arrow bot commented Jul 25, 2023

GH-36456: [R] Link to correct version of OpenSSL when using autobrew #36551

GH-36456: [R] Link to correct version of OpenSSL when using autobrew #36551

Conversation

paleolimbot commented Jul 7, 2023 • edited

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

paleolimbot commented Jul 7, 2023

github-actions bot commented Jul 7, 2023

paleolimbot commented Jul 7, 2023

github-actions bot commented Jul 7, 2023

paleolimbot commented Jul 10, 2023

github-actions bot commented Jul 10, 2023

paleolimbot commented Jul 11, 2023

github-actions bot commented Jul 11, 2023

paleolimbot commented Jul 11, 2023

github-actions bot commented Jul 11, 2023

paleolimbot commented Jul 11, 2023

nealrichardson commented Jul 11, 2023 • edited

assignUser left a comment

Choose a reason for hiding this comment

paleolimbot commented Jul 12, 2023

kou left a comment • edited

Choose a reason for hiding this comment

nealrichardson commented Jul 12, 2023

jeroen commented Jul 12, 2023 • edited

paleolimbot commented Jul 12, 2023

github-actions bot commented Jul 12, 2023

paleolimbot commented Jul 12, 2023

paleolimbot commented Jul 16, 2023

github-actions bot commented Jul 16, 2023

paleolimbot commented Jul 16, 2023

github-actions bot commented Jul 16, 2023

paleolimbot commented Jul 16, 2023

kou left a comment

Choose a reason for hiding this comment

kou Jul 16, 2023

Choose a reason for hiding this comment

conbench-apache-arrow bot commented Jul 25, 2023

paleolimbot commented Jul 7, 2023 •

edited

nealrichardson commented Jul 11, 2023 •

edited

kou left a comment •

edited

jeroen commented Jul 12, 2023 •

edited