Update to OpenSSL 3 by default

[cjones051073]

• update openssl shim port and PG to use openssl 3 by default.
• rev bump all ports depending on openssl (using attached script).

Closes: 63801 - python ports: update to use openssl 3, allowing dependent binaries to be redistributed

macports-bump-rev.sh.txt

[macportsbot]

Notifying maintainers:
@kurthindenburg for port py-scrapy, py-service_identity, lastpass-cli, linkchecker.
@drkp for port gwenhywfar4, gwenhywfar5, mit-scheme, burp, ipmitool.
@gctwnl for port nsd.
@aphor for port haproxy.
@petrrr for port py-libcloud, py-mitmproxy.
@Lord-Kamina for port restinio.
@ci42 for port fossil, git, nodejs10, nodejs12, nodejs13, nodejs14, nodejs15, nodejs16, nodejs8, erlang, ejabberd.
@nerdling for port virtuoso-6, virtuoso-7, unbound, py-psycopg2, xml-security-c, xmltooling.
@judaew for port libwebsockets, dnsperf, haproxy, oha, lib3mf.
@RJVB for port qca, VLC2.
@kimuraw for port ruby, ruby186, ruby19, ruby20, ruby21, ruby22, ruby23, ruby24, ruby25, ruby26, ruby27, ruby30.
@ale275 for port shairport-sync.
@aque for port iperf3, py-m2crypto.
@RoddieKieley for port qpid-proton.
@herbygillot for port duckdb, cargo, gitui, rizin, wrangler, amp, rust, drill, websocat, eureka, flameshot, fluent-bit, lnav, starship, angle-grinder, hurl, zola.
@l2dy for port getdns, libtins, rizin, irssi, neomutt, et, pdns-recursor, libu2f-server, fluent-bit, h2o.
@rubendibattista for port uboot-tools.
@jmroot for port python26, python27, python310, python311-devel, python32, python33, python34, python35, python36, python37, python39, squid2, squid3, squid4.
@jdberry for port dovecot.
@lbschenkel for port neomutt, openpace, opensc, yubico-piv-tool.
@mojca for port py-autobahn, iAIDA, root6.
@trodemaster for port libfido2.
@manojkarthick for port gitweb, reddsaver.
@larryv for port openssl, lynx.
@dirkx for port mod_ca, mod_crl, mod_csr, mod_ocsp, mod_pkcs12, mod_scep, mod_spkac, mod_timestamp.
@causal-agent for port libretls.
@slewsys for port libwebsockets, mosquitto.
@eborisch for port dcmtk, zabbix4, OpenIPMI.
@mattbishop for port tomcat-native.
@rpfisher for port py-lscsoft-glue.
@bgilbert for port tarsnap.
@MarcusCalhoun-Lopez for port phantomjs-qt, qt5, qt511, qt513, qt53, qt55, qt56, qt57, qt58, qt59, cargo-c, libmowgli-2, py-openssl, gdcm, gdcm2.
@KensingtonTech for port libjwt.
@iay for port softhsm.
@neverpanic for port capnproto, openssl, textmate2, nginx.
@xeron for port ngircd.
@ecronin for port blackbag, boxbackup.
@mkhon for port tnftp.
@stromnov for port py-backports-ssl, py-cryptography.
@lhaeger for port xca.
@Ionic for port x2goclient, libssh.
@exg for port isync.
@ghosthound for port wireshark2, wireshark3, wireshark30.
@emaros for port ldas-tools-al, ldas-tools-framecpp, lscsoft-deps.
@scantor for port xml-security-c, xmltooling.
@i0ntempest for port idevicerestore, ldid, libimobiledevice, radare2, axel, libtorrent-rasterbar, openvpn3, qBittorrent, transmission-x11, trojan, hydra.
@JacksonIsaac for port libimobiledevice.
@jyrkiwahlstedt for port postgresql10, postgresql11, postgresql12, postgresql13, postgresql14, postgresql93, postgresql94, postgresql95, postgresql96.
@Tatsh for port openrct2.
@ctreleaven for port mythtv.27, mythtv.28.
@alfredh for port baresip, libre, librem, libsrtp.
@michaelld for port qt4-mac, dvisvgm.
@ryandesign for port dar, minizip-ng, wimlib, faust-devel, faust, faustlive-devel, eet, libhsplasma, libmongo-client, PlasmaClient, bahamut, php, curl, docsis, kerberos5, net-snmp, rabbitmq-c, rsync, shellinabox, php-event, php-mongo, php-stomp, php-swoole, aircrack-ng, openvas-client, rsyncrypto, NetSurf, QupZilla, apache2, lighttpd, slowhttptest, winetricks.
@jolan78 for port cryfs.
@ryanakca for port barrier, opensmtpd.
@markemer for port Io.
@dgilman for port redis.
@g5pw for port pev, radare2, rust, esniper, lnav, socat.
@arjanvandervelde for port poco.
@davidnich for port qore-asn1-module, qore-json-module, qore-xml-module, qore.
@danielluke for port bind9, nmap, sysmon, clamav, ntp.
@lpsinger for port py-twilio, ds9, htcondor, lscsoft-deps.
@agraef for port faust-devel, faust, faustlive-devel.
@roederja for port msodbcsql, msodbcsql17, mico, sequoia-pgp.
@gpanders for port netdata.
@nareshov for port git-crypt.
@schiebel for port mico.
@tobypeterson for port ldmud, epic4, epic5.
@jandemter for port py-trio, py-trustme.
@fhgwright for port libaes_siv, ntpsec.
@jvirkki for port dupd.
@Schamschula for port xar, xeus-sqlite, xeus, lftp, tcpdump, zeek, py-acme, py-ndg_httpsclient, certbot, stunnel, apache2, goaccess, nghttp2, nginx.
@mascguy for port openldap-devel, openldap.
@sikmir for port lagrange.
@jrjsmrtn for port wpa_passphrase.
@tenomoto for port nco.
@Raimondi for port lastpass-cli.
@toy for port stressdrive.
@mklein-de for port cadaver.
@raimue for port gmpc, hexchat, sslscan.
@dbevans for port opusfile, telepathy-idle, libgda5, libgit2, podofo, deluge, libtorrent-rasterbar, p5-crypt-smime.
@mohd-akram for port crystal, websocketpp, cpprestsdk.
@jcvernaleo for port alpine.
@nano103 for port libp11.
@amake for port yubikey-manager.
@catap for port retdec.
@cardi for port weechat.
@alexeyt820 for port librdkafka.
@millerdev for port openfortivpn.
@bwalle for port crackpkcs12.
@Gminfly for port libssh2, medusa.
@NicosPavlov for port kdelibs4, smokekde.
@danchr for port pypy, p5-net-eboks.
@essandess for port mail-server, apple-pki-bundle, calendar-contacts-server, privoxy.
@akkornel for port munge.
@Jakker for port softhsm.
@jerryyhom for port libevent, alpine.
@Veence for port gdal, qgis, qgis3.

[jmroot]

Shouldn't only the ports using the default openssl version be rev bumped?

[cjones051073]

Shouldn't only the ports using the default openssl version be rev bumped?

my script rev-bumps ports that depend on the openssl shim port. nothing more nothing less...

[cjones051073]

Shouldn't only the ports using the default openssl version be rev bumped?

my script rev-bumps ports that depend on the openssl shim port. nothing more nothing less...

to be clear, I have no plans to do any more or less than my script does. if that rev bumps ports that do not need it (unlikely) or misses ports, so be it..

[mascguy]

In terms of the logistics of ultimately merging this, do we expect the buildbots to handle this without dying?

At the very least, the watchers are going to be tied up for a LONG time, processing the full dependency list.

So before merging, we might want to break this up into at a few separate commits. (Preferably no more than 100 rev-bumps each.)

@jmroot and @ryandesign Can you folks offer some guidance?

[cjones051073]

In terms of the logistics of ultimately merging this, do we expect the buildbots to handle this without dying?

no. they will almost certainly timeout...

At the very least, the watchers are going to be tied up for a LONG time, processing the full dependency list...

ultimately I believe it comes down to a judgement call. do we 'merge and deal with the aftermath as is' or .... personally I am not sure there is much of an or...

[cjones051073]

In terms of the logistics of ultimately merging this, do we expect the buildbots to handle this without dying?

no. they will always certainly timeout...

Just be 100% clear, I assume you are referring to the CI test builds here ? these will probably timeout. The macports buildbots will handle then update though just fine...

[mascguy]

Just be 100% clear, I assume you are referring to the CI test builds here ? these will probably timeout. The macports buildbots will handle then update though just fine...

Nope, I'm talking about the buildbots. If it's one giant commit, the watchers will be tied up for an eternity, whilst leaving the builders idle.

[mascguy]

My recommendation - barring more definitive guidance from Josh and Ryan - would be to limit the number of rev-bumps to no more than 50 per commit.

And while we're not dealing with thousands of rev-bumps, I'm concerned we might have an issue like this from a few months ago:

https://lists.macports.org/pipermail/macports-dev/2021-June/043605.html

[cjones051073]

My recommendation - barring more definitive guidance from Josh and Ryan - would be to limit the number of rev-bumps to no more than 50 per commit.

And while we're not dealing with thousands of rev-bumps, I'm concerned we might have an issue like this from a few months ago:

https://lists.macports.org/pipermail/macports-dev/2021-June/043605.html

I am not going to do that. I don’t thing this PR is that large that merging it as is is that much of an issue.

[jmroot]

my script rev-bumps ports that depend on the openssl shim port. nothing more nothing less...

That doesn't appear to be the case. For example python26 is bumped and it only depends on openssl10.

[cjones051073]

my script rev-bumps ports that depend on the openssl shim port. nothing more nothing less...

That doesn't appear to be the case. For example python26 is bumped and it only depends on openssl10.

My script is probably not perfect. Bar doing it by hand, which I am not going to do, its better than nothing.

[cjones051073]

my script rev-bumps ports that depend on the openssl shim port. nothing more nothing less...

That doesn't appear to be the case. For example python26 is bumped and it only depends on openssl10.

my script rev-bumps ports that depend on the openssl shim port. nothing more nothing less...

That doesn't appear to be the case. For example python26 is bumped and it only depends on openssl10.

My script is probably not perfect. Bar doing it by hand, which I am not going to do, its better than nothing.

my script rev-bumps ports that depend on the openssl shim port. nothing more nothing less...

That doesn't appear to be the case. For example python26 is bumped and it only depends on openssl10.

My script is probably not perfect. Bar doing it by hand, which I am not going to do, its better than nothing.

Oberon ~/Downloads > port echo depends:openssl | grep python
python26                        
python27                        
python27-bootstrap              
python32                        
python33                        
python34                        
python35                        
python36                        
python37                        
python38                        
python39                        
python310                       
python311-devel 

if python26 should not appear in the list above, thats a bug in base.

[jmroot]

It's a bug in your regex. Try depends:':openssl(\s|$)'.

[mascguy]

Hey Chris, bear in mind that we're not trying to be critical of what you've done here. And we appreciate you taking the initiative, which is often the most important step!

Others are happy to assist, to help refine all of this. Whether that means breaking up the commits into smaller chunks, trimming the list of rev-bumps, or whatever. As long as you don't block commits to this PR by your fellow members, we'll help too!

p.s. We love ya Man!

[judaew]

LGTM for my ports

[lbschenkel]

I'm happy with whatever is the consensus and if the ports build.

[jmroot]

You get automatic messages when you are mentioned in a PR, e.g. if a port you maintain is touched. It does not necessarily imply you have to do anything specific.

However, you can test if your port builds against openssl 3 by adding PortGroup openssl 1.0 and in most cases simply setting openssl.branch 3 (some extra portgroup options may be needed to help some ports find the right openssl version). If it fails to build, you can set openssl.branch 1.1 instead for the time being, and ask for the port to be excluded from this PR.

[toy]

I've managed to check stressdrive against openssl-3, but for that I needed to change include and lib paths in Portfile (${prefix}/include to ${prefix}/include/openssl-3 and ${prefix}/lib to ${prefix}/lib/openssl-3). Do I understand right that with this PR the version of openssl that will be symlinked directly to include and lib will change to version 3?

[cjones051073]

I've managed to check stressdrive against openssl-3, but for that I needed to change include and lib paths in Portfile (${prefix}/include to ${prefix}/include/openssl-3 and ${prefix}/lib to ${prefix}/lib/openssl-3). Do I understand right that with this PR the version of openssl that will be symlinked directly to include and lib will change to version 3?

Please do not hardcode the paths above, but use the proc methods provided in the PG to access the correct include and lib paths. i.e. [openssl::include_dir] and [openssl::lib_dir].

But yes, there are two main ways you can use openssl.

1. The most common is to use the openssl shim port, that installs sym-links to the default openssl version (currently 1.1, to become 3 with this PR) into the primary macports prefix.
2. Use the openssl PG to configure your port to use the isolated build areas, under libexec. If you do this, but do not give a version you will use the default, which is the same as in 1. If you do give a version that will be used regardless of whatever the default is.

Right now without this PR to test against openss3 you have to follow 2., as the default is 1.1. Once this PR is committed though you are correct the default will become 3, so if your port is OK with this you can if you prefer carry on with approach 1. and just use the shim port, with a rev-bump to trigger a rebuild.

[cjones051073]

I'm happy with whatever is the consensus and if the ports build.

I do not guarantee all the ports build as a) many don't build anyway, regardless of the openssl version and b) If a port needs to be pinned back to openssl 1.1 this will need to be done separately to this PR, I am not going to do that here. This can be tested and done before, or after, this PR is merged.

[cjones051073]

Hey Chris, given that these changes will tie up the CI build jobs until timeout, go ahead and add [skip ci] to the commit comment. Documented via the following blog post:

https://github.blog/changelog/2021-02-08-github-actions-skip-pull-request-and-push-workflows-with-skip-ci/

In the interim, I've been manually cancelling them. But that's a pain. LOL

I was not aware of that - Useful to know thanks...

[Schamschula]

I got clean rebuilds on all my packages.

However, I found two issues unrelated to this PR:

1. zeus-sqlite fails to build under Mojave (and likely any older macOS version) due to an issue with Apple clang and c++-17 (missing the filesystem include file)
2. py-ndg_httpsclient is missing the py-setuptools build dependency. I'm not sure how this could have been missed. This has to be a separate commit.

[cjones051073]

I think I am going to go ahead and merge this, as given the number of ports it touches I am constantly having to rebase it to fix conflicts as those ports get independent updates. We seem to have approval here and I have not heard anything on the mailing lists objecting. I know it hasn't been a very long period, but we are going to have to hit go on this sometime to switch to the latest openssl so it might as well be now...

[catap]

I'm testing both retdec ports, but it requires couple of hours to build each. It should work, but let me be sure.

[RJVB]

On Sunday November 07 2021 04:38:34 Chris Jones wrote: Merged #12807 into master.

No offense, but you *could* have left a little bit more reacting time for those of us who've been too busy with more (sic) urgent matters, esp. on a sunday... I was under the impression that I was (still) co-maintainer on port:qca (and then only port:qca-qt5) so didn't say anything about it. I'll have to take a look how this is going to work out for the qca*-ossl and -pkcs11 subports.

[mascguy]

Geez Chris, thanks for tying up the Buildbot Watchers for hours... ;-)

Just kidding...

[ryandesign]

@ryandesign for port ... php ...

Looks like the relevant php subports did not end up getting revbumped in this PR after all; I'll do them. afflib was missed too; see f08053a. I wonder if any others were missed.

[mascguy]

Thus far, the failures have been few and far between, so that's good!

There is one notable failure though, for cargo-c. It appears that it's being built prior to openssl perhaps...?

--->  Building cargo-c
DEBUG: system -W /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_cargo-c/cargo-c/work/cargo-c-0.7.3: /opt/local/bin/cargo update
dyld: Library not loaded: /opt/local/libexec/openssl11/lib/libssl.1.1.dylib
  Referenced from: /opt/local/bin/cargo
  Reason: image not found
Command failed: /opt/local/bin/cargo update

https://ports.macports.org/port/cargo-c/builds/

The port has a dependency on openssl, so I'd expect that to be built first... preventing this issue. @cjones051073 any thoughts/ideas?

CC: @MarcusCalhoun-Lopez

p.s. I'm also seeing a build failure locally, as is at least one of our users. Note that this is definitely not specific to Monterey, as my failure is occuring on 10.13:

63848 - cargo-c @0.7.3_1 error: pasting formed 'RUST_VERSION_OPENSSL_

[mascguy]

Two more Cargo-dependent ports, exhibiting the same issue (due to a broken openssl dylib on the buildbots):

• amp
• angle-grinder

CC: @herbygillot

[pmetzger]

As I mentioned on the bug tracker, ruby30 seems to be broken. It's mysterious because it's part of portgroup openssl 1.0

https://trac.macports.org/ticket/63845

[CamilleScholtz]

Cargo-c indeed seems to be failing, coincidentally the port is very outdated.

[cjones051073]

Thus far, the failures have been few and far between, so that's good!

There is one notable failure though, for cargo-c. It appears that it's being built prior to openssl perhaps...?

--->  Building cargo-c
DEBUG: system -W /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_cargo-c/cargo-c/work/cargo-c-0.7.3: /opt/local/bin/cargo update
dyld: Library not loaded: /opt/local/libexec/openssl11/lib/libssl.1.1.dylib
  Referenced from: /opt/local/bin/cargo
  Reason: image not found
Command failed: /opt/local/bin/cargo update

https://ports.macports.org/port/cargo-c/builds/

The port has a dependency on openssl, so I'd expect that to be built first... preventing this issue. @cjones051073 any thoughts/ideas?

CC: @MarcusCalhoun-Lopez

p.s. I'm also seeing a build failure locally, as is at least one of our users. Note that this is definitely not specific to Monterey, as my failure is occuring on 10.13:

63848 - cargo-c @0.7.3_1 error: pasting formed 'RUST_VERSION_OPENSSL_

the cargo port needed a rev-bump, it was missed here. I have already submitted that.

[essandess]

OpenSSL3 breaks the port tor build: https://trac.macports.org/ticket/63870.

fixed by a selfupdate.

[RJVB]

So, mostly out of curiosity, now that I'm merging these changes into my own ports tree: Am I right that there was no immediate need to rev-bump the dependents because they would have continued to find the OpenSSL 1.1.1 libraries in the same locations? I also have my questions as to the use of an all-in subprefix in libexec with a bunch of symlinks, instead of using `configure` options to specify the same versioned directories (like lib/openssl-XY) to install just the categories that contain files which have the same name for all opensslXY ports. I've been doing that in my own implementation and it works just fine, but well, whatever :)

[gctwnl]

Now that NLNetLabs has fixed their cert I wanted to go on with an NSD update and that also means OpenSSL3 (as that has been released in MacPorts). I started a compile of the existing NSD version.

OpenSSL3 breaks the existing NSD port (v4.2.1) as openssl has removed ERR_GET_FUNC

NSD 4.3.8 builds fine. I am testing (lightly) and will prepare a pull request for NSD asap.