Update OpenSSL to version 3.0.9 in the base system #760

khorben · 2023-06-01T18:09:17Z

This branch is the closest I am currently to a functional update to OpenSSL 3.0.9 in FreeBSD's base system. It was started from an update to the vendor/openssl-3.0 branch. (see https://reviews.freebsd.org/D40365)

The thorough review expected should include:

The configuration applied (default from the corresponding security/openssl30 port with the legacy provider enabled)
Security patches up to date (the 3.0.9 version was released on May 30th, 2023)
Other patches in the corresponding port may be relevant too (I have not included them)
API changes reflected accurately in secure/lib/libcrypto/Version.map and secure/lib/libssl/Version.map
Whether the choice of 30 as SHLIB_MAJOR is good for the .so files (upstream's 3 has already been obsoleted in FreeBSD's base system)
Functional tests on Tier-1 architectures (amd64, aarch64...)
Relevant and functional ossl-modules providers
Patches to software using OpenSSL are correct (currently libarchive, dumpon, and Kerberos)
This does not touch the assembly OpenSSL files that were moved into sys/crypto/openssl to avoid any trouble with the kernel, and re-imports them into secure/lib/libcrypto/arch instead as per Makefile.asm; security fixes beware of both locations.

In most software users of OpenSSL, a compatibility compilation flag was used in order to expose and use the former OpenSSL 1.1 API, which is still provided by OpenSSL 3.0 on request.

PR: 271615
Sponsored by: The FreeBSD Foundation

Summary: Release notes can be found at https://www.openssl.org/news/openssl-3.0-notes.html . Obtained from: https://www.openssl.org/source/openssl-3.0.8.tar.gz Differential Revision: https://reviews.freebsd.org/D38835 Test Plan: ``` $ git status On branch vendor/openssl-3.0 nothing to commit, working tree clean $ (cd ..; fetch http://www.openssl.org/source/openssl-${OSSLVER}.tar.gz http://www.openssl.org/source/openssl-${OSSLVER}.tar.gz.asc) openssl-3.0.8.tar.gz 14 MB 4507 kBps 04s openssl-3.0.8.tar.gz.asc 833 B 10 MBps 00s $ set | egrep '(XLIST|OSSLVER)=' OSSLVER=3.0.8 XLIST=FREEBSD-Xlist $ gpg --list-keys /home/ngie/.gnupg/pubring.kbx ----------------------------- pub rsa4096 2014-10-04 [SC] 7953AC1FBC3DC8B3B292393ED5E9E43F7DF9EE8C uid [ unknown] Richard Levitte <richard@levitte.org> uid [ unknown] Richard Levitte <levitte@lp.se> uid [ unknown] Richard Levitte <levitte@openssl.org> sub rsa4096 2014-10-04 [E] $ gpg --verify openssl-${OSSLVER}.tar.gz.asc openssl-${OSSLVER}.tar.gz gpg: Signature made Tue Feb 7 05:43:55 2023 PST gpg: using RSA key 7953AC1FBC3DC8B3B292393ED5E9E43F7DF9EE8C gpg: Good signature from "Richard Levitte <richard@levitte.org>" [unknown] gpg: aka "Richard Levitte <levitte@lp.se>" [unknown] gpg: aka "Richard Levitte <levitte@openssl.org>" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 7953 AC1F BC3D C8B3 B292 393E D5E9 E43F 7DF9 EE8C $ (cd vendor.checkout/; git status; find . -type f -or -type l | cut -c 3- | sort > ../old) On branch vendor/openssl-3.0 nothing to commit, working tree clean $ tar -x -X $XLIST -f ../openssl-${OSSLVER}.tar.gz -C .. $ rsync --exclude FREEBSD.* --delete -avzz ../openssl-${OSSLVER}/* . $ cat .git gitdir: /home/ngie/git/freebsd-src/.git/worktrees/vendor.checkout $ diff -arq ../openssl-3.0.8 . Only in .: .git Only in .: FREEBSD-Xlist Only in .: FREEBSD-upgrade $ git status FREEBSD* On branch vendor/openssl-3.0 nothing to commit, working tree clean $ ``` Reviewers: emaste, jkim Subscribers: imp, andrew, dab Differential Revision: https://reviews.freebsd.org/D38835

Summary: Release notes can be found at https://www.openssl.org/news/openssl-3.0-notes.html . Obtained from: https://www.openssl.org/source/openssl-3.0.9.tar.gz Test Plan: ``` $ git status On branch vendor/openssl-3.0 Your branch is up to date with 'origin/vendor/openssl-3.0'. nothing to commit, working tree clean $ (cd ..; fetch http://www.openssl.org/source/openssl-${OSSLVER}.tar.gz http://www.openssl.org/source/openssl-${OSSLVER}.tar.gz.asc) openssl-3.0.9.tar.gz 14 MB 74 MBps 01s openssl-3.0.9.tar.gz.asc 833 B 10 MBps 00s $ set | egrep '(XLIST|OSSLVER)=' OSSLVER=3.0.9 XLIST=FREEBSD-Xlist $ gpg --list-keys /home/khorben/.gnupg/pubring.kbx -------------------------------- pub rsa4096 2021-07-16 [SC] [expires: 2031-07-14] A21FAB74B0088AA361152586B8EF1A6BA9DA2D5C uid [ unknown] Tomáš Mráz <tm@t8m.info> uid [ unknown] Tomáš Mráz <tomas@arleto.cz> uid [ unknown] Tomáš Mráz <tomas@openssl.org> sub rsa4096 2021-07-16 [S] [expires: 2027-07-15] sub rsa4096 2021-07-16 [E] [expires: 2031-07-14] $ gpg --verify ../openssl-${OSSLVER}.tar.gz.asc ../openssl-${OSSLVER}.tar.gz gpg: Signature made Tue May 30 14:32:24 2023 CEST gpg: using RSA key DC7032662AF885E2F47F243F527466A21CA79E6D gpg: Good signature from "Tomáš Mráz <tm@t8m.info>" [unknown] gpg: aka "Tomáš Mráz <tomas@arleto.cz>" [unknown] gpg: aka "Tomáš Mráz <tomas@openssl.org>" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: A21F AB74 B008 8AA3 6115 2586 B8EF 1A6B A9DA 2D5C Subkey fingerprint: DC70 3266 2AF8 85E2 F47F 243F 5274 66A2 1CA7 9E6D $ tar -x -X $XLIST -f ../openssl-${OSSLVER}.tar.gz -C .. $ rsync --exclude FREEBSD.* --delete -avzz ../openssl-${OSSLVER}/* . [...] $ diff -arq ../openssl-${OSSLVER} . Only in .: .git Only in .: FREEBSD-Xlist Only in .: FREEBSD-upgrade $ git status FREEBSD* On branch vendor/openssl-3.0 Your branch is up to date with 'origin/vendor/openssl-3.0'. nothing to commit, working tree clean ```

khorben · 2023-06-01T18:33:08Z

cc @bsdjhb @emaste

khorben · 2023-06-01T18:37:28Z

I realized moments ago that I haven't taken care of the manual pages yet; this will also require some work here.

khorben · 2023-06-02T16:20:00Z

The new manual pages are now generated, and the obsolete files listed.

khorben · 2023-06-02T22:51:19Z

9669855 is the HEAD of my candidate for inclusion.
The two checks failing here do not seem to be related to these changes to me; they both choke like this:

===> usr.bin/factor/tests (install)
install -N /tmp/cirrus-ci-build/etc -U -M /usr/obj/tmp/cirrus-ci-build/amd64.amd64/worldstage//METALOG -D /usr/obj/tmp/cirrus-ci-build/amd64.amd64/worldstage -T package=tests -o root  -g wheel -m 555  factor_tests  /usr/obj/tmp/cirrus-ci-build/amd64.amd64/worldstage/usr/tests/usr.bin/factor/factor_tests
install: /usr/obj/tmp/cirrus-ci-build/amd64.amd64/worldstage/usr/tests/usr.bin/factor/factor_tests: No such file or directory
*** Error code 71

kevans91 · 2023-06-06T03:27:00Z

9669855 is the HEAD of my candidate for inclusion. The two checks failing here do not seem to be related to these changes to me; they both choke like this:

===> usr.bin/factor/tests (install)
install -N /tmp/cirrus-ci-build/etc -U -M /usr/obj/tmp/cirrus-ci-build/amd64.amd64/worldstage//METALOG -D /usr/obj/tmp/cirrus-ci-build/amd64.amd64/worldstage -T package=tests -o root  -g wheel -m 555  factor_tests  /usr/obj/tmp/cirrus-ci-build/amd64.amd64/worldstage/usr/tests/usr.bin/factor/factor_tests
install: /usr/obj/tmp/cirrus-ci-build/amd64.amd64/worldstage/usr/tests/usr.bin/factor/factor_tests: No such file or directory
*** Error code 71

Note that 8ec9669 reverts the pertinent part of etc/mtree/BSD.tests.dist that would've prevented that particular error.

jurajlutter · 2023-06-06T08:03:32Z

@khorben please, could you rebase on top of today's HEAD?

khorben · 2023-06-07T18:44:03Z

@khorben please, could you rebase on top of today's HEAD?

Hi @jurajlutter, sorry I got sick yesterday; I have just managed to push a first attempt at rebasing at https://github.com/khorben/freebsd-src/tree/khorben/openssl-3.0.9-rebased.
I haven't build-tested it yet.

khorben · 2023-06-07T19:44:41Z

Note that 8ec9669 reverts the pertinent part of etc/mtree/BSD.tests.dist that would've prevented that particular error.

I had fixed this issue up the chain of commits in 435a2ee already when creating this PR, so this shouldn't be related.

kevans91 · 2023-06-07T20:30:41Z

Note that 8ec9669 reverts the pertinent part of etc/mtree/BSD.tests.dist that would've prevented that particular error.

I had fixed this issue up the chain of commits in 435a2ee already when creating this PR, so this shouldn't be related.

It's not clear that we're talking about the same thing. In the current version of this branch, factor is still missing from etc/mtree/BSD.tests.dist: https://github.com/khorben/freebsd-src/blob/khorben/openssl-3.0.9/etc/mtree/BSD.tests.dist#L1039

There's no commit after 8ec9669 in this branch that puts factor back.

khorben · 2023-06-09T20:04:40Z

It's not clear that we're talking about the same thing. In the current version of this branch, factor is still missing from etc/mtree/BSD.tests.dist: https://github.com/khorben/freebsd-src/blob/khorben/openssl-3.0.9/etc/mtree/BSD.tests.dist#L1039

There's no commit after 8ec9669 in this branch that puts factor back.

Yes I'm sorry I remembered an issue with ObsoleteFiles.inc and looked at that instead. I believe the latest push at 5c15257 actually addresses what you reported; thanks!

kevans91 · 2023-06-09T20:29:32Z

It's not clear that we're talking about the same thing. In the current version of this branch, factor is still missing from etc/mtree/BSD.tests.dist: https://github.com/khorben/freebsd-src/blob/khorben/openssl-3.0.9/etc/mtree/BSD.tests.dist#L1039
There's no commit after 8ec9669 in this branch that puts factor back.

Yes I'm sorry I remembered an issue with ObsoleteFiles.inc and looked at that instead. I believe the latest push at 5c15257 actually addresses what you reported; thanks!

Yup, thanks! I had opened PR khorben#2 in case that was easier, but I've closed that now as it's been addressed and install looks happier now.

OPENSSL_API_COMPAT can be used to specify the OpenSSL API version in use for the purpose of hiding deprecated interfaces and enabling the appropriate deprecation notices. This change is a NFC while we're still using OpenSSL 1.1.1 but will avoid deprecation warnings upon the switch to OpenSSL 3.0. A future update may migrate to use the OpenSSL 3.0 APIs. PR: 271615 Pull request: freebsd#757 Sponsored by: The FreeBSD Foundation

OPENSSL_API_COMPAT can be used to specify the OpenSSL API version in use for the purpose of hiding deprecated interfaces and enabling the appropriate deprecation notices. This change is a NFC while we're still using OpenSSL 1.1.1 but will avoid deprecation warnings upon the switch to OpenSSL 3.0. A future update may migrate to use the OpenSSL 3.0 APIs. PR: 271615 Sponsored by: The FreeBSD Foundation

This API is not supported on 32-bit platforms, or on big endian platforms.

Even though the .so file is at 30 in FreeBSD base (and perhaps wrongly at 12 in security/openssl30), calculations for API compatibility should match upstream here at 3.

Reported by @otis@bsd.network from Mastodon; thanks!

Use __SIZEOF_LONG__ to define either SIXTY_FOUR_BIT_LONG or THIRTY_TWO_BIT consistenly in crypto's bn_conf.h and openssl's configuration.h. Otherwise, for example on i386, the openssl bignum routines will attempt to use 32-bit shifts on 32-bit unsigned longs, which is undefined behavior.

vstakhov · 2023-06-14T20:06:33Z

It seems you have forgotten to add file ec/ec_deprecated.c to SRCS in secure/lib/libcrypto/Makefile. It causes troubles with some symbols that are defined in the includes but missing from the libcrypto.

@vstakhov

Reported by: Vsevolod Stakhov (@vstakhov on GitHub)

vstakhov · 2023-06-15T14:21:56Z

secure/lib/libcrypto/Makefile

@@ -210,8 +210,8 @@ SRCS+=	dso_dlfcn.c dso_err.c dso_lib.c
 # ec
 SRCS+=	curve25519.c curve448.c curve448_tables.c ec2_oct.c ec2_smpl.c
 SRCS+=	ec_ameth.c ec_asn1.c ec_backend.c ec_check.c ec_curve.c ec_cvt.c
-SRCS+=	ec_err.c ec_key.c ec_kmeth.c ec_lib.c ec_mult.c ec_oct.c ec_pmeth.c
-SRCS+=	ec_print.c ecdh_kdf.c ecdh_ossl.c ecdsa_ossl.c ecdsa_sign.c
+SRCS+=	ec_deprecated.c ec_err.c ec_key.c ec_kmeth.c ec_lib.c ec_mult.c ec_oct.c


jurajlutter · 2023-06-15T18:31:30Z

For me: In file included from /usr/src/lib/libradius/radlib.c:38: In file included from /usr/obj/usr/src/amd64.amd64/tmp/usr/include/openssl/hmac.h:14: /usr/obj/usr/src/amd64.amd64/tmp/usr/include/openssl/macros.h:139:4: error: "The requested API level higher than the configured API compatibility level" # error "The requested API level higher than the configured API compatibility level"

khorben · 2023-06-15T22:15:53Z

Hi Juraj,

For me:
In file included from /usr/src/lib/libradius/radlib.c:38: In file included from
/usr/obj/usr/src/amd64.amd64/tmp/usr/include/openssl/hmac.h:14:
/usr/obj/usr/src/amd64.amd64/tmp/usr/include/openssl/macros.h:139:4:
error: "The requested API level higher than the configured API compatibility level"
# error "The requested API level higher than the configured API compatibility level"

Would you have more information about your build environment? Ideally the contents of your make.conf file if you configured it, or any other build option?

emaste · 2023-06-24T00:07:24Z

b077aed

juikim and others added 5 commits February 28, 2023 19:28

Import OpenSSL 1.1.1s

aba33b3

Import OpenSSL 1.1.1t

3c320f4

Merge commit '68967d66b12523adf3fd480112005bb0b5e75cb1' into main

ed6f360

khorben force-pushed the khorben/openssl-3.0.9 branch from 368c547 to b3cdc8f Compare June 1, 2023 18:19

khorben changed the title ~~khorben/openssl 3.0.9~~ Update OpenSSL to version 3.0.9 in the base system Jun 1, 2023

khorben mentioned this pull request Jun 1, 2023

Update OpenSSL to version 3.0.8 in the base system #740

Closed

khorben force-pushed the khorben/openssl-3.0.9 branch 5 times, most recently from 882d171 to 98d6bf8 Compare June 2, 2023 16:18

khorben force-pushed the khorben/openssl-3.0.9 branch 2 times, most recently from 6a7cd56 to 9669855 Compare June 2, 2023 17:44

khorben marked this pull request as ready for review June 2, 2023 17:50

khorben mentioned this pull request Jun 7, 2023

khorben/openssl 3.0.9 rebased #770

Closed

khorben force-pushed the khorben/openssl-3.0.9 branch from d098e01 to 5c15257 Compare June 9, 2023 20:02

openssl: Import the changes specific to FreeBSD

34f8c8d

khorben and others added 20 commits June 14, 2023 20:38

openssl: Generate the assembly files

28a2874

openssl: Import the other files generated

4391b1b

libhx509: Request the OpenSSL 1.1 API

f521dea

libkrb5: Request the OpenSSL 1.1 API

73172b6

libheimntlm: Request the OpenSSL 1.1 API

4ba29f0

libkdc: Request the OpenSSL 1.1 API

0f4f0b2

libgssapi_krb5: Request the OpenSSL 1.1 API

18dddc0

libgssapi_ntlm: Request the OpenSSL 1.1 API

d57caf4

hxtool: Request the OpenSSL 1.1 API

f55705e

libarchive: Avoid a build failure with OpenSSL 3.0

a80e27e

openssl: Automatically disable EC_NISTP_64_GCC_128

b582bcf

This API is not supported on 32-bit platforms, or on big endian platforms.

openssl: Keep OPENSSL_SHLIB_VERSION at 3

bdf80e9

Even though the .so file is at 30 in FreeBSD base (and perhaps wrongly at 12 in security/openssl30), calculations for API compatibility should match upstream here at 3.

openssl: Re-generate the manual pages

e134bb3

openssl: Mark deprecated manual pages as obsolete

a19b8bd

openssl: Limit the changes to obsolete files to us

934331f

openssl: Fix typos in manual pages

d32019d

openssl: Enable support for RFC3779

ef1c068

Reported by @otis@bsd.network from Mastodon; thanks!

openssl: also build ec_deprecated.c in libcrypto

1335516

Reported by: Vsevolod Stakhov (@vstakhov on GitHub)

khorben force-pushed the khorben/openssl-3.0.9 branch from 61d17d8 to 1335516 Compare June 15, 2023 14:21

vstakhov reviewed Jun 15, 2023

View reviewed changes

emaste closed this Jun 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update OpenSSL to version 3.0.9 in the base system #760

Update OpenSSL to version 3.0.9 in the base system #760

khorben commented Jun 1, 2023 •

edited

Loading

khorben commented Jun 1, 2023

khorben commented Jun 1, 2023

khorben commented Jun 2, 2023

khorben commented Jun 2, 2023 •

edited

Loading

kevans91 commented Jun 6, 2023

jurajlutter commented Jun 6, 2023

khorben commented Jun 7, 2023

khorben commented Jun 7, 2023

kevans91 commented Jun 7, 2023

khorben commented Jun 9, 2023

kevans91 commented Jun 9, 2023

vstakhov commented Jun 14, 2023

vstakhov Jun 15, 2023

jurajlutter commented Jun 15, 2023

khorben commented Jun 15, 2023

emaste commented Jun 24, 2023

Update OpenSSL to version 3.0.9 in the base system #760

Update OpenSSL to version 3.0.9 in the base system #760

Conversation

khorben commented Jun 1, 2023 • edited Loading

khorben commented Jun 1, 2023

khorben commented Jun 1, 2023

khorben commented Jun 2, 2023

khorben commented Jun 2, 2023 • edited Loading

kevans91 commented Jun 6, 2023

jurajlutter commented Jun 6, 2023

khorben commented Jun 7, 2023

khorben commented Jun 7, 2023

kevans91 commented Jun 7, 2023

khorben commented Jun 9, 2023

kevans91 commented Jun 9, 2023

vstakhov commented Jun 14, 2023

vstakhov Jun 15, 2023

Choose a reason for hiding this comment

jurajlutter commented Jun 15, 2023

khorben commented Jun 15, 2023

emaste commented Jun 24, 2023

khorben commented Jun 1, 2023 •

edited

Loading

khorben commented Jun 2, 2023 •

edited

Loading