Try migration to OpenSSL #53891
Try migration to OpenSSL #53891
Conversation
inkydragon
added
build
|
OK I'm learning about stdlib as I go along, feedback is very welcome. I think I've done what is needed to add OpenSSL_jl to stdlib. It allows me to build locally. There is one thing I think I missed, but I don't know how to fix it: I can't import OpenSSL_jll. I think it's built (because I had a bug in it at some point, and it was complaining which meant it tried to compile it, and now it doesn't complain anymore)… but I'm not sure. |
|
OK, somehow the OpenSSL_jll sources are not getting copied to the right directory. I see this message (not an error, but worrying): |
|
I think you need to add it to |
fxcoudert
force-pushed
the
openssl
branch
2 times, most recently
from
df3f08b
to
bbc918d
Compare
Thanks, that works. |
|
Once we finish JuliaPackaging/Yggdrasil#8377 (and JuliaPackaging/Yggdrasil#8386) and the corresponding builds are updated here, we can remove mbedTLS_jll as stdlib from here, since it isn't used for anything else, I think. |
|
Build works for me locally, but on CI it fails with: |
|
I'm not sure, but maybe that's during documentation building? |
|
Yes, it's during docs building, which I'm not doing locally (not sure what the |
|
@IanButterworth @KristofferC do I understand correctly we need to update |
|
@giordano yeah any manifests may need to be re-resolved |
May I ask how this is done? I couldn't figure it out from the doc. |
fxcoudert
force-pushed
the
openssl
branch
2 times, most recently
from
7c1c2de
to
44859e1
Compare
|
Manually updated |
|
It usually requires Pkg to resolve it to be accurate I just pushed the result of that. I also checked for other manifests in this repo that need updating and there doesn't appear to be any. Surprisingly there's only one in Pkg https://github.com/IanButterworth/Pkg.jl/blob/205309092c861d58bf787628ccbff4f5a41a0840/test/test_packages/ShouldPreserveSemver/Manifest.toml#L90 |
|
Thanks @IanButterworth That does not seem to pass CI (https://buildkite.com/julialang/julia-master/builds/35131#018e94a6-3c00-4b01-8b2a-8797e5b2ddf0), but… I can now run that command from my branch. I don't get the same output as you, here is the diff between your version and mine: Some versions and commit ids are different. But maybe more importantly, it does remove the |
|
OK, that makes the build pass on FreeBSD and macOS (both Intel and ARM). It's also failing on Windows. It's failing on My guess is that it's because in So: should I got back to the OpenSSL package and make it put libraries in On Windows, the naming convention is I'll let testing finish here, but I think Windows can be handled with: diff --git a/stdlib/OpenSSL_jll/src/OpenSSL_jll.jl b/stdlib/OpenSSL_jll/src/OpenSSL_jll.jl
index 26eb3d243e..bba9a0a299 100644
--- a/stdlib/OpenSSL_jll/src/OpenSSL_jll.jl
+++ b/stdlib/OpenSSL_jll/src/OpenSSL_jll.jl
@@ -3,7 +3,7 @@
## dummy stub for https://github.com/JuliaBinaryWrappers/OpenSSL_jll.jl
baremodule OpenSSL_jll
-using Base, Libdl
+using Base, Libdl, Base.BinaryPlatforms
const PATH_list = String[]
const LIBPATH_list = String[]
@@ -20,8 +20,13 @@ libssl_handle::Ptr{Cvoid} = C_NULL
libssl_path::String = ""
if Sys.iswindows()
- const libcrypto = "libcrypto.dll"
- const libssl = "libssl.dll"
+ if arch(HostPlatform()) == "x86_64"
+ const libcrypto = "libcrypto-3-x64.dll"
+ const libssl = "libssl-3-x64.dll"
+ else
+ const libcrypto = "libcrypto-3.dll"
+ const libssl = "libssl-3.dll"
+ end
elseif Sys.isapple()
const libcrypto = "@rpath/libcrypto.3.dylib"
const libssl = "@rpath/libssl.3.dylib" |
|
Status:
I have to dig to understand why this used to work and doesn't anymore. It is working for me on a full source build, but not with binarybuilder. Go figure… |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Yes, we stumbled upon that already: JuliaPackaging/Yggdrasil#7576 (comment) |
It works locally for me with BinaryBuilder binaries: but it's possible libgit2/openssl is picking up from my system whatever certificates it needs. Maybe we somehow need to inform openssl/libgit2 where the certificates are ( |
|
@nanosoldier |
|
@quinnj what do you want to do about the legacy plugin? Should we ship in in a separate package? Is it ok (and feasible) to ship it in a separate package? Or maybe you want to stop supporting it? |
|
The package evaluation job you requested has completed - possible new issues were detected. |
|
With JuliaRegistries/General#104322 to address #53891 (comment) we're now down from 100 to 15 failing packages 🙂 As far as I can tell, main issues are OpenSSL.jl (see above), and PyCall not able to download stuff (see e.g. ScikitLearn.jl), which is very likely relevant to this PR. Edit: I can't reproduce locally the error with PyCall not able to download files 😕 We may still need to tweak something in the registry, because UnitfulRecipes.jl is failing, but it's due to installing a very old (and apparently incompatible) version of GR.jl (we get GR@0.52.0, which doesn't work for me on aarch64-darwin even on Julia v1.9, so that's very much unrelated to this PR). GR@v0.73 works fine for me on this PR, |
Sorry, not quite following; what is your linked comment supposed to link to? I assume we're talking about the mariadb connector? Is there a piece there that isn't upgrading cleanly? |
|
If you look at the linked log https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/955b85b_vs_a69aa30/OpenSSL.primary.log, tests are failing when trying to call the legacy plugin This is failing because we aren't currently installing the legacy module here |
|
I looked up your "DSO support routines" error and maybe worth to try this simple:
https://askubuntu.com/a/1410124 I think this basically disables OpenSSL, or its security/old cipher or something, but it it may help PkgEval to show which package work; after some (insecure) download (so only do as a temp workaround until right config found, then re-PkgEval). I guess the /etc/ssl/openssl.cnf file needs changing, you look at it but it's for the older OpenSLL installed on the testing machine asking for something now in legacy, and you could just not use that thus not the legacy .so file. [This issue would also be bypassed if system OpenSSL is just used, since then it would match its config file?] |
I think @mkitti added that code when we did the 1 -> 3 upgrade. @mkitti, do you have an opinion on the legacy plugin? I don't personally know of any direct uses of it, so I'd be fine removing and doing a breaking release. We could at least make the tests probably versioned to run only if it's loaded? |
|
I have no specific opinion on the matter other than making things just work. Honestly, I need review my own changes. |
|
About the legacy modules.
We don't want to modify the configuration file, so two options are available:
The second option seems more reasonable, unless there is evidence they are actually used in actual code. Enabling loading of modules that upstream describes as “fallen out of use, have been deemed insecure by the cryptography community” does not appear to be good practice. |
|
Option 2: JuliaWeb/OpenSSL.jl#34 |
This is an old and unmaintained package (jw3126/UnitfulRecipes.jl#81), so even if we had to fix anything in that package, nothing could be done about it. Note: this is currently failing tests in JuliaLang/julia#53891 (comment) because it's trying to install a very ancient version of `GR.jl`, I don't see any fix possible if the package is deprecated.
|
I'm having a look at the most recent failing tests. There are three packages which require a missing binary dependency:
|
This is an old and unmaintained package (jw3126/UnitfulRecipes.jl#81), so even if we had to fix anything in that package, nothing could be done about it. Note: this is currently failing tests in JuliaLang/julia#53891 (comment) because it's trying to install a very ancient version of `GR.jl`, I don't see any fix possible if the package is deprecated.
|
Is there anything more I can do to help push this over the finish line? |
|
I was waiting for a new OpenSSL.jl version with the new update tests, to be able to make sure it's compatible with this branch. UnitfulRecipes.jl has been already excluded from future PkgEval tests (it's an unmaintained package), but I still need to adjust the General registry not to install super old versions of GR.jl. The only other remaining thing I believe is figuring out why packages calling python via PyCall, #53891 (comment), problem which I couldn't reproduce locally on this branch with x86_64-linux nor aarch64-darwin. |
|
Cannot reproduce Test with:
ScikitLearn.jl passed all tests. |
|
I think you have the OpenSSL.jl version now: JuliaRegistries/General#105279 The Python error appears to be as follows: My current guess is that PyCall.jl is pulling in a version of Python compiled against OpenSSL 3.2.1. Conda-forge does seem to have an openssl 3.0 branch in their feedstock: If I just request Python via conda/mamba, I get the following install: If I also specify that I want openssl 3.0, then I get the following packages: If we continue on the current course of building official Julia against OpenSSL 3.0 LTS (supported through September 7th, 2026), then we may need to be proactive about requesting a Python that also uses OpenSSL 3.0. |
|
Here's a minimum demonstration of the Python issue via PythonCall: using Pkg
Pkg.activate(; temp=true)
Pkg.add(["OpenSSL_jll", "PythonCall"])
using OpenSSL_jll, PythonCall, SHA
if !isempty(ARGS) && ARGS[1] == "--fix"
using NetworkOptions
ENV["SSL_CERT_FILE"] = NetworkOptions.ca_roots()
end
urllib_request = pyimport("urllib.request")
f = urllib_request.urlopen("https://ndownloader.figshare.com/files/5976036")
bytes = pyconvert(Vector, f.read())
f.close()
# should print aaa5c9a6afe2225cc2aed2723682ae403280c4a3695a2ddda4ffb5d8215ea681
println(bytes2hex(sha256(bytes)))The problem is that the default trusted certificate store refers to julia> using OpenSSL_jll
julia> run(`$(OpenSSL_jll.openssl()) version -d`);
OPENSSLDIR: "/workspace/destdir/ssl"According the OpenSSL docs this can be overriden via the environment variables A Python-based fix is to configure a SSL context. julia> begin
using PythonCall, NetworkOptions
ssl = pyimport("ssl")
ctx = ssl.create_default_context()
ctx.load_verify_locations(NetworkOptions.ca_roots())
urllib_request = pyimport("urllib.request")
f = urllib_request.urlopen("https://ndownloader.figshare.com/files/5976036", context=ctx)
bytes = pyconvert(Vector, f.read())
f.close()
# should print aaa5c9a6afe2225cc2aed2723682ae403280c4a3695a2ddda4ffb5d8215ea681
println(bytes2hex(sha256(bytes)))
end
aaa5c9a6afe2225cc2aed2723682ae403280c4a3695a2ddda4ffb5d8215ea681 |
|
Here's Python reporting the default verify paths and environment variables. julia> begin
using OpenSSL_jll, PythonCall
ssl = pyimport("ssl")
ssl.get_default_verify_paths()
end
Python: DefaultVerifyPaths(
cafile=None, capath=None,
openssl_cafile_env='SSL_CERT_FILE',
openssl_cafile='/workspace/destdir/ssl/cert.pem',
openssl_capath_env='SSL_CERT_DIR',
openssl_capath='/workspace/destdir/ssl/certs'
)
julia> @ccall(OpenSSL_jll.libssl.X509_get_default_cert_dir()::Cstring) |> unsafe_string
"/workspace/destdir/ssl/certs"
julia> @ccall(OpenSSL_jll.libssl.X509_get_default_cert_file()::Cstring) |> unsafe_string
"/workspace/destdir/ssl/cert.pem"
julia> @ccall(OpenSSL_jll.libssl.X509_get_default_cert_file_env()::Cstring) |> unsafe_string
"SSL_CERT_FILE"
julia> @ccall(OpenSSL_jll.libssl.X509_get_default_cert_dir_env()::Cstring) |> unsafe_string
"SSL_CERT_DIR"@stevengj , @cjdoris I wonder if you had any thoughts on how to have properly configure Python's OpenSSL library if Julia is supplying the the SSL library? |
|
Why with PyCall on this branch I get while with PythonCall ? |
|
@giordano My guess is that PyCall is defaulting to relatively older install that might be loading an older OpenSSL? Meanwhile, PythonCall is setting up a brand new environment corresponding with your current Julia environment. Run the following for your PyCall install. |
|
Here's mine for what it's worth. julia> using OpenSSL_jll, PyCall
julia> py"""
import ssl
print(ssl.get_default_verify_paths())
"""
DefaultVerifyPaths(cafile=None, capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/workspace/destdir/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/workspace/destdir/ssl/certs')
|
|
Ah, I guess it's because my PyCall is actually using OpenSSL 1.1, while PythonCall uses OpenSSL 3: |
If I understand correctly, you link to OpenSSL, and Python (libpython or e.g. PythonCall?) does too, and it must be the same one, since you can only link to one .so version in the process? [You can call to R with RCall.jl etc. and getting all to use the same version might prove problematic.] But does Julia have to link at all (by default)? I mean this is only for Downloads, and indirectly Pkg, to download (or also upload? does Julia per se support that?), but does it need to be in the same process? When you call Download.download it could open a separate Julia process (or some simpler one) that does the download and only it needs to link to OpenSLL, avoiding all possibility of conflict. It can even be async (not by default? though that might be great, at least eventually). [For Pkg there's precedent, it forks processes to precompile why not for the downloads also...]
PyCall is not going away (at least in the near term, and because of packages like Pandas.jl), and you can use it with PythonCall, something to have in mind. Then you get either 1.1 with PyCall or both otherwise 3.0... |
|
@PallHaraldsson Julia currently uses mbedTLS for SSL. This PR is an attempt to move away from mbedTLS to OpenSSL. The standard library packages LibSSH2_jll, LibGit2_jll, and LibCurl_jll all currently depend on mbedTLS_jll but if this PR goes through, then they will depend on OpenSSL_jll instead. Python also depends on OpenSSL. The above discussion involves loading libpython into a Julia process. If the Julia process has already loaded libssl and libcrypto, then libpython will also attempt to use the same libssl and libcrypto. It's not compatible to have two different OpenSSL 3.x libraries loaded at the same time. PyCall creates a single miniconda environment and tends to reuse that same conda environment for all Julia environments unless configured otherwise. In contrast, PythonCall tends to build a distinct conda environment for each Julia environment. One consequence of this is that PyCall installations tend to have older versions of Python installed using OpenSSL v1, which is no longer supported upstream. Fresh installs of PyCall will usually have been updated with Python built against OpenSSL v3. Moving to an OpenSSL based standard library will make it much more likely that Julia will have dynamically loaded libssl and libcrypto. A new challenge for language interop will be ensuring that both Python and Julia can function with the same dynamically loaded OpenSSL libraries. Overall, OpenSSL based Julia is probably a good idea for reasons I have elaborated on elsewhere. Namely mbedTLS seems to be mainly targeting embedded applications whereas OpenSSL seems to more aligned with where Julia is currently used. |
|
@StefanKarpinski since you deal a bit with network-related stuff, what do you think we should do with regards to the cert file:
|
|
The above question is applicable to OpenSSL.jl sitting upstream of HTTP.jl. The constructor of https://github.com/JuliaWeb/OpenSSL.jl/blob/544711038ab32b04789f86f316f65bae3920e901/src/ssl.jl#L137 |
This is a test of migrating the source build to OpenSSL.