Undefined symbol: mbedtls_x509_crt_verify_restartable

[maleadt]

I've been encountering the following on my GitLab CI while doing coverage submission on Julia 1.1:

julia: symbol lookup error: /builds/JuliaGPU/CUDAnative.jl/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable

Full log: https://gitlab.com/JuliaGPU/CUDAnative.jl/-/jobs/151716689 -- it has worked before on 1.1 though, see eg. https://gitlab.com/JuliaGPU/CUDAnative.jl/-/jobs/149248223, but quite some packages seem to have been upgraded since (HTTP.jl, Coverage.jl, etc).

This is on a very basic Ubuntu 18.04 image with Julia binaries installed: https://github.com/JuliaGPU/gitlab-ci/blob/master/images/base/v1.1/Dockerfile

[rschwarz]

I have the same issue (see travis log) based on Ubuntu Xenial (16.04) offered by Travis.

[DilumAluthge]

I have the same error (full log) on Travis.

This is the line with the error:

julia: symbol lookup error: /home/travis/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable

These are the lines immediately preceding:

Coverage.process_file: Detecting coverage for src/version.jl
Coverage.process_folder: Skipping version.jl.3853.cov, not a .jl file
Coverage.process_file: Detecting coverage for src/welcome.jl
Coverage.process_folder: Skipping welcome.jl.3853.cov, not a .jl file
Codecov.io API URL:
https://codecov.io/upload/v2?&service=travis-org&branch=master&commit=f3ec60c9a438ab1c3a9e51b750f44f288452b24f&pull_request=false&job=173244732&slug=UnofficialJuliaMirror/MirrorUpdater.jl&build=259.22

Version information:

• Julia Version 1.0.3
• MbedTLS.jl v0.6.7
• Coverage.jl v0.6.0

[curtd]

I have the same error (full log) on Travis.

This is the line with the error:

julia: symbol lookup error: /home/travis/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable

These are the lines immediately preceding:

Coverage.process_file: Detecting coverage for src/version.jl
Coverage.process_folder: Skipping version.jl.3853.cov, not a .jl file
Coverage.process_file: Detecting coverage for src/welcome.jl
Coverage.process_folder: Skipping welcome.jl.3853.cov, not a .jl file
Codecov.io API URL:
https://codecov.io/upload/v2?&service=travis-org&branch=master&commit=f3ec60c9a438ab1c3a9e51b750f44f288452b24f&pull_request=false&job=173244732&slug=UnofficialJuliaMirror/MirrorUpdater.jl&build=259.22

Version information:

• Julia Version 1.0.3
• MbedTLS.jl v0.6.7
• Coverage.jl v0.6.0

I noticed in my own testing that HTTP.jl uses MbedTLS version 0.6.0 in its Project.toml. Manually updating this to 0.6.7 and rebuilding the package seemed to alleviate the problem.

[tkoolen]

I noticed in my own testing that HTTP.jl uses MbedTLS version 0.6.0 in its Project.toml. Manually updating this to 0.6.7 and rebuilding the package seemed to alleviate the problem.

What do you mean by alleviate? I just tried this: JuliaWeb/HTTP.jl@342d9c2, and after a pkg> build, pkg> test HTTP still fails with the error message in the issue description.

[rofinn]

Not sure if this is related, but on 0.6.7 I appear to be getting:

libmbedtls.so: undefined symbol: mbedtls_pk_verify_restartable

I seem to be getting the

undefined symbol: mbedtls_x509_crt_verify_restartable

error from the HTTP (0.8.0) tests with on both MbedTLS (0.6.7)

NOTE: I only seem to be getting this inside an amazon linux 2 docker container (macOS seems fine), but for now downgrading to 0.6.6 seems to work.

[curtd]

I noticed in my own testing that HTTP.jl uses MbedTLS version 0.6.0 in its Project.toml. Manually updating this to 0.6.7 and rebuilding the package seemed to alleviate the problem.

What do you mean by alleviate? I just tried this: JuliaWeb/HTTP.jl@342d9c2, and after a pkg> build, pkg> test HTTP still fails with the error message in the issue description.

My mistake, updating MbedTLS to v"0.6.7" fixed the issue my package was having with HTTP.request() but the tests themselves are still broken

[tkoolen]

I believe I know what's wrong. The deps.jl file generated by BinaryProvider looks like this:

## This file autogenerated by BinaryProvider.write_deps_file().
## Do not edit.
##
## Include this file within your main top-level source, and call
## `check_deps()` from within your module's `__init__()` method

if isdefined((@static VERSION < v"0.7.0-DEV.484" ? current_module() : @__MODULE__), :Compat)
    import Compat.Libdl
elseif VERSION >= v"0.7.0-DEV.3382"
    import Libdl
end
const libmbedcrypto = joinpath(dirname(@__FILE__), "usr/lib/libmbedcrypto.2.16.0.dylib")
const libmbedtls = joinpath(dirname(@__FILE__), "usr/lib/libmbedtls.12.dylib")
const libmbedx509 = joinpath(dirname(@__FILE__), "usr/lib/libmbedx509.0.dylib")
function check_deps()
    global libmbedcrypto
    if !isfile(libmbedcrypto)
        error("$(libmbedcrypto) does not exist, Please re-run Pkg.build(\"MbedTLS\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libmbedcrypto) in (C_NULL, nothing)
        error("$(libmbedcrypto) cannot be opened, Please re-run Pkg.build(\"MbedTLS\"), and restart Julia.")
    end

    global libmbedtls
    if !isfile(libmbedtls)
        error("$(libmbedtls) does not exist, Please re-run Pkg.build(\"MbedTLS\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libmbedtls) in (C_NULL, nothing)
        error("$(libmbedtls) cannot be opened, Please re-run Pkg.build(\"MbedTLS\"), and restart Julia.")
    end

    global libmbedx509
    if !isfile(libmbedx509)
        error("$(libmbedx509) does not exist, Please re-run Pkg.build(\"MbedTLS\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libmbedx509) in (C_NULL, nothing)
        error("$(libmbedx509) cannot be opened, Please re-run Pkg.build(\"MbedTLS\"), and restart Julia.")
    end

end

So it first opens libmbedcrypto, then libmbedtls, then libmbedx509. However, the mbedtls readme states:

[...] when loading shared libraries using dlopen(), you'll need to load libmbedcrypto first, then libmbedx509, before you can load libmbedtls.

This seems plausible given that mbedtls_x509_crt_verify_restartable is a symbol in libmbedx509.so.

[tkoolen]

Unfortunately, HTTP.jl tests still fail for me after changing the order locally. Still I think it is a real issue.

[rschwarz]

I thought that it was likely a version issue, since the problem did not occur on Travis/Debian, but Travis/Ubuntu. (I did not actually retry Debian recently, it's just that I remember it to work.)

[tkoolen]

The latest release of MbedTLS upgraded to a new version. Julia ships with a libmbedtls.so that's older, and on my system those are the only libmbedtls.so copies. Also the libmbedx509.so / libmbedtls.so that ship with Julia don't have mbedtls_x509_crt_verify_restartable. So I think it's likely it's somehow using the Julia-shipped libs, but I don't quite see why yet, nor why it would be platform-specific (HTTP.jl tests also pass on OSX) or distribution-specific.

[giordano]

This bug doesn't allow me to automatically setup Travis in BinaryBuilder.jl. System used: Arch Linux. As a temporary workaround I downgraded this package to v0.6.6.

[DilumAluthge]

Any update from a package maintainer? @quinnj

…

On Tue, Feb 5, 2019 at 19:24 Mosè Giordano ***@***.***> wrote: This bug doesn't allow me to automatically setup Travis in BinaryBuilder.jl. System used: Arch Linux. As a temporary workaround I downgraded this package to v0.6.6. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#193 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFXAraVo-L0GOcGtbcY8Riz90uzwcx3mks5vKiCegaJpZM4aUniZ> .

[quinnj]

Sorry for the slow response here, but I don't really have much to contribute; I haven't personally had any issues on OSX Julia 1.1, or Linux Julia 1.0/1.1; do we have a reliable reproduction somewhere where we could debug what's going on?

[DilumAluthge]

Yep, it happens on my Travis builds. Here is an example. https://travis-ci.com/UnofficialJuliaMirror/MirrorUpdater.jl/jobs/175298500

…

On Wed, Feb 6, 2019 at 15:27 Jacob Quinn ***@***.***> wrote: Sorry for the slow response here, but I don't really have much to contribute; I haven't personally had any issues on OSX Julia 1.1, or Linux Julia 1.0/1.1; do we have a reliable reproduction somewhere where we could debug what's going on? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#193 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFXArcJDW3cH5NFsrsm0RXr-224dm5vjks5vKzrKgaJpZM4aUniZ> .

[martinholters]

Just to verify: There are systems (Windows and macOS?) which are not affected by this issue? Otherwise we could bump the required Julia version of the latest MbedTLS to 1.2- in METADATA to prevent it from being installed on incompatible Julia versions.

[vtjnash]

bump. This is preventing us from getting information out of Coverage.jl: https://travis-ci.org/JuliaCI/CoverageBase.jl/jobs/501818500

[fingolfin]

This issue has been open for over a month, and apparently there is no fix in sight (at least looking in from the outside; I'd be happy to be wrong here!)

Since this is breaking tests for dozens (hundreds?) of projects, wouldn't it be better to revert the breaking change for now, make a new release, and then work on fixing the issue properly, but with all the time you need?

[ronisbr]

This issue has been open for over a month, and apparently there is no fix in sight (at least looking in from the outside; I'd be happy to be wrong here!)

Since this is breaking tests for dozens (hundreds?) of projects, wouldn't it be better to revert the breaking change for now, make a new release, and then work on fixing the issue properly, but with all the time you need?

I second that. I had to pin MbedTLS.jl to v0.6.6 to fix the issue for now.

[tkoolen]

Alright, how about this: explicitly dlopen the libs (in the right order) and change all the ccalls to explicitly use the handles. At least that seems to me to be a straightforward fix.

[quinnj]

@fingolfin, @ronisbr, please read the other comments in this thread. There's no consensus that just "reverting" the latest release actually resolves the issue (see @tanmaykm's comment and my own). At this point, I suspect that we're getting a weird interaction with the Julia Base-shipped mbedtls shared libraries conflicting with the MbedTLS.jl-shipped ones and that's causing the issues. I've spent at least 3-4 days trying to track down various root causes, but shared-library loading/symbol resolving is far from my skillset. If someone can show concrete evidence of an mbedtls library version that works on Julia 1.0 and 1.1 without issues, I'm happy to make changes and merge PRs, but so far, AFAIU, we haven't seen that kind of 100% solution.

@tkoolen, similarly, if we can show that your suggestions solve the issue here in all cases, then I'm happy to make changes. There just seems to be a lot of "hey, we should do this!" or "just revert!" comments here without anyone actually going thru the work of seeing if that solves everything.

[vtjnash]

Here's a simple quick reproducer (from inside a Julia v1.0 build):

julia> using Libdl

julia> using HTTP

julia> Libdl.dlopen("libmbedx509.so")
Ptr{Nothing} @0x000055f064544560

julia> HTTP.get("https://google.com")
./julia: symbol lookup error: /home/vtjnash/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable

...and at this point the process is now killed also.

[vtjnash]

Also, for a debugging trick, set export LD_BIND_NOW=1. This'll cause the errors to surface as assertion failures much sooner. Than we can, for example, just do this minimal demo:

julia> using Libdl

julia> Libdl.dlopen("libmbedtls.so")
Ptr{Nothing} @0x000055e22e1c1830

julia> Libdl.dlopen("/home/vtjnash/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so")
ERROR: could not load library "/home/vtjnash/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so"
/home/vtjnash/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable
Stacktrace:
 [1] dlopen(::String, ::UInt32) at /data/vtjnash/julia10/usr/share/julia/stdlib/v1.0/Libdl/src/Libdl.jl:97 (repeats 2 times)
 [2] top-level scope at none:0

[ronisbr]

This bug was filled on the same day v0.6.7 was released and no one has reported that reverting back to v0.6.6 did not fix the issue. Hence, there is a strong correlation. Of course I cannot know if this fixes the issues for everyone, but it did fix in all repositories I tested so far. The previous version can still have problems but those seem at least more sparse than the current one.

[davidanthoff]

In particular, I read @tanmaykm's comment as saying that reverting did fix things for him.

[quinnj]

@staticfloat and I had a good debugging session today and have a pretty good handle on the root cause (julia-shipped mbedtls binaries interfering w/ MbedTLS.jl binaries). I'll try to get things cleaned up and get a new release out soon.

[fingolfin]

Works for me now in Coverage.jl -- thanks!!!