Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Undefined symbol: mbedtls_x509_crt_verify_restartable #193

Closed
maleadt opened this Issue Jan 27, 2019 · 51 comments

Comments

Projects
None yet
@maleadt
Copy link

maleadt commented Jan 27, 2019

I've been encountering the following on my GitLab CI while doing coverage submission on Julia 1.1:

julia: symbol lookup error: /builds/JuliaGPU/CUDAnative.jl/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable

Full log: https://gitlab.com/JuliaGPU/CUDAnative.jl/-/jobs/151716689 -- it has worked before on 1.1 though, see eg. https://gitlab.com/JuliaGPU/CUDAnative.jl/-/jobs/149248223, but quite some packages seem to have been upgraded since (HTTP.jl, Coverage.jl, etc).

This is on a very basic Ubuntu 18.04 image with Julia binaries installed: https://github.com/JuliaGPU/gitlab-ci/blob/master/images/base/v1.1/Dockerfile

@leethargo

This comment has been minimized.

Copy link

leethargo commented Jan 27, 2019

I have the same issue (see travis log) based on Ubuntu Xenial (16.04) offered by Travis.

@DilumAluthge

This comment has been minimized.

Copy link

DilumAluthge commented Jan 27, 2019

I have the same error (full log) on Travis.

This is the line with the error:

julia: symbol lookup error: /home/travis/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable

These are the lines immediately preceding:

Coverage.process_file: Detecting coverage for src/version.jl
Coverage.process_folder: Skipping version.jl.3853.cov, not a .jl file
Coverage.process_file: Detecting coverage for src/welcome.jl
Coverage.process_folder: Skipping welcome.jl.3853.cov, not a .jl file
Codecov.io API URL:
https://codecov.io/upload/v2?&service=travis-org&branch=master&commit=f3ec60c9a438ab1c3a9e51b750f44f288452b24f&pull_request=false&job=173244732&slug=UnofficialJuliaMirror/MirrorUpdater.jl&build=259.22

Version information:

  • Julia Version 1.0.3
  • MbedTLS.jl v0.6.7
  • Coverage.jl v0.6.0
@curtd

This comment has been minimized.

Copy link

curtd commented Jan 30, 2019

I have the same error (full log) on Travis.

This is the line with the error:

julia: symbol lookup error: /home/travis/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable

These are the lines immediately preceding:

Coverage.process_file: Detecting coverage for src/version.jl
Coverage.process_folder: Skipping version.jl.3853.cov, not a .jl file
Coverage.process_file: Detecting coverage for src/welcome.jl
Coverage.process_folder: Skipping welcome.jl.3853.cov, not a .jl file
Codecov.io API URL:
https://codecov.io/upload/v2?&service=travis-org&branch=master&commit=f3ec60c9a438ab1c3a9e51b750f44f288452b24f&pull_request=false&job=173244732&slug=UnofficialJuliaMirror/MirrorUpdater.jl&build=259.22

Version information:

  • Julia Version 1.0.3
  • MbedTLS.jl v0.6.7
  • Coverage.jl v0.6.0

I noticed in my own testing that HTTP.jl uses MbedTLS version 0.6.0 in its Project.toml. Manually updating this to 0.6.7 and rebuilding the package seemed to alleviate the problem.

@tkoolen

This comment has been minimized.

Copy link

tkoolen commented Jan 30, 2019

I noticed in my own testing that HTTP.jl uses MbedTLS version 0.6.0 in its Project.toml. Manually updating this to 0.6.7 and rebuilding the package seemed to alleviate the problem.

What do you mean by alleviate? I just tried this: JuliaWeb/HTTP.jl@342d9c2, and after a pkg> build, pkg> test HTTP still fails with the error message in the issue description.

@rofinn

This comment has been minimized.

Copy link

rofinn commented Jan 30, 2019

Not sure if this is related, but on 0.6.7 I appear to be getting:

libmbedtls.so: undefined symbol: mbedtls_pk_verify_restartable

I seem to be getting the

undefined symbol: mbedtls_x509_crt_verify_restartable

error from the HTTP (0.8.0) tests with on both MbedTLS (0.6.7)

NOTE: I only seem to be getting this inside an amazon linux 2 docker container (macOS seems fine), but for now downgrading to 0.6.6 seems to work.

@curtd

This comment has been minimized.

Copy link

curtd commented Jan 31, 2019

I noticed in my own testing that HTTP.jl uses MbedTLS version 0.6.0 in its Project.toml. Manually updating this to 0.6.7 and rebuilding the package seemed to alleviate the problem.

What do you mean by alleviate? I just tried this: JuliaWeb/HTTP.jl@342d9c2, and after a pkg> build, pkg> test HTTP still fails with the error message in the issue description.

My mistake, updating MbedTLS to v"0.6.7" fixed the issue my package was having with HTTP.request() but the tests themselves are still broken

konkam pushed a commit to konkam/FeynmanKacParticleFilters.jl that referenced this issue Feb 1, 2019

s-baumann added a commit to s-baumann/StochasticIntegrals.jl that referenced this issue Feb 1, 2019

@tkoolen

This comment has been minimized.

Copy link

tkoolen commented Feb 5, 2019

I believe I know what's wrong. The deps.jl file generated by BinaryProvider looks like this:

## This file autogenerated by BinaryProvider.write_deps_file().
## Do not edit.
##
## Include this file within your main top-level source, and call
## `check_deps()` from within your module's `__init__()` method

if isdefined((@static VERSION < v"0.7.0-DEV.484" ? current_module() : @__MODULE__), :Compat)
    import Compat.Libdl
elseif VERSION >= v"0.7.0-DEV.3382"
    import Libdl
end
const libmbedcrypto = joinpath(dirname(@__FILE__), "usr/lib/libmbedcrypto.2.16.0.dylib")
const libmbedtls = joinpath(dirname(@__FILE__), "usr/lib/libmbedtls.12.dylib")
const libmbedx509 = joinpath(dirname(@__FILE__), "usr/lib/libmbedx509.0.dylib")
function check_deps()
    global libmbedcrypto
    if !isfile(libmbedcrypto)
        error("$(libmbedcrypto) does not exist, Please re-run Pkg.build(\"MbedTLS\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libmbedcrypto) in (C_NULL, nothing)
        error("$(libmbedcrypto) cannot be opened, Please re-run Pkg.build(\"MbedTLS\"), and restart Julia.")
    end

    global libmbedtls
    if !isfile(libmbedtls)
        error("$(libmbedtls) does not exist, Please re-run Pkg.build(\"MbedTLS\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libmbedtls) in (C_NULL, nothing)
        error("$(libmbedtls) cannot be opened, Please re-run Pkg.build(\"MbedTLS\"), and restart Julia.")
    end

    global libmbedx509
    if !isfile(libmbedx509)
        error("$(libmbedx509) does not exist, Please re-run Pkg.build(\"MbedTLS\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libmbedx509) in (C_NULL, nothing)
        error("$(libmbedx509) cannot be opened, Please re-run Pkg.build(\"MbedTLS\"), and restart Julia.")
    end

end

So it first opens libmbedcrypto, then libmbedtls, then libmbedx509. However, the mbedtls readme states:

[...] when loading shared libraries using dlopen(), you'll need to load libmbedcrypto first, then libmbedx509, before you can load libmbedtls.

This seems plausible given that mbedtls_x509_crt_verify_restartable is a symbol in libmbedx509.so.

@tkoolen

This comment has been minimized.

Copy link

tkoolen commented Feb 5, 2019

Unfortunately, HTTP.jl tests still fail for me after changing the order locally. Still I think it is a real issue.

@leethargo

This comment has been minimized.

Copy link

leethargo commented Feb 5, 2019

I thought that it was likely a version issue, since the problem did not occur on Travis/Debian, but Travis/Ubuntu. (I did not actually retry Debian recently, it's just that I remember it to work.)

@tkoolen

This comment has been minimized.

Copy link

tkoolen commented Feb 5, 2019

The latest release of MbedTLS upgraded to a new version. Julia ships with a libmbedtls.so that's older, and on my system those are the only libmbedtls.so copies. Also the libmbedx509.so / libmbedtls.so that ship with Julia don't have mbedtls_x509_crt_verify_restartable. So I think it's likely it's somehow using the Julia-shipped libs, but I don't quite see why yet, nor why it would be platform-specific (HTTP.jl tests also pass on OSX) or distribution-specific.

@giordano

This comment has been minimized.

Copy link

giordano commented Feb 6, 2019

This bug doesn't allow me to automatically setup Travis in BinaryBuilder.jl. System used: Arch Linux. As a temporary workaround I downgraded this package to v0.6.6.

@DilumAluthge

This comment has been minimized.

Copy link

DilumAluthge commented Feb 6, 2019

@quinnj

This comment has been minimized.

Copy link
Contributor

quinnj commented Feb 6, 2019

Sorry for the slow response here, but I don't really have much to contribute; I haven't personally had any issues on OSX Julia 1.1, or Linux Julia 1.0/1.1; do we have a reliable reproduction somewhere where we could debug what's going on?

@DilumAluthge

This comment has been minimized.

Copy link

DilumAluthge commented Feb 6, 2019

tinyrock added a commit to tinyrock/hydro.jl that referenced this issue Feb 7, 2019

anowacki added a commit to anowacki/CIJ.jl that referenced this issue Feb 7, 2019

Temporarily disable Coveralls coverage submission
Submitting coverage currently errors on Travis because
of:
JuliaWeb/MbedTLS.jl#193

Skip this until this is fixed.
@tanmaykm

This comment has been minimized.

Copy link
Member

tanmaykm commented Feb 22, 2019

bump! got hit by this, had to downgrade

@visr

This comment has been minimized.

Copy link
Contributor

visr commented Feb 22, 2019

@tanmaykm all that needs to happen is merge and tag a few things. Since I see you are a JuliaWeb member, you may be able to help.

  1. Merge JuliaWeb/MbedTLSBuilder#17
  2. Tag a new MbedTLSBuilder release, wait for Travis to upload all binaries and build.jl script to the release.
  3. Download new build.jl script from MbedTLSBuilder and use in this repo.
  4. Merge f51e810
  5. Tag a new MbedTLS.jl

Or if you want to take a shortcut I guess you can also just merge #194 (both commits) and tag.

@tanmaykm

This comment has been minimized.

Copy link
Member

tanmaykm commented Feb 23, 2019

I still get this error even if I apply these changes to MbedTLS.jl. The PRs referred to here seem to be valid. But the error that I am facing seems to be due to incompatibility between shared libraries shipped with Julia and MbedTLS.jl.

Symbol present in lib shipped with MbedTLS.jl:

~/.julia/dev/MbedTLS/deps/usr/lib$ nm libmbedx509.so | grep mbedtls_x509_crt_verify_restartable
000000000000c5b9 T mbedtls_x509_crt_verify_restartable

But not there in lib shipped with Julia:

/data/Work/julia/binaries/julia-1.1.0/lib/julia$ nm libmbedx509.so | grep mbedtls_x509_crt_verify_restartable

And I get this error from within Pkg in Julia stdlib:

julia> using Pkg;
julia> Pkg.activate(".");
julia> Pkg.test("HTTP");
...
Running WebSockets.jl tests...
/data/Work/julia/binaries/julia-1.1.0/bin/julia: symbol lookup error: /home/tan/.julia/dev/MbedTLS/deps/usr/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable
ERROR: Package HTTP errored during testing
Stacktrace:
 [1] pkgerror(::String, ::Vararg{String,N} where N) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Types.jl:120
 [2] #test#66(::Bool, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1328
 [3] #test at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:0 [inlined]
 [4] #test#44(::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:193
 [5] test at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:178 [inlined]
 [6] #test#43 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:175 [inlined]
 [7] test at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:175 [inlined]
 [8] #test#42 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:174 [inlined]
 [9] test at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:174 [inlined]
 [10] #test#41(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:173
 [11] test(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/API.jl:173
 [12] top-level scope at none:0
@martinholters

This comment has been minimized.

Copy link

martinholters commented Feb 26, 2019

Just to verify: There are systems (Windows and macOS?) which are not affected by this issue? Otherwise we could bump the required Julia version of the latest MbedTLS to 1.2- in METADATA to prevent it from being installed on incompatible Julia versions.

@vtjnash

This comment has been minimized.

Copy link
Contributor

vtjnash commented Mar 5, 2019

bump. This is preventing us from getting information out of Coverage.jl: https://travis-ci.org/JuliaCI/CoverageBase.jl/jobs/501818500

@fingolfin

This comment has been minimized.

Copy link

fingolfin commented Mar 5, 2019

This issue has been open for over a month, and apparently there is no fix in sight (at least looking in from the outside; I'd be happy to be wrong here!)

Since this is breaking tests for dozens (hundreds?) of projects, wouldn't it be better to revert the breaking change for now, make a new release, and then work on fixing the issue properly, but with all the time you need?

ronisbr added a commit to JuliaSpace/ReferenceFrameRotations.jl that referenced this issue Mar 5, 2019

🐛 Pin MbedTLS.jl to version 0.6.6
We must pin MbedTLS.jl for now to v0.6.6 as a workarround to bug:

    JuliaWeb/MbedTLS.jl#193
@ronisbr

This comment has been minimized.

Copy link

ronisbr commented Mar 5, 2019

This issue has been open for over a month, and apparently there is no fix in sight (at least looking in from the outside; I'd be happy to be wrong here!)

Since this is breaking tests for dozens (hundreds?) of projects, wouldn't it be better to revert the breaking change for now, make a new release, and then work on fixing the issue properly, but with all the time you need?

I second that. I had to pin MbedTLS.jl to v0.6.6 to fix the issue for now.

@tkoolen

This comment has been minimized.

Copy link

tkoolen commented Mar 5, 2019

Alright, how about this: explicitly dlopen the libs (in the right order) and change all the ccalls to explicitly use the handles. At least that seems to me to be a straightforward fix.

@quinnj

This comment has been minimized.

Copy link
Contributor

quinnj commented Mar 5, 2019

@fingolfin, @ronisbr, please read the other comments in this thread. There's no consensus that just "reverting" the latest release actually resolves the issue (see @tanmaykm's comment and my own). At this point, I suspect that we're getting a weird interaction with the Julia Base-shipped mbedtls shared libraries conflicting with the MbedTLS.jl-shipped ones and that's causing the issues. I've spent at least 3-4 days trying to track down various root causes, but shared-library loading/symbol resolving is far from my skillset. If someone can show concrete evidence of an mbedtls library version that works on Julia 1.0 and 1.1 without issues, I'm happy to make changes and merge PRs, but so far, AFAIU, we haven't seen that kind of 100% solution.

@tkoolen, similarly, if we can show that your suggestions solve the issue here in all cases, then I'm happy to make changes. There just seems to be a lot of "hey, we should do this!" or "just revert!" comments here without anyone actually going thru the work of seeing if that solves everything.

@vtjnash

This comment has been minimized.

Copy link
Contributor

vtjnash commented Mar 5, 2019

Here's a simple quick reproducer (from inside a Julia v1.0 build):

julia> using Libdl

julia> using HTTP

julia> Libdl.dlopen("libmbedx509.so")
Ptr{Nothing} @0x000055f064544560

julia> HTTP.get("https://google.com")
./julia: symbol lookup error: /home/vtjnash/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable

...and at this point the process is now killed also.

@vtjnash

This comment has been minimized.

Copy link
Contributor

vtjnash commented Mar 5, 2019

Also, for a debugging trick, set export LD_BIND_NOW=1. This'll cause the errors to surface as assertion failures much sooner. Than we can, for example, just do this minimal demo:

julia> using Libdl

julia> Libdl.dlopen("libmbedtls.so")
Ptr{Nothing} @0x000055e22e1c1830

julia> Libdl.dlopen("/home/vtjnash/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so")
ERROR: could not load library "/home/vtjnash/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so"
/home/vtjnash/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable
Stacktrace:
 [1] dlopen(::String, ::UInt32) at /data/vtjnash/julia10/usr/share/julia/stdlib/v1.0/Libdl/src/Libdl.jl:97 (repeats 2 times)
 [2] top-level scope at none:0
@ronisbr

This comment has been minimized.

Copy link

ronisbr commented Mar 5, 2019

This bug was filled on the same day v0.6.7 was released and no one has reported that reverting back to v0.6.6 did not fix the issue. Hence, there is a strong correlation. Of course I cannot know if this fixes the issues for everyone, but it did fix in all repositories I tested so far. The previous version can still have problems but those seem at least more sparse than the current one.

@davidanthoff

This comment has been minimized.

Copy link
Contributor

davidanthoff commented Mar 5, 2019

In particular, I read @tanmaykm's comment as saying that reverting did fix things for him.

@quinnj

This comment has been minimized.

Copy link
Contributor

quinnj commented Mar 5, 2019

@staticfloat and I had a good debugging session today and have a pretty good handle on the root cause (julia-shipped mbedtls binaries interfering w/ MbedTLS.jl binaries). I'll try to get things cleaned up and get a new release out soon.

quinnj added a commit that referenced this issue Mar 6, 2019

Change mbedtls library version to 2.7.9 to ensure compatibility with …
…the mbedtls binaries shipped with julia 1.0/1.1. We also add a test to ensure MbedTLS.jl tests run properly in the presence of the julia-shipped mbedtls libraries loaded. Fixes #193

@quinnj quinnj closed this in #196 Mar 6, 2019

@fingolfin

This comment has been minimized.

Copy link

fingolfin commented Mar 6, 2019

Works for me now in Coverage.jl -- thanks!!!

omus added a commit to JuliaWeb/LibCURLBuilder that referenced this issue Mar 14, 2019

Update CURL, MbedTLS, and Zlib (#12)
* Update CURL, MbedTLS, and Zlib

- CURL: 7.61.0 -> 7.64.0
- MbedTLS: 0.11.0 -> 0.17.0
- Zlib: 1.0.2 -> 1.0.3
- FreeBSD Doesn't seem to want to compile anymore

* Need to use v0.6.6 of MbedTLS for BinaryBuilder

- If we use 0.6.7 then we get this error JuliaWeb/MbedTLS.jl#193

* Add certpath to configure. Add default LDFLAGS

* Pin BinaryBuilder to 0.1.4 instead of using master

* Need to jump to 1.0 for the build

* add Cmment about MbedTLS, don't pin BinaryBuilder

* Remove the --with-ca-bundle flag
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.