An incompatible version of MbedTLS is installed at wrong version_slug for Julia 1.0.5 on Travis

[tkf]

Following MWE shows some strange behavior:

• Pkg says Installed MbedTLS ──────── v0.6.8 but using MbedTLS tries to load MbedTLS 1.0.2.
• julia uses X4xar as the version_slug which is correct one for MbedTLS v0.6.8. However, the actual directory contains MbedTLS 1.0.2.

$ julia -e 'using Pkg; pkg"add MbedTLS"'
   Cloning default registries into /home/travis/.julia/registries
   Cloning registry General from "https://github.com/JuliaRegistries/General.git"
 Resolving package versions...
 Installed BinaryProvider ─ v0.5.10
 Installed MbedTLS ──────── v0.6.8
  Updating `~/build/tkf/MyPlayground.jl/Project.toml`
  [739be429] + MbedTLS v0.6.8
  Updating `~/build/tkf/MyPlayground.jl/Manifest.toml`
  [b99e7846] + BinaryProvider v0.5.10
  [739be429] + MbedTLS v0.6.8
  [2a0f44e3] + Base64 
  [ade2ca70] + Dates 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [8f399da3] + Libdl 
  [37e2e46d] + LinearAlgebra 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [de0858da] + Printf 
  [9a3f8284] + Random 
  [ea8e919c] + SHA 
  [9e88b42a] + Serialization 
  [6462fe0b] + Sockets 
  [8dfed614] + Test 
  [4ec0a83e] + Unicode 
The command "julia -e 'using Pkg; pkg"add MbedTLS"'" exited with 0.

$ head /home/travis/.julia/packages/MbedTLS/X4xar/src/MbedTLS.jl
module MbedTLS
using Random, Sockets, MbedTLS_jll
import Sockets: TCPSocket
export
# Message digests
    MD_NONE,
    MD_MD2,
    MD_MD4,
The command "head /home/travis/.julia/packages/MbedTLS/X4xar/src/MbedTLS.jl" exited with 0.

$ cat /home/travis/.julia/packages/MbedTLS/X4xar/REQUIRE
cat: /home/travis/.julia/packages/MbedTLS/X4xar/REQUIRE: No such file or directory
The command "cat /home/travis/.julia/packages/MbedTLS/X4xar/REQUIRE" exited with 1.

$ cat /home/travis/.julia/packages/MbedTLS/X4xar/Project.toml
name = "MbedTLS"
uuid = "739be429-bea8-5141-9913-cc70e7f3736d"
authors = ["Jacob Quinn <quinn.jacobd@gmail.com>"]
version = "1.0.2"
[deps]
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
MbedTLS_jll = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
Sockets = "6462fe0b-24de-5631-8697-dd941f90decc"
[compat]
julia = "1.3"
MbedTLS_jll = "2"
[extras]
Distributed = "8ba89e20-285c-5b6f-9357-94700520ee1b"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
[targets]
test = ["Test", "Distributed"]
The command "cat /home/travis/.julia/packages/MbedTLS/X4xar/Project.toml" exited with 0.

$ julia -e 'using MbedTLS'
ERROR: LoadError: ArgumentError: Package MbedTLS does not have MbedTLS_jll in its dependencies:
- If you have MbedTLS checked out for development and have
  added MbedTLS_jll as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with MbedTLS
Stacktrace:
 [1] require(::Module, ::Symbol) at ./loading.jl:836
 [2] include at ./boot.jl:317 [inlined]
 [3] include_relative(::Module, ::String) at ./loading.jl:1044
 [4] include(::Module, ::String) at ./sysimg.jl:29
 [5] top-level scope at none:2
 [6] eval at ./boot.jl:319 [inlined]
 [7] eval(::Expr) at ./client.jl:393
 [8] top-level scope at ./none:3
in expression starting at /home/travis/.julia/packages/MbedTLS/X4xar/src/MbedTLS.jl:3
ERROR: Failed to precompile MbedTLS [739be429-bea8-5141-9913-cc70e7f3736d] to /home/travis/.julia/compiled/v1.0/MbedTLS/bf9T0.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1203
 [3] _require(::Base.PkgId) at ./loading.jl:960
 [4] require(::Base.PkgId) at ./loading.jl:858
 [5] require(::Module, ::Symbol) at ./loading.jl:853

https://travis-ci.org/github/tkf/MyPlayground.jl/builds/714907915#L214
https://github.com/tkf/MyPlayground.jl/blob/8483fda3a60371f897fbefec3f6ed473d863c089/.travis.yml

Note that X4xar is the correct version_slug of MbedTLS v0.6.8.

julia> Base.version_slug(Base.UUID("739be429-bea8-5141-9913-cc70e7f3736d"), Base.SHA1("2d94286a9c2f52c63a16146bb86fd6cdfbf677c6"))
"X4xar"

where 2d94286a9c2f52c63a16146bb86fd6cdfbf677c6 is the tree sha of b71d85de30afb1b4638d1169fa1f6432094a665a (v0.6.8) JuliaLang/MbedTLS.jl@b71d85d.

(Originally reported in JuliaLang/MbedTLS.jl#224)

[KristofferC]

Downloading https://api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/2d94286a9c2f52c63a16146bb86fd6cdfbf677c6 which is the tree-sha for v0.6.8 seems to give the 1.0.2 tarball...

[KristofferC]

Wait, it seems to redirect to https://codeload.github.com/JuliaLang/MbedTLS.jl/legacy.tar.gz/master...

[KristofferC]

Eeh, is this the case for all our API tarball downloads? That's a bit worrying.

[staticfloat]

I can confirm that the https://api.github.com/repos/<org>/<repo>/tarball/<treehash>.tar.gz API endpoint seems to be redirecting to the latest master via the URL https://codeload.github.com/<org>/<repo>/legacy.tar.gz/master. Neither tree hashes nor commit hashes work; everything seems to redirect to the same tarball.

I guess we should have been verifying treehashes this whole time. ;)

[StefanKarpinski]

This used to work, so GitHub broke this and this is really screwing us over badly: the URL rewrite is baked into the client and cannot be fixed after the fact. Another check mark in "don't rely on some 3rd party service that can go away or break their own API" column. Thank goodness we're switching to our own protocol and servers and away from downloading from GitHub.

[KristofferC]

Is there any info about this API being deprecated?

[tkf]

It looks like it's still documented here? https://docs.github.com/en/rest/reference/repos#download-a-repository-archive

[StefanKarpinski]

I've tweeted about it: https://twitter.com/home. Does GitHub have a place to file issues these days?

[Nosferican]

So... Doing https://api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/b71d85de30afb1b4638d1169fa1f6432094a665a
which is the sha1 of the commit for v0.6.8 works. Per the API docs, when the Git ref is not valid it defaults to giving the latest. In other words, it should be using the sha1 for the commit rather than the tree-sha apparently. Refs may be branch, SHA, tag. So our API call should probably be: https://api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/v0.6.8

[tkf]

https://codeload.github.com/JuliaLang/MbedTLS.jl/legacy.tar.gz/2d94286a9c2f52c63a16146bb86fd6cdfbf677c6 seems to work. So it's only the redirection part that is not working...

[StefanKarpinski]

It's been worked around in a very hacky way.

[tkf]

Wow, that's a clever hack. Thanks a lot for the quick fix!

It's working now https://travis-ci.org/github/tkf/MyPlayground.jl/jobs/714938750#L191