Breaking change in Julia 1.11 w.r.t. MD5.jl

[bensetterholm]

It seems that there was a breaking change introduced somewhere between Julia 1.11.0-DEV.550 (working) and 1.11.0-DEV.555 (failing) which caused my code depending on the MD5.jl library to fail (and is still failing under today's nightly, namely 1.11.0-DEV.643). Since neither the MD5.jl codebase nor SHA.jl (which the former depends on) have been touched in a while, the issue seems to be coming from a change in Julia itself.

MWE:

julia> using MD5

julia> md5("test")
ERROR: type MD5_CTX has no field used
Stacktrace:
 [1] getproperty
   @ Base ./Base.jl:37 [inlined]
 [2] update!(context::MD5.MD5_CTX, data::Base.CodeUnits{UInt8, String}, datalen::Int64)
   @ SHA ~/julia-8a847d520c/share/julia/stdlib/v1.11/SHA/src/common.jl:20
 [3] update!(context::MD5.MD5_CTX, data::Base.CodeUnits{UInt8, String})
   @ SHA ~/julia-8a847d520c/share/julia/stdlib/v1.11/SHA/src/common.jl:20
 [4] md5(data::Base.CodeUnits{UInt8, String})
   @ MD5 ~/.julia/packages/MD5/H5Cdd/src/MD5.jl:18
 [5] md5(str::String)
   @ MD5 ~/.julia/packages/MD5/H5Cdd/src/MD5.jl:24
 [6] top-level scope
   @ REPL[1]:1

julia> versioninfo()
Julia Version 1.11.0-DEV.643
Commit 8a847d520c2 (2023-10-12 15:02 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 24 × Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz
  WORD_SIZE: 64
  LLVM: libLLVM-15.0.7 (ORCJIT, icelake-server)
  Threads: 1 on 24 virtual cores
Environment:
  JULIA_PKG_USE_CLI_GIT = true

The above example was run under Red Hat 8.8; I get the same error on a different machine (macOS) as well.

Original bug report (where it was recommended that I file the bug here instead):

• MD5 stopped working for Julia 1.11 JuliaCrypto/MD5.jl#11

[vchuravy]

@jakobnissen this seems to be caused by JuliaCrypto/SHA.jl#86 picked up in Julia by #51049

[staticfloat]

The issue is that MD5 is using SHA's update!() algorithm with its own type. The MD5 package needs to add a used field to its context object.

I think the right thing to do here is to:

• Bump the major version number of SHA.jl, to denote a compat change in the package.
• Fix the version bounds of MD5.jl to disallow instantiation of older projects with a newer SHA.jl
• Make a new release of MD5.jl to add a used field to its context.

[bensetterholm]

@vchuravy - Sorry if I am just adding noise at this point... If #51049 was added to nightly in late August, why did my unit GitHub CI unit tests, based on the nightly build, still work until Sept 25 (and only start failing on Sept 26)?

[jakobnissen]

Of course the first priority is to fix the breakage - but surely, the real solution is for MD5 to not use internals of SHA?
MD5 subtypes an undocumented, unexported abstract type of SHA, and then re-uses functions defined in SHA for that abstract type. That's a recipe for breakage.
Normally, I'd release a new patch version for MD5 that fixed the compat for SHA. However, since these are stdlibs and there is no real versioning (yet), I'm not sure what to do here.

[jakobnissen]

This will be closed by JuliaCrypto/MD5.jl#12 .
IMO this is not a Julia issue, as the external package MD5 was using internals that were subject to change. I'm therefore closing.

[oxinabox]

For historical note: MD5.jl was written using those internals when both it and SHA.jl were external libraries with few users.
Including changes to SHA.jl specifically for supporting MD5.jl: JuliaCrypto/SHA.jl#49
Probably what should have happened then is we should have documented those internal abstractions as part of the public API.
Or we should have stuck MD5 into SHA.jl