KeyError on internal use of extensions

[MilesCranmer]

As discussed in #49990, one is able to use extensions internally in a package to lazily load a module, without the user needing to manually load code. This requires the backports in #49680 to work - in particular the fix #49701.

Here is an example of how you could use this. Say you want to use Zygote.jl in some rarely-used branch of your code, but you don't want the user to have to load Zygote.jl themselves. You can utilize extensions as follows in src/MyPackage.jl:

module MyPackage

generate_diff_operators(args...) = error("`Zygote` not loaded.")

function create_diffs(binary_operators, unary_operators)
    Base.require(@__MODULE__, :Zygote)
    Base.invokelatest(generate_diff_operators, binary_operators, unary_operators)
end

end

then, in ext/MyPackageZygoteExt.jl, you can define the specialized behavior for generate_diff_operators:

module MyPackageZygoteExt

import Zygote: gradient
import MyPackage: generate_diff_operators

function generate_diff_operators(
    binary_operators::Vector{Function}, unary_operators::Vector{Function}
)
    ...
end

end

This should in principle work. But it seems like the extension triggering code does not check both [deps] and [weakdeps] for the dependency; it only expects it to be in [weakdeps] when used in an extension.

Here is what you see if your dependency is defined in [deps] only:

julia> using MyPackage
ERROR: KeyError: key "Zygote" not found
Stacktrace:
 [1] getindex
   @ ./dict.jl:484 [inlined]
 [2] _insert_extension_triggers(parent::Base.PkgId, extensions::Dict{String, Any}, weakdeps::Dict{String, Any})
   @ Base ./loading.jl:1236
 [3] insert_extension_triggers(env::String, pkg::Base.PkgId)
   @ Base ./loading.jl:1179
 [4] insert_extension_triggers
   @ ./loading.jl:1164 [inlined]
 [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1665
 [6] macro expansion
   @ ./loading.jl:1648 [inlined]
 [7] macro expansion
   @ ./lock.jl:267 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1611

If you put the dependency in both [deps] and [weakdeps], this error goes away.

However, a later error will appear in a downstream package which depends on MyPackage:

ERROR: LoadError: ArgumentError: Package MyPackage does not have Zygote in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have MyPackage checked out for development and have
  added Zygote as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with MyPackage
Stacktrace:
  [1] macro expansion
    @ ./loading.jl:1599 [inlined]
  [2] macro expansion
    @ ./lock.jl:267 [inlined]
  [3] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1576

So it seems like a different check is seeing that Zygote.jl appears in [weakdeps], and is raising an error because of this.

---

cc @KristofferC

[MilesCranmer]

Ping on this. Let me know if I can help at all.

[MilesCranmer]

Ping on this

[brenhinkeller]

Looks like we need a new issue label for extensions / weakdeps. Who can make new labels?

[DilumAluthge]

I created a label with a rough name. We should definitely change the name.

Also, if we decide later that we don't need this label, we can fold it into another label.