Proposal for allowing packages to opt-into import A.B only loading B without loading A.

[KristofferC]

Issue

I have seen it requested in different places independently that package authors want some way to allow users to load only a part (I will from now on call it a fragment) of a package. In addition, some of these don't want the fragment to be part of the default loading of the package at all but have that as an optional (by default not enabled) "feature".

Right now, this is not really possible because for example the syntax import MachineLearning.GPU` is more or less equivalent to:

var"#tmp" = import MachineLearning
const GPU = var"#tmp".GPU

meaning that import MachineLearning.GPU forces you to:

1. import the full MachineLearning package.
2. Require that GPU is a submodule inside MachineLearning.

Proposal.

The proposal is quite simple and can pretty much be described in one sentence:

Allow a package author to opt into making import MachineLearning.GPU load the module GPU in MachineLearning/src/GPU.jl, have that module precompile and not force it to load the full MachineLearning.

The packages loaded by the GPU module will use the logic as for MachineLearning (i.e. the same Project and Manifest).

Implementation

Since this does not have any interaction with how Pkg reads or writes Projects or manifest, there is no code change needed in Pkg itself. The change in code loading is to change the signature

    require(into::Module, module::Symbol)

into something like

    require(into::Module, module::Vector{Symbol})

where for import MachineLearning.GPU the module variable would be [:MachineLearning, :GPU]. The job of code loading is then to resolve MachineLearning -> UUID -> package_path. Look in package_pathfor a Project file and see if that has opted- in to loadingGPUas a fragment. If that is the case, we just load$package_path/src/GPU.jl`. I

Interaction with Requires.jl

Requires.jl exist to automatically run code by
It's implemented in code loading by associating the loading of a package to running a callback. This means that you can add an extra overload to your function when e.g. DataFrames is loaded without the user having to do anything more than that. The fragment-proposal isn't mean to replace such a hook but it does provide a convenient

Currently, you write:

@requires DataFrames include("dataframe_extra_code.jl")

You would know write

@requires DataFrames import MyPkg.DataFramesExtra

which would precompile and load the fragment DataFramesExtra in MyPkg. The advantage to the status quo is that this will be fully precompiled. The second advantage is that you need to say that you depend on DataFrames and provide compat bounds which is not the case with Requires right now.

Interaction with namespace

@StefanKarpinski mentioned some concerns in e.g. #1874 (comment) about how this would interact with the namespace proposal in #1836. At first, I was confused since the proposal there only talks about allowing Pkg to resolve to different package UUIDs by giving some kind of hints when adding the package (for example pkg> add JuliaFinance/Currencies) and doesn't interact with code loading at all. However, I think he considers a "extended" proposal where you can also do import JuliaFinance.Currencies and have that load the Currencies package. To be clear, I don't really see a point to this but these things are composable so you could have import JuliaML.MachineLearning.GPU where JuliaML is a "namespace" (but not a package) and GPU is a fragment. This proposal is what the dot to the right of the package does, and the namespace proposal seems to be about what the thing to the left of the dot does.

[DilumAluthge]

Interaction with Requires.jl

If I understand correctly, @StefanKarpinski is hoping to implement #1285 ("first class support of conditional dependencies") in time for 1.6.

If that comes to fruition, then we can eliminate the "interaction with Requires.jl" section of and instead have an "interaction with Pkg's built-in support for conditional dependencies" section.

So, what would be the relationship between this proposal and the possible upcoming support for conditional dependencies?

[KristofferC]

If I understand correctly, @StefanKarpinski is hoping to implement #1285 ("first class support of conditional dependencies") in time for 1.6.

I guess you know more than me.

[DilumAluthge]

😂 This is just from a short Slack conversation. Nothing set in stone.

[StefanKarpinski]

Requires is kind of a big problem for compilation of code in general. There's so much I still have to finish for 1.6, but I think first class support of conditional dependencies would be a good one to get sorted out for the release.

[KristofferC]

Probably redundant with the new extension feature