[RFC] Packages in subdirectories need more registry information

[GunnarFarneback]

As discussed in JuliaLang/Pkg.jl#1251, having registered packages in a subdirectory of a git repository works partially today. If you prepare a registry with the right tree hashes, add Package and add Package@revision work out of the box. Additional Pkg functionality is added in the WIP PR JuliaLang/Pkg.jl#1422.

However, for add Package#branch or dev Package there is no information available in the registry about the subdirectory. Thus either Pkg would have to hunt through the repository for a matching Project.toml or the user would have to provide the subdirectory path for these operations. Neither of these options seems very satisfactory.

My proposal is that Package.toml is modified to include this information. Two possibilities are to change (in a hypothetical world where RegistryTools lives in a subdirectory) today's

name = "RegistryTools"
uuid = "d1eb7eb1-105f-429d-abf5-b0f65cb9e2c4"
repo = "https://github.com/JuliaRegistries/RegistryTools.jl.git"

into either

name = "RegistryTools"
uuid = "d1eb7eb1-105f-429d-abf5-b0f65cb9e2c4"
repo = "https://github.com/JuliaRegistries/RegistryTools.jl.git"
path = "path/to/package"

or

name = "RegistryTools"
uuid = "d1eb7eb1-105f-429d-abf5-b0f65cb9e2c4"
repo = "https://github.com/JuliaRegistries/RegistryTools.jl.git?path=path/to/package"

Both of these would be backwards compatible with current Julia versions since no changes are required for normal packages. For packages in subdirectories they differ in how they don't work properly beyond add Package and add Package@version. In the former case, as well as without path information in the registry, add Package#branch and dev Package downloads/clones the top level repository and end up with a broken package. In the latter case these operations instead result in an immediate, though not very clear, error.

Note that packages in subdirectories is a useful enough feature that it would be worth using even without improved Pkg support. It would be good to settle this issue right away though, since adding path information to existing registries with such packages would be an unnecessary complication.

Opinions?

[StefanKarpinski]

I prefer option 2 with the path entry. However, I don't think you technically need this: you can search the cloned repo for a project file with the correct UUID and that tells you where it is.

[GunnarFarneback]

No, it is not technically necessary, but as I wrote:

Thus either Pkg would have to hunt through the repository for a matching Project.toml or the user would have to provide the subdirectory path for these operations. Neither of these options seems very satisfactory.

[StefanKarpinski]

Fair enough. So option 2 with explicit path = entry seems good.

[StefanKarpinski]

Although note that it's not necessary when just installing a package because the package version is associated with a git tree, which need not be at the root of the repo (which you also noted).

[fredrikekre]

Pkg would have to hunt through the repository for a matching Project.toml

What's wrong with this? it will usually be just one or two directories deep.

[GunnarFarneback]

It's not unrealistic that people place dozens of packages in one repository and you would have to parse lots of project files, or you are unlucky and add a branch where your project file has been removed, which you won't know until you have searched through the entire repository.

But pathological cases aside, it's really about code complexity. If I were to implement it I'd much prefer to look up the path in the registry and go straight at the project file than having to traverse the file system and see if something matches.

[KristofferC]

FWIW; I think a subdir entry makes sense (let's call it that because that is what Pkg calls it).

[KristofferC]

Another thing to discuss is how you register/tag a subdir package. Something like @JuliaRegistrator register subdir=... maybe?

[fingolfin]

I'd second that subdir entry, to match JuliaLang/Pkg.jl#1422 (makes it easier to understand how these things fit together, IMHO). Also note that one use case I have in mind would end up with a repository with > 100 subpackages in it. All nested only one level deep, though.

As a use, I'd also be happy with the @JuliaRegistrator register subdir=... syntax, as also discussed on JuliaRegistries/Registrator.jl#230.

This syntax would then also be amenable to generalization: @JuliaRegistrator register subdir=* to scan all immediate subdirs, or perhaps @JuliaRegistrator register subdir=*/* to scan all subdirs two levels down. Of course not necessary for an initial implementation.

[StefanKarpinski]

How about registering any directory whose project file has been changed? Although that leaves questions: changed in the last commit or at any point (since when)? Changed in any way or just the version number? So perhaps being explicit about it is better. However, the way that I usually do releases is to make a commit directly on master changing only the version number in the project file and then triggering registrator on that. If others do the same, then taking what directories to register from the triggering commit and including those with project files with changed version numbers would be pretty convenient.

[GunnarFarneback]

For what it's worth, LocalRegistry will do registration of packages in subdirectories exactly like normal packages. In an environment where you have deved MyPackage (dev by path already works fine with subdirectory packages) you register it with register("MyPackage"). The code locates the Project.toml, uses git rev-parse --show-prefix to determine the subdir and proceeds from there.