Feature Request: multi-package .git repositories

[ma-laforge]

I am creating this feature request in response to @KristofferC 's comment: #1874 (comment).

Overview

The following describes a workflow I would like to achieve using Julia's Pkg/Registration/CI systems. The intent is to reduce the .git repository clutter, while maintaining the fine-grained package hierarchy we have now. Oh, did I mention I want to do this while simultaneously maintaining the minimal dependency requirements of our "base" packages (i.e. RecipesBase.jl for Plots.jl, or SnoopCompileCore.jl for SnoopCompile.jl).

Ideally, this feature should provide the facilities necessary to make package development, registration, and testing relatively easy.

Multiple packages per repository

To avoid repo-clutter, a Julia repository should be able to store multiple (typically related) packages:

PkgRepo
├── somepkgdir
│   ├── PkgA
│   │   └── Project.toml
│   ├── PkgB
│   │   └── Project.toml
│   └── PkgC
│       └── Project.toml
├── RepoManifest.toml
└── RepoProject.toml

Locating packages

Unless we intend on freely moving packages across .git repositories, UUIDs only really need to be assigned to the repositories themselves. The same goes for the repository version number:

RepoProject.toml

name = "PkgRepo"
uuid = "aa65fe97-06da-5843-b5b1-d5d13cad87d2"
version = "1.7.1"

[packages]
PkgA = "somepkgdir/PkgA"
PkgB = "somepkgdir/PkgB"
PkgC = "somepkgdir/PkgC"

Note that we can place packages anywhere (somepkgdir) to accommodate multi-use repositories (not only pure-julia repos). See mention of "Arrow repository" in #1874 (comment).

As mentioned above, individual packages don't need a UUID - only the repo. The (full) UUID of a package, say PkgA is therefore:

{PkgRepo_UUID}!PkgA #Or something similar

So, using this system, your code can import packages from multi-package repo with the following:

import PkgRepo!PkgA

Note: I decided to introduce ! because (:, .) are already used in import statements, and @ is a bit confusing (PkgRepo@PkgA or PkgA@PkgRepo??)

Registration at the repository level

To simplify the registration process for developers (but not necessarily developers of Pkg.jl, Registrator.jl, ...), only package repositories should be registered directly.

When a new version is registered with JuliaRegistrator, PkgRepo.RepoProject.toml is parsed, then all packages in that repo get registered simultaneously. I guess this will be somewhat taxing on the dependency tree parser because it will have to be run once for each project in that repository.

The advantage of registering julia package repositories instead of individual packages is that developers can more easily add/remove packages as they split up/combine their solution. No need to re-register each individual package, and no need to worry about potential naming collisions because packages are imported through the repository name (PkgRepo!PkgA).

The limitation in this scheme is that versioning is done at the repository level, not the package level (which is kindof what we had at the beginning anyways). I think this is great! It will be much simpler for developers to function with this mindset anyways!

Package development

To keep things simple, I think it is best we triggered the dev command at the repository level, and not the package level:

This is the most natural solution because really, as soon as you clone the repository, you are able to make changes to all packages anyhow. It is also very natural because when you add a feature to your package collection, chances are you will need to add code across multiple, interdependent packages. Let's also not forget that the point of this feature request is so we can collect related packages together in a single repository.

Unit testing

Again, I think the best idea is to keep unit testing at the repository level. Of course, from the point of view of the developer, testing can be broken down into individual packages. That's quite fine and logical. Nonetheless, CI testing gets triggered when we push our changes to Github. Moreover, a single commit in PkgRepo!PkgA might require PkgRepo!PkgB to be re-tested if it depends on PkgRepo!PkgA. It therefore makes sense that CI be configured to perform package testing for the entire repo irrespective of the changes being made.

[KristofferC]

Unless we intend on freely moving packages across .git repositories, UUIDs only really need to be assigned to the repositories themselves.

Disagree with this. Each package should have a unique UUID. Moving packages between .git repositories should be possible. This is assuming we are talking about actual packages and not "individually loadable submodules".

On the surface, this looks pretty similar to my proposal in #1874 (comment). For example, the proposal here suggests a new way of importing packages import PkgRepo!PkgA while my suggestion used import PkgRepo@PkgA.

However, this proposal suggests a bunch of additions to the project file which is not needed with my proposal which could be done with no changes to Pkg and small changes to the code loading in Julia. What is the advantage here over the one I posted?

[ma-laforge]

Disagree with this. Each package should have a unique UUID. Moving packages between .git repositories must be possible. This is assuming we are talking about actual packages and not "individually loadable submodules".

Unique UUID: Not sure this is true - and also, I have to take back what I said as well. You could still move your package between .git repositories. If we import modules with the PkgRepo!PkgA model, movement between packages is still possible.

What changes is that the calling program would have to change from: import OriginalPkgRepo!PkgA to import NewPkgRepo!PkgA when it decides it requires to upgrade to the new version of PkgA. Of course, it would also have to add NewPkgRepo (and its UUID) to its Project.toml file when it chooses to do so. I don't think this is an unreasonable thing to do.

On the surface, this looks pretty similar to my proposal in #1874 (comment). For example, the proposal here suggests a new way of importing packages import PkgRepo!PkgA while my suggestion used import PkgRepo@PkgA.

Indeed. I very much liked that part of your proposal. I hope you don't feel like misappropriated your idea. But as I implied, I find import PkgRepo@PkgA to be a bit confusing because it sort of sounds like you want to use package PkgRepo which is located "at" repository PkgA, instead of the inverse.

However, this proposal suggests a bunch of additions to the project file which is not needed with my proposal which could be done with no changes to Pkg and small changes to the code loading in Julia. What is the advantage here over the one I posted?

1. Developers don't need to add "version" fields to their Project.toml files and try to ensure they all match within that repository.
2. and developers don't need to run special registration functions that will ensure all package version numbers match, to avoid doing this completely manually.
3. dev PkgRepo always affects all packages simultaneously, and there is no possibility that individual packages in that repository get out of sync from the rest. I think that is much easier way for developers to think.
4. The barrier to adding/removing projects from a repository as your multi-package solution evolves is severely lowered because the registration tool can infer the list of supported projects from the RepoProject.toml file.
5. The risk of naming collision is lowered significantly because project names are always prefixed with the repository name. In other words both SnoopCompile.jl and Plots.jl could supply their own "Core" Package: import SnoopCompile@Core, import Plots@Core. This is not true in the current scheme, because packages are currently only registered in a global fashion (not relative to the parent repository).

[KristofferC]

1-5 are all true for my proposal as well though?

[ma-laforge]

Well if so, I'm all for it!

The impression I got when I read your proposal was that these things were not possible. If I am mistaken, then I am sorry.

I would be very happy to use your solution if it does this.

Add/remove projects

I am curious though: how can you easily add/remove projects from a repository with your solution?

I get the impression you have to register each one manually with JuliaRegistries to have this happen. Also, from my understanding, once something is registered, it can't be un-registered. I'm ok with that for registering package repositories, but I am less fond of this when it comes to individual projects of those repositories.

Lowered risk of naming collisions

I also don't quite get how you can release "Core" packages for two different repositories (ex: SnoopCompile@Core, Plots@Core) if packages are registered in a global fashion. I must not have understood your proposal, then.