Pkg3: package namespaces

[StefanKarpinski]

The current proposal doesn't mention namespaces anywhere. This issue is to discuss how namespaces could fit into the proposal and Julia's own namespacing in general.

[StefanKarpinski]

Some open issues:

• Namespaces could be one-to-one with registries or they could be an orthogonal feature.
• Are namespaces reflected in Julia? I.e. to use JuliaOpt/JuMP do you write using JuliaOpt.JuMP or can you just write using JuMP? If the latter, is there a namespace path?

[ararslan]

My thoughts on this, for however many cents ≤2 they're worth:

Namespaces could be one-to-one with registries, which in turn could work like Homebrew taps. Say there are two registries, JuliaVegetables and JuliaGreens, that each have a package called Broccoli. If a user has opted into the JuliaVegetables registry and not JuliaGreens, they can do using Broccoli and it would implicitly mean using JuliaVegetables.Broccoli. But if the user has opted into both registries, Broccoli would be ambiguous, which would trigger an error. This would be a nice convenience for users, but would be bad for reproducibility.

A safe option may be to require the fully qualified Registry.Package for everything that isn't in the "cathedral" registry (currently METADATA, if I understand that part of the proposal correctly). That would be a non-breaking change, and would also alleviate the problem of registries with identical package names.

[jkroso]

Isn't the package metadata a good place to disambiguate between registries?

So instead of:

[library.libXYZ]
uuid = "994d35e9-862f-42c9-aa51-d40fef54ab41"
versions = "2.3-2.5"

You would write:

[libXYZ](github.com/library/libXYZ)
uuid = "994d35e9-862f-42c9-aa51-d40fef54ab41"
versions = "2.3-2.5"

[bicycle1885]

I believe namespaces are important, so the next-generation packaging system should include namespacing mechanism. It will make it easy for developers/organizers to create "biotopes" to cultivate special-interested packages there. A real-world example is Bioconductor, which includes thousands of R packages, but it has a dedicated package manager because its release schedule is different from R. I think this should not happen in Julia.

Namespaces could be one-to-one with registries or they could be an orthogonal feature.

One-to-one relation with registries sounds reasonable. I want the BioJulia registry and it would be enough.

Are namespaces reflected in Julia? I.e. to use JuliaOpt/JuMP do you write using JuliaOpt.JuMP or can you just write using JuMP? If the latter, is there a namespace path?

Always requiring namespace prefixes looks a little bit verbose to me, but may be consistent.

[StefanKarpinski]

A reason not to make namespaces and registries one-to-one is that we want to allow people to use their GitHub usernames (for example) as namespaces and still publish their packages. But maybe that's too fine-grained a level for namespacing and that use case should instead be handled by improved support for installing unregistered packages.

[tbreloff]

Can we have "implicit registries" based on a github account ("tbreloff" or "JuliaML") or alternate directory/url? I suppose it depends what you require a registry defines, but at a minimum there should be an automated way to initialize a registry for an existing account/directory/url using the repos that exist there.

I think it's fine to be one-to-one as long as registries are dirt-simple to create and manage. Some related ideas: tbreloff/MetaPkg.jl#5

[JeffreySarnoff]

The package manager must offer clean and tidy management behind corporate walls. There may exist contractual constraints or turf-protective limits on interdepartmental access. Distinct versions of a package may not permit the same package manager activities. Package access, use, testing, upgrading, may be subject to legal obligations and the existence of those obligations may be confidential. Permissions and restrictions that may apply to upgrading for direct use may not apply to upgrading that same package happens indirectly in support of another project.

The ability to tell systems "keep [these packages] just as they are" and "keep [these packages] up to date" and "[these packages] should be kept in sync with the latest stable [patch] to [version]" helps. Namespaces improve that help. Checkpoints augment the helpfulness of the help.

As I understand it, Julia's Registrating, Namespacing and Checkpointing are mutually informing and jointly supporting. The availability of one simplifies some parts of the others. They should co-exist.

[samoconnor]

I've recently been working on generating AWS Julia API packages from the Amazon JSON service description files.

My work-in-progress output results in 170 kloc across 102 service interface packages: AmazonAPIGateway.jl ... through ... AmazonXRay.jl.
See: generated documentation, and generated code

Currently there are no types, no input/output validation or translation, just one-line wrapper functions and docstrings for each service operation (i.e. the size of the packages will probably grow).

I'm mentioning this here as a potential use-case for package namespaces.
(Including all these modules in a single AWSSDK.jl package is impractical, the package would be huge, the pre-compile time would be long, and the vast majority of users would use at most ~10% of the modules.)

Will Pkg3 namespaces allow all the 100+ Amazon*.jl packages to be neatly grouped together?
It would be good to have an easy was to install all of them for dev systems where the bloat is not a problem, but to allow lean dependancy driven installation for space constrained deployment environments (cluster nodes, embedded systems, AWSLambda, etc).

[quinnj]

Great thoughts @samoconnor; I've had similar thoughts recently w/ HTTP.jl. Currently, it has code for URI parsing, handling cookies, client-side requests, a server, etc. all built in under the same package. I could split these "subpackages" out into stand-alone packages, but it's actually kind of nice having them all under one roof from a maintenance & development standpoint. It'd be great if they (the submodules/packages) could be registered and used individually, while using HTTP.jl would bring them all in.

[StefanKarpinski]

Do you guys have any thoughts on how this would look and work? I.e. how would one use a namespaced package and what would it mean? Are you imagining using HTTP/Server or using HTTP.Server? Would this essentially just allow levels of hierarchy in package names? Or would the behavior be something more than that? If there's a package named HTTP/Server do you always use it as HTTP.Server or can it sometimes be accessed as just Server as in using HTTP; using Server?

[JeffreySarnoff]

If there is a package named HTTP/Server imo it should only be used as HTTP.Server because if it were accessed as Server that opens up the potential for conflict with e.g. Task/Server.

[StefanKarpinski]

Ok, it seems like this may just be a layer of hierarchy in package names. This usage suggests that the Julia-side syntax probably be HTTP.Server since HTTP/Server already has a meaning, although I kind of like writing using HTTP/Server instead of HTTP.Server so that the / indicates that it's a hierarchical package name, not a submodule – although it should probably be exposed as a submodule on the Julia side, so 🤷‍♂️ . Would it be too weird to write using HTTP/Server but then access the package module as HTTP.Server or just Server? Would the HTTP module be made available after doing using HTTP/Server as well? There could also be more levels of hierarchy, e.g. using Net.HTTP.Server. Would that be possible to abbreviate as using HTTP.Server?

[quinnj]

Here's a few of my thoughts:

• I don't have a strong preference on using HTTP/Server vs. using HTTP.Server
• I do think it's important to allow using HTTP.Server to only load the Server code. A better example would be HTTP.URIs, where there are several packages that only need HTTP.URIs module functionality, and don't want to have to depend on all of HTTP
• In reality, I'd be fine registering HTTP.URIs as it's own package, including all the required package files in the HTTP/src/URIs/ directory.

So just clarify, I would imagine my package directory looking like:

HTTP/
          URIs/
                    REQUIRE
                    src/URIs.jl
                    appveyor.yml
                    etc...
          Server/
                    REQUIRE
                    src/Server.jl
                    appveyor.yml
                    etc....

So it'd be a way to have my package live inside the directory of another package, so if someone depends on HTTP, they get the whole kitchen sink, but if a package only needs URIs code, they just depend on HTTP.URIs (or it can just be called URIs) and just get that code (w/ maybe the HTTP directory stubbed out at least).