switching between pkg servers with different registries

[johnnychen94]

In summary, I believe we can improve Pkg's behavior on two small issues:

• case 2: support installing all available registries of used pkg server. -- I believe this was a bug.
• case 4: when the pkg server is used, optionally disable the git operation for packages not available in registries served by the pkg server. -- This provides a better user experience (clearer error messages).

Before I start to put my hands on this, I'd like to know if you think the same.

---

I don't know if there's a public pkg server that serves multiple registries, thus I use the private pkg server we use in TongYuan to provide the example.

❯ curl -L https://releases-dev.tongyuan.cc/registries
/registry/fe1a5c12-5b3e-461d-9252-adc475b18e8c/c159bd9903151fcba081a0dfe18228cbf27108cd
/registry/23338594-aafe-5451-b93e-139f81909106/dea1026b1d1e51f165a6e5c95fbf4fcf038f5feb

Edit: we now have a public pkg server to test: https://releases.tongyuan.cc/juliapkg

The 23338594 is the public "General" registry, and fe1a5c12 is the private "Syslab" registry.

Case 1: a new depot with no registry installed

When Pkg.status() or Pkg.add() get called, it installs both registries recorded in the pkg server.

❯ export JULIA_PKG_SERVER="https://releases-dev.tongyuan.cc"

❯ JULIA_DEPOT_PATH=$PWD/tmp julia -e 'using Pkg; Pkg.status(); Pkg.Registry.status()'
  Installing known registries into `~/tmp`
Status `~/tmp/environments/v1.9/Project.toml` (empty project)
Registry Status
 [23338594] General (https://github.com/JuliaRegistries/General.git)
 [fe1a5c12] Syslab (https://git.tongyuan.cc/syslab-release/stream/SyslabRegistry.git)

Case 2: an existing depot with General installed

This case often happens when one needs to switch from the default public pkg server to the private pkg server.

In this case, extra registries served by pkg server are ignored when General is installed.

First we set the default depot up using default pkg server:

❯ rm -rf tmp

❯ export JULIA_PKG_SERVER="https://pkg.julialang.org"

❯ JULIA_DEPOT_PATH=$PWD/tmp julia -e 'using Pkg; Pkg.status(); Pkg.Registry.status()'
  Installing known registries into `~/tmp`
Status `~/tmp/environments/v1.9/Project.toml` (empty project)
Registry Status
 [23338594] General (https://github.com/JuliaRegistries/General.git)

then we switch to the private pkg server:

❯ export JULIA_PKG_SERVER="https://releases-dev.tongyuan.cc"

❯ JULIA_DEPOT_PATH=$PWD/tmp julia -e 'using Pkg; Pkg.status(); Pkg.Registry.status()'
Status `~/tmp/environments/v1.9/Project.toml` (empty project)
Registry Status
 [23338594] General (https://github.com/JuliaRegistries/General.git)

In this case, Pkg doesn't install the extra Syslab registry recorded in the pkg server. That would make pkg> add <some_private_package> fail.

Case 3: switch back to the public pkg server

With a clean depot, let's add a private pkg first, then switch back to the public pkg server. The private registry doesn't get updated, and the private package doesn't get updated (which is great).

❯ rm -rf tmp

❯ export JULIA_PKG_SERVER="https://releases-dev.tongyuan.cc"

❯ JULIA_DEPOT_PATH=$PWD/tmp julia -e 'using Pkg; Pkg.add("TyExample")'
  Installing known registries into `~/tmp`
    Updating registry at `~/tmp/registries/General.toml`
    Updating registry at `~/tmp/registries/Syslab.toml`
   Resolving package versions...
   Installed TyExample ─ v0.2.0
   Installed TyAPI ───── v0.3.8
    Updating `~/tmp/environments/v1.9/Project.toml`
  [7876af07] + TyExample v0.2.0
    Updating `~/tmp/environments/v1.9/Manifest.toml`
  [47291b8a] + TyAPI v0.3.8
  [7876af07] + TyExample v0.2.0
Precompiling project...
  2 dependencies successfully precompiled in 1 seconds

❯ export JULIA_PKG_SERVER="https://pkg.julialang.org"

❯ JULIA_DEPOT_PATH=$PWD/tmp julia -e 'using Pkg; Pkg.update()'
    Updating registry at `~/tmp/registries/General.toml`
    Updating registry at `~/tmp/registries/Syslab.toml`
  No Changes to `~/tmp/environments/v1.9/Project.toml`
  No Changes to `~/tmp/environments/v1.9/Manifest.toml`
[ Info: We haven't cleaned this depot up for a bit, running Pkg.gc()...
      Active manifest files: 1 found
      Active artifact files: 0 found
      Active scratchspaces: 0 found
     Deleted no artifacts, repos, packages or scratchspaces

Case 4: switch back to the public pkg server with compat conflict

TyGaloisFieldNumbers is a private version of GaloisFieldNumbers. There are two things to make it special here:

• TyGaloisFieldNumbers v0.1.4 requires Polynomials="3 - 3.2.8"
• TyGaloisFieldNumbers v0.1.5 requires Polynomials="3"

To create a case that has compat conflict, I first installed TyGaloisFieldNumbers v0.1.4 with the private pkg server, and then switched to install Polynomials v3.2.12 with the public pkg server. Then Julia will try to fallback to git clone the package and give "encrypted" error messages.

❯ rm -rf tmp

❯ export JULIA_PKG_SERVER="https://releases-dev.tongyuan.cc"

❯ JULIA_DEPOT_PATH=$PWD/tmp julia -e 'using Pkg; pkg"add TyGaloisFieldNumbers@0.1.4"'
┌ Warning: The Pkg REPL mode is intended for interactive use only, and should not be used from scripts. It is recommended to use the functional API instead.
└ @ Pkg.REPLMode /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Pkg/src/REPLMode/REPLMode.jl:382
  Installing known registries into `~/tmp`
    Updating registry at `~/tmp/registries/General.toml`
    Updating registry at `~/tmp/registries/Syslab.toml`
   Resolving package versions...
   Installed TyCEI ──────────────── v0.3.6
   Installed Polynomials ────────── v3.2.8
   Installed TyGaloisFieldNumbers ─ v0.1.4
...

❯ JULIA_DEPOT_PATH=$PWD/tmp julia -e 'using Pkg; pkg"add Polynomials@3.2.12"'
┌ Warning: The Pkg REPL mode is intended for interactive use only, and should not be used from scripts. It is recommended to use the functional API instead.
└ @ Pkg.REPLMode /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Pkg/src/REPLMode/REPLMode.jl:382
    Updating registry at `~/tmp/registries/General.toml`
    Updating registry at `~/tmp/registries/Syslab.toml`
   Resolving package versions...
   Installed Polynomials ────────── v3.2.12
     Cloning [0b0ef4ed-0311-459f-9fa8-24d44fa913b9] TyGaloisFieldNumbers from ssh://git@git.tongyuan.cc:222/syslab-msp/packages/TyGaloisFieldNumbers.jl.git
    Updating git-repo `ssh://git@git.tongyuan.cc:222/syslab-msp/packages/TyGaloisFieldNumbers.jl.git` <---- trying to add TyGaloisFieldNumbers v0.1.5
ERROR: TyGaloisFieldNumbers: git object 90db8b2ca303faf7d3cd06f805205c909e5cce1e could not be found
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] install_git(io::Base.TTY, uuid::Base.UUID, name::String, hash::Base.SHA1, urls::Set{String}, version_path::String)
    @ Pkg.Operations /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:694
  [3] download_source(ctx::Pkg.Types.Context; readonly::Bool)
    @ Pkg.Operations /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:905
  [4] download_source
    @ /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:791 [inlined]
...