Question

[Cody-G]

I thought I could temporarily work on ImagineInterface.jl without adding it to this registry, but I've noticed that cloning private repos in the build script doesn't work with Julia 0.7+. build ImagineInterface completes without error, but the cloned packages aren't accessible when I say using DependentPkg.

Is this because registries are now the only option for private dependencies? If so I can add ImagineInterface to HolyLabRegistry. I have a couple of questions about that process, but before I ask them I want to make sure I understand whether registries like this are now required in our workflow.

[timholy]

It's because the new Pkg is stingy with names as part of its solution of the "dependency hell" problem: https://julialang.github.io/Pkg.jl/v1/#Background-and-Design-1. You might have a local package named Mine, and you might install Yours that depends on a different package that also happens to be called Mine. As a consequence, adding Yours to your environment doesn't automatically add Mine; indeed if you already have your own Mine installed there will be no conflict between them: when you say using Mine you'll be using your version, but Yours will internally use the other Mine. And never the twain shall meet.

You can fix this by explicitly saying pkg> add DependentPkg, because you're declaring "I want my default environment to include this registered package."

[kdw503]

If we use Pkg.clone, there would be warning saying "Pkg.clone is only kept for legacy CI script reasons..."
In this case, we can use 'Pkg.add' in the build.jl.
As an example, let TestPkg have a dependency 'ImagineFormat', then TestPkg/deps/build.jl would be

using Pkg
Pkg.add(PackageSpec(url="git@github.com:timholy/ImagineFormat.jl.git"))

And, adding TestPkg would work well also.
But, if we try to use TestPkg, julia shows warning

┌ Warning: Package TestPkg does not have ImagineFormat in its dependencies:
│ - If you have TestPkg checked out for development and have
│   added ImagineFormat as a dependency but haven't updated your primary
│   environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with TestPkg
└ Loading ImagineFormat into TestPkg from project dependency, future warnings for TestPkg are suppressed.

To add 'ImagineFormat' to TestPkg as a dependency, we need to use 'Project.toml' file in TestPkg root directory. (REQUIRE file doesn't work for private repo.)
In this case, we can add 'ImagineFormat' to 'Project.toml' by cloning without using registry.
But, in this case the url of 'ImagineFormat' is recorded in the 'Manifest.toml'.
That means 'TestPkg' also need to keep 'Manifest.toml' file in its repository.
But, if we manage this url in our private registry, we can just add 'ImagineFormat' to 'Project.toml'
Maybe, there would be better way around using registry.
Buit, this was I found until now...

[Cody-G]

It's because the new Pkg is stingy with names as part of its solution of the "dependency hell" problem:

Your explanation seems to apply to registered packages, but something extra seems to be going on with unregistered packages. Given your explanation it makes sense that using DependentPkg fails, but it doesn't make sense that in my case DependentPkg is inaccessible even to the package that added it via the build script:

(substitute DependentPkg for UnitAliases)

julia> using ImagineInterface
[ Info: Precompiling ImagineInterface [3fdf7afa-b2e9-5a34-a557-9a56655c4c08]
ERROR: LoadError: ArgumentError: Package ImagineInterface does not have UnitAliases in its dependencies:
- If you have ImagineInterface checked out for development and have
  added UnitAliases as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with ImagineInterface
Stacktrace:
 [1] require(::Module, ::Symbol) at ./loading.jl:830
 [2] include at ./boot.jl:317 [inlined]
 [3] include_relative(::Module, ::String) at ./loading.jl:1038
 [4] include(::Module, ::String) at ./sysimg.jl:29
 [5] top-level scope at none:2
 [6] eval at ./boot.jl:319 [inlined]
 [7] eval(::Expr) at ./client.jl:399
 [8] top-level scope at ./none:3
in expression starting at /home/cody/.julia/dev/ImagineInterface/src/ImagineInterface.jl:5

My case seems to be what Dae Woo is describing here

But, if we try to use TestPkg, julia shows warning

But for some reason I get an error instead of a warning. Maybe the error changed to a warning in Julia 1.0?

The registry approach seems powerful, but it seems to require a lot of copy-paste between various .toml files and manually generating / interacting with UUIDs. Maybe there's a less labor-intensive way that I didn't discover yet?

[kdw503]

How about trying 'Pkg.resolve()' as error message suggested?
But if the dependency is deeper than one layer (ex) TestPkg->SubTestPkg->SubSubTestPkg..), I don't know what will happen.
There is a section named 'For Package Developer' in README file in this repository.
That instruction will cover the minimum required work to register your packages to this registry.
Buy the way, how many packages do you have to register?

[timholy]

You can use

(v1.0) pkg> activate .

(StaticArrays) pkg> add Foo

to add dependents to a project (you should be in the top-level directory of the package when you do this). I've noticed some errors, though, when using privately registered packages, so in my .julia/config/startup.jl file I also have this:

function getpkgid(name)
    project = Base.active_project()
    Base.project_deps_get(project, name)
end

and frequently copy/paste the UUIDs in. 😦

On balance the experience with Pkg3 for developers is current less ideal than for users; this is pretty well-known ("package users are living in the promised land, but package developers are still awkwardly straddling the old and new worlds (but not for long)"), though fixing it seems slower than I would have expected. It helps to check out this branch of PkgDev.

[Cody-G]

How about trying 'Pkg.resolve()' as error message suggested?

This doesn't change anything :(

.julia/config/startup.jl file I also have this:

Thanks! I had been looking for functions like Base.active_project() and Base.project_deps_get. I guess the reason they're hidden in Base and undocumented is that they shouldn't be needed when everything works well?

I think I'm getting a sense of how things work. That branch of PkgDev is also very nice, especially PkgDev.add_project(pkgdir). I'm realizing that I should have started from the leaves of the private dependency tree, updated each of those for Julia 0.7 and added them to HolyLabRegistry before working on their parents. If you try starting from a parent package (with or without updating the registry) you get stuck on confusing Pkg errors.

A note and some links to relevant issues to watch (for my own reference, maybe others)
Clearing out .julia/clones sometimes fixes things
JuliaLang/Pkg.jl#810
JuliaLang/Pkg.jl#492
JuliaLang/Pkg.jl#465
JuliaLang/Pkg.jl#812

[Cody-G]

One more question. Quoting from the README:

Then, You will see the value in the first line. If you just want to publish current 'master' branch, try this.
$ git cat-file -p master
In this case, every time master branch in the repository of the package is updated with new commit, you need to update this git-tree-sha value also in this registry.

Is there any way to relax this requirement so that the registry always uses a certain branch? (like master). Making a PR here in the registry repo every time a package updates sounds like a lot of work. Or maybe I'm misunderstanding? I looked and wasn't able to find any documentation on Versions.toml.

Edit: this discourse post is the best doc I've found:
https://discourse.julialang.org/t/creating-a-registry/12094

[Cody-G]

And yet another question... I'm trying to add a repo to the registry and getting the "registry dirty" problem that Tim brought up here:

JuliaLang/Pkg.jl#812

How do you usually deal with that? I tried deleting the General registry as suggested here:
https://discourse.julialang.org/t/registry-dirty/17989

but this made things more broken: The General registry does not reinstall automatically as suggested in that post. Is this a 0.7 vs 1.0 difference? I am still on 0.7 but starting to wonder whether I should just skip to 1.0.

[kdw503]

Is there any way to relax this requirement so that the registry always uses a certain branch? (like master).

I think regular package user can use certain branch by pkg> add TileTree#master.
But, if developer wants regular package user to be able to use a specific(mostly latest) update of the package by just pkg> add TileTree, the developer needs to specify 'git-tree-sha' value of the update in the registry.

Making a PR here in the registry repo every time a package updates sounds like a lot of work.

I think this is the reason we manage versions of a package.
We don't need to update 'git-tree-sha' value until next version is released.

Edit: this discourse post is the best doc I've found: https://discourse.julialang.org/t/creating-a-registry/12094

Thanks. This post is quite detail. I'd like to add this link to README.

And yet another question... I'm trying to add a repo to the registry and getting the "registry dirty" problem

When we update packages installed, julia update registry first and than resolve specific version of each installed package. I think when julia update registry, it seems they just pull commits from the same branch in repository to the current branch of local registry location.
So, it looks julia issues this error when local registry has some changes conflicting during the pulling the branch.

[Cody-G]

Thanks!

See also the last few comments here: JuliaLang/Pkg.jl#770.
By committing the Manifest.toml we may need fewer PRs to this registry during development.