Feature request: scan subdirs for updated versions/projects

[timholy]

Summary

For repositories that have more than one package on them, and some of the packages depend on others, CI runs into trouble because julia --project picks the "umbrella" Project.toml files which sets a new version just for that one package. If the umbrella package depends on new versions of the subdirectory packages, the resolver doesn't know that it can get them from the current PR.

Current workaround

If you're running CI with GitHub Actions, add something like

defaults:
  run:
    shell: bash
...

    steps:
      ...
      - run: julia --project -e 'using Pkg; Pkg.develop([PackageSpec(path="SubDirA"),
                                                         PackageSpec(path="SubDirB"), ...])'

Travis users should add something similar.

Reproducer

Following on the lengthy discussion in #1251, I thought since all the people who actually know Pkg internals were confused about what I was asking, I would create something that is close to an actual test case (and therefore may assist in developing the requested feature). The executive summary here is that Pkg does not yet support updating its awareness of sub-package versions based on a new pull request.

Setup: create a blank directory to do this test in, I chose /tmp/pkgs. Save the script below as /tmp/pkgs/generate_test.jl, make sure you've cd /tmp/pkgs, launch Julia as JULIA_DEPOT_PATH=/tmp/pkgs julia --startup-file=no (I was using Julia 1.6 just to get the latest) and then include the script. Here's the script:

function writelines(path, lines)
    open(path, "w") do io
        for line in lines
            println(io, line)
        end
    end
end

# Create a package BigPkg with a subdirectory SubPkg
# Make BigPkg depend on SubPkg
# Register it
using Pkg
Pkg.add("LocalRegistry")
using LocalRegistry
regpath = joinpath(@__DIR__, "registries", "MyRegistry")
create_registry("MyRegistry", regpath)
# Pkg.Registry.add(regpath)
devpath = joinpath(@__DIR__, "dev")
mkpath(devpath)
bigpkgpath = joinpath(devpath, "BigPkg")
subpkgpath = joinpath(bigpkgpath, "SubPkg")
Pkg.generate(bigpkgpath)
Pkg.generate(subpkgpath)
Pkg.develop(path=bigpkgpath)
Pkg.develop(path=subpkgpath)
cd(bigpkgpath) do
    run(`git init`)
    run(`git add Project.toml src/BigPkg.jl SubPkg/Project.toml SubPkg/src/SubPkg.jl`)
    run(`git commit -m "Skeleton"`)

    Pkg.activate(".")
    Pkg.add(path=".", subdir="SubPkg")
    lines = readlines("Project.toml")
    append!(lines, ["", "[compat]", "SubPkg = \"0.1\""])
    writelines("Project.toml", lines)
    lines = readlines("src/BigPkg.jl")
    insert!(lines, 2, "using SubPkg")
    writelines("src/BigPkg.jl", lines)
    Pkg.activate()
    Pkg.resolve()
    run(`git add Project.toml src/BigPkg.jl`)
    run(`git commit -m "Add SubPkg dependency"`)
end
using BigPkg
using SubPkg
register(SubPkg, "MyRegistry"; repo=bigpkgpath)
register(BigPkg, "MyRegistry"; repo=bigpkgpath)

# OK, now both packages are registered at v0.1.
# Let's add a new feature to SubPkg and bump its version
# to 0.2. Simultaneously, let's require the new feature
# in BigPkg and bump its version to 0.2.
cd(subpkgpath) do
    lines = readlines("src/SubPkg.jl")
    insert!(lines, 2, "export f")
    insert!(lines, 3, "f() = 1")
    writelines("src/SubPkg.jl", lines)
    lines = readlines("Project.toml")
    idx = findfirst(str->startswith(str, "version"), lines)
    lines[idx] = "version = \"0.2.0\""
    writelines("Project.toml", lines)
end
cd(bigpkgpath) do
    lines = readlines("src/BigPkg.jl")
    insert!(lines, 3, "g() = f()")
    writelines("src/BigPkg.jl", lines)
    lines = readlines("Project.toml")
    idx = findfirst(str->startswith(str, "version"), lines)
    lines[idx] = "version = \"0.2.0\""
    lines[end] = "SubPkg = \"0.2\""
    writelines("Project.toml", lines)
    mkdir("test")
    open("test/runtests.jl", "w") do io
        println(io, """
        using BigPkg, Test
        @test BigPkg.g() == 1
        """)
    end

    run(`git add Project.toml src/BigPkg.jl SubPkg/Project.toml SubPkg/src/SubPkg.jl test/runtests.jl`)
    run(`git commit -m "Version 0.2"`)
end

# Remove packages in preparation for simulating what happens when we
# submit a PR and CI runs on Travis
Pkg.rm("SubPkg")
Pkg.rm("BigPkg")

Then I navigate to dev and do this:

tim@diva:/tmp/pkgs/dev$ JULIA_DEPOT_PATH=/tmp/pkgs JL_PKG=BigPkg julia-master --startup-file=no -e 'using Pkg; Pkg.build(verbose=true)'

For some reason, executing the tests as in the default travis script doesn't seem to work for me, but I can do this:

tim@diva:/tmp/pkgs/dev/BigPkg$ JULIA_DEPOT_PATH=/tmp/pkgs julia-master --startup-file=no --project -e 'using Pkg; Pkg.test("BigPkg", coverage=false)'
    Testing BigPkg
┌ Error: Pkg.Resolve.ResolverError("empty intersection between SubPkg@0.1.0 and project compatibility 0.2", nothing)
└ @ Pkg.Operations ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1410
ERROR: empty intersection between SubPkg@0.1.0 and project compatibility 0.2
Stacktrace:
 [1] resolve_versions!(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:370
 [2] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Pkg.Types.UpgradeLevel) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1209
 [3] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; level::Pkg.Types.UpgradeLevel, mode::Pkg.Types.PackageMode, update_registry::Bool, kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:245
 [4] up(::Pkg.Types.Context; kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:level, :mode, :update_registry, :io),Tuple{Pkg.Types.UpgradeLevel,Pkg.Types.PackageMode,Bool,Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:68
 [5] resolve(::Pkg.Types.Context; kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:251
 [6] (::Pkg.Operations.var"#98#102"{String,Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.PackageSpec})() at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1412
 [7] with_temp_env(::Pkg.Operations.var"#98#102"{String,Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.PackageSpec}, ::String) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1331
 [8] (::Pkg.Operations.var"#97#101"{Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String})(::String) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1402
 [9] mktempdir(::Pkg.Operations.var"#97#101"{Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String}, ::String; prefix::String) at ./file.jl:682
 [10] mktempdir(::Function, ::String) at ./file.jl:680 (repeats 2 times)
 [11] sandbox(::Function, ::Pkg.Types.Context, ::Pkg.Types.PackageSpec, ::String, ::String, ::Pkg.Types.Project) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1369
 [12] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1533
 [13] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:327
 [14] #test#61 at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:67 [inlined]
 [15] #test#60 at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:66 [inlined]
 [16] test(::String; kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:coverage,),Tuple{Bool}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:65
 [17] top-level scope at none:1
caused by [exception 1]
empty intersection between SubPkg@0.1.0 and project compatibility 0.2
Stacktrace:
 [1] resolve_versions!(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:370
 [2] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Pkg.Types.UpgradeLevel) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1209
 [3] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; level::Pkg.Types.UpgradeLevel, mode::Pkg.Types.PackageMode, update_registry::Bool, kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:245
 [4] up(::Pkg.Types.Context; kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:level, :mode, :update_registry, :io),Tuple{Pkg.Types.UpgradeLevel,Pkg.Types.PackageMode,Bool,Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:68
 [5] resolve(::Pkg.Types.Context; kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:251
 [6] resolve(; kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:249
 [7] (::Pkg.Operations.var"#98#102"{String,Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.PackageSpec})() at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1407
 [8] with_temp_env(::Pkg.Operations.var"#98#102"{String,Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.PackageSpec}, ::String) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1331
 [9] (::Pkg.Operations.var"#97#101"{Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String})(::String) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1402
 [10] mktempdir(::Pkg.Operations.var"#97#101"{Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String}, ::String; prefix::String) at ./file.jl:682
 [11] mktempdir(::Function, ::String) at ./file.jl:680 (repeats 2 times)
 [12] sandbox(::Function, ::Pkg.Types.Context, ::Pkg.Types.PackageSpec, ::String, ::String, ::Pkg.Types.Project) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1369
 [13] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1533
 [14] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:327
 [15] #test#61 at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:67 [inlined]
 [16] #test#60 at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:66 [inlined]
 [17] test(::String; kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:coverage,),Tuple{Bool}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:65
 [18] top-level scope at none:1

So, the obvious problem is that SubPkg is a dependency of BigPkg, and this new version of BigPkg requires SubPkg@0.2, but SubPkg@0.2 has not been registered yet.

In #1251, @fredrikekre correctly points out that this is no different from how it works normally. But when you have long dependency chains, how it works normally is really painful. When you have a single git repo, you would like to be able to leverage that organization to develop all the packages in concert. To the developer, the most sensible unit is the repo, not the package.

What do we need? I think the only thing missing is some command that will update the session's notion of what package versions are available by scanning the Project.toml files of all packages listed in the [deps] section of BigPkg. If it does that, and then discovers that in the checked-out copy, lo and behold there is a SubPkg@0.2, then it can proceed and try everything together.

[timholy]

I'm trying to work around this issue by modifying my ci.yml file (GitHub Actions) so that the relevant section reads

    steps:
      - uses: actions/checkout@v2
      - uses: julia-actions/setup-julia@v1
        with:
          version: ${{ matrix.version }}
          arch: ${{ matrix.arch }}
      - run: julia -e 'using Pkg; pkg"dev SubPkg"'
      - uses: julia-actions/julia-buildpkg@latest
      - uses: julia-actions/julia-runtest@latest

Sadly, it's not working. Any thoughts? I could go back to adding a build/deps.jl step but that will affect users if I don't forget to strip it out before I ship.

Here's the stacktrace:

ERROR: MethodError: no method matching getindex(::Nothing, ::String)
Stacktrace:
 [1] #build_versions#47(::Bool, ::Function, ::Pkg.Types.Context, ::Array{Base.UUID,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1044
 [2] build_versions at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1033 [inlined]
 [3] #add_or_develop#62(::Array{Base.UUID,1}, ::Symbol, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1204
 [4] #add_or_develop at ./none:0 [inlined]
 [5] #add_or_develop#15(::Symbol, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:69
 [6] (::getfield(Pkg.API, Symbol("#kw##add_or_develop")))(::NamedTuple{(:mode,),Tuple{Symbol}}, ::typeof(Pkg.API.add_or_develop), ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at ./none:0
 [7] do_develop!(::Dict{Symbol,Any}, ::Array{Pkg.Types.PackageSpec,1}, ::Dict{Symbol,Any}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:714
 [8] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:697
 [9] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:696
 [10] do_cmd!(::Pkg.REPLMode.PkgCommand, ::Pkg.REPLMode.MiniREPL) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:603
 [11] #do_cmd#33(::Bool, ::Function, ::Pkg.REPLMode.MiniREPL, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:577
 [12] (::getfield(Pkg.REPLMode, Symbol("#kw##do_cmd")))(::NamedTuple{(:do_rethrow,),Tuple{Bool}}, ::typeof(Pkg.REPLMode.do_cmd), ::Pkg.REPLMode.MiniREPL, ::String) at ./none:0
 [13] top-level scope at none:0

[KristofferC]

The code to repro this seems to not work 100%? It gave me a Project.toml that looked like

name = "BigPkg"
uuid = "e4dbb02c-8206-4006-8b08-6815cbd025b8"
SubPkg = "0.2"

I guess it should be something like

name = "BigPkg"
uuid = "e4dbb02c-8206-4006-8b08-6815cbd025b8"

[deps]
SubPkg = "acd...."

[compat]
SubPkg = "0.2"

Couldn't you check in a manifest in BigPkg where all subdir dependencies are devved? That way you always run CI on the "state of the repo". You could run Pkg.update before testing if you want to use the last version of the other dependencies.

[timholy]

Yes, that looks borked. Strange, it worked perfectly for me. I just tested again, and again it was fine.

The Manifest solution seems to work, many thanks for suggesting what should have been obvious! I have actively avoided them for various reasons (worries about testing stale rather than latest versions of pkgs, concerns about how well they generalize across Julia versions, etc), but this does sound like an excellent workaround. I think we still might consider not making an Manifest.toml obligate, but for the time being this is a simple solution.

[timholy]

Is it possible to add multiple Manifest files for different Julia versions? We're getting CI failures due to what appear to be Julia-version [compat] requirements of some of our dependencies. Yes, we could manually pick an earlier version, but it seems likely that there will be some packages for which there is no single version that works across all Julia versions.

[StefanKarpinski]

There's no way to do that currently, and I'm not sure we want to go down that path. I could turn out to be wrong but it feels like one of those annoyances that may sort itself out over time if we suffer through the discomfort for a while and figure out other ways to deal with it. If you're running code against multiple different Julia versions, isn't it better to not check the manifest in and let the resolver pick versions that are compatible with your Julia version?

[timholy]

Currently that's the only way I know of, other than adding a deps/build.jl script that devs subdirectory-packages (which is a worse solution), of solvingcircumventing this issue. But I'd rather not check in the Manifest.toml at all, frankly.

[KristofferC]

You could dev the sub-packages before running the tests in CI perhaps?

[aminya]

You could dev the sub-packages before running the tests in CI perhaps?

I am confused a bit. This is similar to JuliaDebug/SnoopCompile.jl@bb56316 and my whole issue on it #1828.

If running this script is a bad thing we should not do it at all (including in CI). If it is a good thing why we don't do it by default.

[fredrikekre]

You could dev the sub-packages before running the tests in CI perhaps?

I am confused a bit. This is similar to JuliaDebug/SnoopCompile.jl@bb56316 and my whole issue on it #1828

No, the dev suggestion is part of installing the package, not some runtime adding of packages.