CLI: allow passing sub-directories of a git repo

[shykes]

Problem

When passing an argument of type Directory, the CLI accepts a git remote url (eg. https://github.com/dagger/dagger) which is awesome; but it does not support passing a sub-directory of that git repo (eg. https://github.com/dagger/dagger/docs) which is not awesome.

Solution

Support passing a sub-directory of a git remote as CLI argument to Dagger CLI.

[jedevc]

This can be done today using something like the following syntax:

dagger -m github.com/vikram-dagger/daggerverse/fileutils call tree --dir=https://github.com/dagger/dagger#main:cmd/dagger

This is the same syntax as found in dockerfiles - see https://docs.docker.com/build/building/context/#url-fragments.

---

Supporting generic slashes is interesting, but I'm not actually sure how possible this is given how nesting works:

• github.com/foo/bar/baz/qux is an org foo with project bar with baz/qux as the path
• gitlab.com/foo/bar/baz/qux could be:
  • an org foo with project bar with baz/qux as the path
  • an org foo with sub-org bar with project baz with qux as the path
  • an org foo with sub-org bar with sub-org baz with project qux

We've discussed this before in relation to module refs - cc @vito. We could just steal what go does today, and use special meta tags in pages (maybe it's worth just biting that bullet?)

[shykes]

This works fine for me. We should just document it.

[shykes]

Supporting generic slashes is interesting, but I'm not actually sure how possible this is given how nesting works:

• github.com/foo/bar/baz/qux is an org foo with project bar with baz/qux as the path

• gitlab.com/foo/bar/baz/qux could be:

  • an org foo with project bar with baz/qux as the path
  • an org foo with sub-org bar with project baz with qux as the path
  • an org foo with sub-org bar with sub-org baz with project qux

We've discussed this before in relation to module refs - cc @vito. We could just steal what go does today, and use special meta tags in pages (maybe it's worth just biting that bullet?)

Don't we have to solve this problem anyway for dagger -m <MOD>?

I do think it's a little weird for users, that we have 2 completely different ways to describe the same thing (a directory inside a remote git repo) depending on which flag it's being passed to.

[vito]

@shykes Yep, that's what the v2 refs proposal was all about (#6187); it was meant to be a unified ref syntax for any Directory, which could be used for both module refs and arbitrary git + subpath refs. But then we got stuck in discussions around whether we want short refs for the Daggerverse and the proposal died.

Combined with #6895 where a contributor is trying to decouple our module refs from GitHub, it feels like it's time to revive that proposal (in spirit, not necessarily in its last form).

[jedevc]

Potential proposal - could we "borrow" the go method of working out where the split is, and then also add our own version in as well (essentially s/go/dagger) for anyone implementing it from scratch?

That would help at least unblock #6895, even if we don't fully get to the full power of refs v2 described in #6187. Personally, I don't love the git syntax we pull from buildkit, I'd happily deprecate it and replace it with go-style refs - it's incredibly non obvious how to use it, I always forget the order of the fragment.

[pjmagee]

How does terraform do it?

module "example_module" {
  source = "git::https://github.com/username/repo.git//path/to/subdirectory?ref=branch_or_tag"
}

[helderco]

@grouville, can you own this as well?