🐞`dagger project update` vendors all universe

[shykes]

Problem

dagger project update, with no arguments, vendors the entire content of universe.dagger.io in ./cue.mod/pkg/. This causes surprise, confusion and frustration for our developers.

Solution

Modify dagger project update to 1) load the cue project, 2) scan its dependencies, and 3) vendor only the universe packages that are imported.

[helderco]

I see three features here.

New sub-command

• Support "tidy"'ing #2508

I think we need a command that just "installs" what's in cue.mod/dagger.mod, without changing it. The purpose is so that in CI you run this command after a clean git clone that installs your dependencies and verifies the checksums in cue.mod/dagger.sum. This avoids vendoring (committing pkg to git), while protecting against changes.

dagger project update, right now, is doing this. It's both adding the embedded built-in packages to pkg, and also downloading and adding the external packages in dagger.mod. Forget for a moment that there's a difference. Let's assume for the sake of discussion that built-in packages are handled as external in dagger.mod (another planned issue).

What commands should we use for A) just downloading/installing packages; B) Do the tidying, like scanning imports, clean pkg and dagger.mod?

Take into consideration that there's an --update flag to the update sub-sub-command, which can bring confusion (dagger project update --update). Right now, the update sub-command is for updating the dependencies of the project in pkg. And the --update flag is for finding latest versions of packages, instead of just using the version in cue.mod/dagger.mod.

Also, we do have plans to add a tidy, and I think it can be used for what you're describing.

Here's some options for names:

• dagger project update|get|download|install - just download/install packages listed in dagger.mod (get dependencies)
• dagger project tidy|update - scan imports and update dagger.mod (adding and removing requirements), do the same as the above command and clean pkg if necessary

Monorepos/module subdirectories

If we only add universe sub-packages in pkg how do we verify the checksum?

In Go, module paths consist of a repository root path and a module subdirectory (forget major version suffix for the sake of discussion). For most modules, the module path is equal to the repository root path, so the module's root directory is the repository's root directory.

If we add a cue.mod file to each universe package, we can handle it as a mono repo. So there'd be multiple requires with same version (or not), but different subdirs in dagger.mod and dagger.sum.

Avoid vendoring in the first place (global cache)

• Do not require vendoring packages in git (i.e., use a global cache) #2503

You still have to download the whole repo to get access to at least one universe package. If you only vendor some of it, you'll have to download multiple times for any changes.

What I want to do is take an hint from cuemod, which downloads modules into a global cache (GOMODCACHE in that case), and then symlinking them into cue.mod/pkg. The author took inspiration from pnpm, although it uses hard links instead of symlinks (there's a whole discussion on why that's better).

We could do the same. Use the global cache to avoid re-downloading repos and also for verifying the checksums. At which point, you're free to copy only what's needed into cue.mod/pkg, or make links. If you link instead of copy, there's no much need to even commit cue.mod/pkg to git anyway, in which case the need to "break" universe into sub-modules is diminished.

I think both solutions provide greater flexibility. Supporting requiring sub-directories and using a global cache.

[helderco]

We had these two items in the epic list:

• Do not require to commit vendored packages in git (e.g. commit only dagger.mod and dagger.sum, just like go)
• Support "tidy"'ing, i.e. remove stuff in cue.mod/pkg that's not being used.

I've converted them to issues which also act as requirements for this one. Let's focus here on what's in the title which is to only vendor universe partially (or other mono-repos).

---

It's much simpler to just not vendor pkg (not committing to git). If you can rely on dagger.mod and dagger.sum, just reinstall, with global cache or not. There's no advantage in handling sub modules here because it doesn't save you from having to download the whole repo at least once. And it doesn't change the size of your repo.

So this only matters when vendoring.

There's more considerations to supporting module subdirectories. For one, it allows importing different package versions in universe (e.g., netlify version ≠ terraform version), which is fine in itself. But there's some packages that depend on each other. To do this right we would have do declare these dependencies in each sub module's dagger.mod (e.g., universe.dagger.io/netlify/cue.mod/dagger.mod). The alternative is to handle universe in a special way internally. But if not, this pattern can be adopted by others (i.e., single repo with several modules).

Apart from that, I think there's value in a dagger project vendor command, that's optional, for those that prefer committing the dependencies in pkg. This would copy (instead of linking) files over from global cache to pkg (or maybe in dagger project update --vendor), performing a dagger project tidy beforehand to keep it lean, which is what you're asking.

Notice, both tidy and vendor seem similar, but I'm only making this distinction in the context of having a global cache, which I think we shouldn't wait for upstream.

[helderco]

Just to be clear, I'm not opposed to prioritizing a quick fix for this now, which I agree would add quite a bit of value. I was just thinking ahead on how it would look like when some of the essentials get fixed (e.g., handling built-ins in dagger.mod, having a global cache, adding a tidy subcommand, etc).

So, while we still have the binary embedding and recommending vendoring, it makes sense to do this in dagger project update for now. This would be only for universe, and handled in a special way internally, during the vendoring that's already taking place.

We need special handling to avoid adding the cue.mod directories to detect the sub modules. Just matching top level dirs in pkg/universe.dagger.io, excluding x/<email> to then match top level dirs from there as well. Then any import matching those prefixes get copied.

This means importing universe.dagger.io/docker brings the cli sub package along, but that's already a big improvement over copying the entirety of universe.

Do we care for the test dirs?

[shykes]

That all makes sense to me. No strong opinion on the test dirs.

[helderco]

It's harder than on first look. Let's say you only import alpine. Alpine imports docker. So you have to be aware of these dependencies to add them too. Should we also filter some dirs out, like any cue.mod dir?

[helderco]

While developing a reusable module, you can also symlink from outside of your project into pkg. That has it’s own dependencies.

The same thing with modules in dagger.mod. Those may have universe dependencies too.

We need at least an escape hatch to continue vendoring the whole universe, so we avoid people getting stuck over an unforseen case.

[shykes]

I thought the cue API exposed all imports after loading, including transitive dependencies. But I may be wrong. If there are too many edge cases, and rushing a fix creates more problems than it solves, it's OK to skip it or add an escape hatch. Whatever you think is best @helderco .

[helderco]

When you run dagger project update you may not have anything in pkg, so loading like that would fail. I'd need to first move everything there, then see what can be removed.

But even then, you can load multiple different plans in the same project, so the imports potentially look different between "loads".

I'm just overriding Config.ParseFile when using load.Instances(), which parses every cue file in a dir to get their imports:

	buildConfig := &cueload.Config{
		Dir: parentDir,
		ParseFile: func(filename string, src interface{}) (*cueast.File, error) {
			f, err := parser.ParseFile(filename, src)
			if err != nil {
				return nil, err
			}
			for i := range f.Imports {
				importPath, _ := strconv.Unquote(f.Imports[i].Path.Value)

				fmt.Printf("%s => %s\n", filename, importPath)
			}
			return f, nil
		},
	}

	cueload.Instances(args, buildConfig)

This requires some filtering of files, and tracking dependencies in a map, following symlinks, etc.

Is there a better way?

[morlay]

@helderco

ParseFile is the only hook without change origin cue codes.

https://github.com/octohelm/cuemod/blob/main/pkg/cuemod/build.go#L29-L64

type ImportFunc = func(importPath string, importedAt string) (resolvedDir string, err error)

func OptImportFunc(importFunc ImportFunc) OptionFunc {
	return func(c *load.Config) {
		c.ParseFile = func(filename string, src any) (*cueast.File, error) {
			f, err := parser.ParseFile(filename, src)
			if err != nil {
				return nil, err
			}

			for i := range f.Imports {
				importPath, _ := strconv.Unquote(f.Imports[i].Path.Value)

				// skip abs path and rel path
				if filepath.IsAbs(importPath) {
					continue
				}

				// "xxx/xxxx:xxx"
				importPath = strings.Split(importPath, ":")[0]

				// skip builtin
				if builtin.IsBuiltIn(importPath) {
					continue
				}

				_, err := importFunc(importPath, filename)
				if err != nil {
					return nil, err
				}
			}

			return f, nil
		}
	}
}

Notice the imports fixes and skips.