Use `.gitignore` by default when loading modules

[helderco]

Motivation

I have to constantly add "exclude": ["sdk", ".venv"] to dagger.json after dagger init.

I don't expect these files to be uploaded. They're not completely replaced in codegen. During local dev, some files can be created in there and they will stay after codegen in the container. It affects the Source dir cache and slows down module loading.

Solution

The engine already loads dagger.json first. Load .gitignore too. Prepend the .gitignore patterns to the exclude patterns from dagger.json when collecting the patterns for loading the module’s files.

As an escape hatch, if the user needs to include something in .gitignore they can add the same pattern manually to include in dagger.json.

Rationale

This seems like the easiest solution, with less work for the user but keeping control.

Doesn’t require extra work. Most modules in the Daggerverse have the automatically generated .gitignore but no patterns in "exclude". By using .gitignore, those modules would benefit automatically. Of course, this is only necessary when dagger developing modules locally.

When a module is loaded via a git reference, those patterns are already applied because the git clone won’t have those files. So using .gitignore is actually more consistent because it avoids differences in dirty working copies.

It’s still possible to have dirty woking copies because global ignores won’t be picked up, but rather than adding some of those to exclude, users might as well just add them to .gitignore which also helps other users contributing to a module that don’t have the same global ignores.

It’s also possible for files in a module to be ignored by some other .gitignore up the file tree until the git repo root, but we could automatically find those and add their patterns, prefixed with their file path relative to the context directory, to the module’s excludes.

[helderco]

@sipsma, I updated the description. Previously, my thinking was just to add the ignore patterns to the execlude in dagger.json in dagger { init | develop } --sdk=, but wdyt about loading from .gitignore?

[shykes]

This seems reasonable to me but I may not be aware of all the subtleties of the design.

In general I'm in favor of adopting the standard gitignore format whenever we need to express filters on directories (see for example the ignore part of #7647 ). This is one step further in the same general direction.

How would the default behavior be disabled or customized?

[jpadams]

@sipsma, I updated the description. Previously, my thinking was just to add the ignore patterns to the execlude in dagger.json in dagger { init | develop } --sdk=, but wdyt about loading from .gitignore?

@drmikecrowe love to get your thoughts ☝️

[drmikecrowe]

I guess my base question would be: Why wouldn't you adopt .gitignore as the standard? In my case, I'm controlling the release target contents via .gitignore. There's no situation where I want to use local files in dagger that are outside .gitignore because they will never be in my target environment (unless I'm missing something).

As background, here was my new user experience:

• Install dagger
• Run dagger init --sdk=typescript and wait 6 minutes
• Start researching how to exclude (and you can't on dagger init)

In my local directory, I have .yarn/cache which is 6G. Yes, it probably shouldn't be there, but in this case we were experimenting with yarn's pnp mode, which includes the cache in the git lfs

[helderco]

I guess my base question would be: Why wouldn't you adopt .gitignore as the standard? In my case, I'm controlling the release target contents via .gitignore. There's no situation where I want to use local files in dagger that are outside .gitignore because they will never be in my target environment (unless I'm missing something).

Yes, that's what I'm suggesting. I'll reformulate the description when I can to provide more details on how I see this working, including picking up on global git excludes and parent .gitignore files too.

Also, notice this part of the rationale:

When a module is loaded via a git reference, those patterns are already applied because the git clone won’t have those files. So using .gitignore is actually more consistent because it avoids differences in dirty working copies.

So in the same line of thought, when we support other VCSs, I'd do the equivalent to .gitignore in those cases if possible.

[helderco]

How would the default behavior be disabled or customized?

A few ideas:

• You could use dagger.json's own include/exclude patterns to negate or override a .gitignore pattern.
• Disable with a --no-ignore flag that a lot of CLI's use (e.g., rg).
• Disable with a ignore-vcs option in dagger.json (e.g., see hatch).

[drmikecrowe]

Remember, dagger init will not run if there is a dagger.json present -- options 1 and 3 don't seem possible. I think the second option is the only choice of those 3.

Alternatively, scope out a .daggerignore?

Also, does dagger respect .dockerignore?

[helderco]

Remember, dagger init will not run if there is a dagger.json present -- options 1 and 3 don't seem possible. I think the second option is the only choice of those 3.

Those options aren't targetting dagger init, it's mostly dagger call. Remember, those options are for disabling the default use of .gitignore files, but your use case doesn't call for disabling. If you did want to disable, you could do it after dagger init as I can't think why a dagger init would fail based on something you have on .gitignore. Even if that's desirable, a CLI flag can be added there as well.

Also, does dagger respect .dockerignore?

No, it doesn't. I don't see a need for it other than possibly on getting a directory for a docker build, which is different use case, however we use a common Directory for that, uncoupled to the docker build.

[drmikecrowe]

Those options aren't targetting dagger init, it's mostly dagger call.

Help me understand please:

• When would you want to include files in dagger init and then exclude them in dagger call?
• Conversely, when would you need to include files in dagger call that you previously excluded in dagger init?

I'm trying to understand the different use-cases (and to date, my angst has been with excluding and dagger init, so I haven't had time to experience the need in dagger call yet)