Unable to overlap .gitignore setting

[kvokka]

Bug

I can not manually include garden.yaml, when the project has own .git.

simplified folders structure

├── core
│  ├── .git
│  ├── garden.yml
└── garden.yml

where core is excluded from git.

# .gitignore

/core

# core/garden.yaml
---
# some config here

# ./garden.yaml
kind: Project
name: rp
modules:
  include:
    - core/garden.yaml
...

If try the same config with out .git folder in the projects- garden scan founds the configuration file.

Current Behavior

garden scan               
✔ providers                 → Getting status... → Done
modules: []

Additional context

I tried to put it in another folder, which is not in .gitignore and with out own git repo, but it produced:

$ garden scan
Error validating module poker-core (support/core/garden.yml): key .dockerfile must be a relative sub-path (may not contain '..' segments or be an absolute path)

# support/core/garden.yaml

---
kind: Module
description: Image for poker-core
type: container
name: poker-core
dockerfile: ../../core/docker/development/Dockerfile

For me it is a blocker now.

Your environment

garden version
0.10.9
kubectl version
1.15.2
docker version
2.1.0.2

[edvald]

This is mentioned here in the docs: https://docs.garden.io/using-garden/configuration-files#including-excluding-files-and-directories

By default, we respect both .gitignore and .gardenignore files, and those take precedence over the include/exclude fields. So in your case, if you don't want Garden to respect the .gitignore, you need to set dotIgnoreFiles in your project config. For example:

kind: Project
name: my-project
dotIgnoreFiles: [.gardenignore]

Then you can add files you do want to ignore in a separate .gardenignore. I'm kinda curious why you'd have your Garden modules in .gitignore though?

[kvokka]

Thank you for the solution, but it does not helps. Just nothing changes.

Maybe, the problem root is in the folder structure? in the real world example it is

├── dev
├─── service1

│  ├─── .git
│  ├─── garden.yml
└── garden.yml
└─ .gitignore

There is a few services in dev folder, each has own git repo. Each service should have own garden.yaml file (or, i have to fill in garden.yaml in the project root with the details of each project. It does not looks clean + will require to change tons of paths with duplications). That's why all of them are in .gitignore from the root of the infrastructure repo.

At this moment it looks like in the background garden use git ls which does not follow the files from nested repos, and this is the edge case, when it does not works.

[kvokka]

Maybe, my explanation is clumsy. Just add some bash, which may be handy:

$ garden scan       
✔ providers                 → Getting status... → Done
modules: []

 $ cd core                       
 $ mv .git .git_backup
 $ cd ..                    
 $ garden scan           
✔ providers                 → Getting status... → Done
modules:
  - allowPublish: true
    apiVersion: garden.io/v0
    build:
      dependencies: []
    configPath: /Users/mike/proj/replay/infrastructure/development/core/garden.yaml
    description: Image for poker-core
    kind: Module
    name: poker-core
    outputs:
      deployment-image-name: poker-core
      local-image-name: poker-core
    path: /Users/mike/proj/replay/infrastructure/development/core
    serviceConfigs: []
    taskConfigs: []
...

dotIgnoreFiles only helps to omit ignoring the ignore in .gitignore (pun intended), so this option makes the code cleaner (but it is unusable).

[edvald]

Ah, now I understand (I think). You have submodules in your repo, right? There is a known issue for that: #1097. I'll be working on that soon.

We do support remote repositories however, so until the issue is resolved, you could explicitly point at those repos in your project config, as per this guide: https://docs.garden.io/using-garden/using-remote-sources

[kvokka]

Thank you, @edvald , it may be a solution. Will try it a bit later and give back the feedback a bit later.

And I feel, that I understood the roop of the problem here. Garden calculates the files for .gitignore and .gardenignore basics on git ls for both options. But it is wrong, since .gardenignore is not a git and should look through all subfolders, no matter to what repo they belongs to. It is only a guess, but it has a good match.

[edvald]

It's a little tricky to get all this right, and so that it behaves how people expect it to in all/most situations. We do use git for resolution of files, for a number of good reasons, and I'd argue that it would in many cases be unexpected for .gardenignore files to work somehow differently than .gitignore. There are no special semantics for ignore files of different names, we just allow the user to pick which .ignore filenames to look for.

I'd say the real issue is that we're currently not supporting submodules properly, which is something we definitely need to resolve, as well as to document any potential caveats.

[kvokka]

It's totally ok to use the same semantics. And it means that I described myself earlier not quite clear, sorry about that.

Garden also use .gitignore files scope for .gardenignore. So, all files in mask git ls are managed by .gardenignore, when it should be just **/*.

It is the case when you do not use .gitignore, with dotIgnoreFiles: [.gardenignore] option. In this case for developer it is obvious to think that all files in this sub-folders should be managed only by .gardenignore. With the same regular semantics of ignore files, but all files.

[edvald]

Just to explain a little: We still only scope .ignore files within repos, but include/exclude fields work across those boundaries. This is to stay in line with standard Git behavior, and also because other options would have been prohibitively complex to implement. Making special cases for different types of ignore files felt confusing to explain, so I went with something reasonably straightforward.

Check the added section here: https://docs.garden.io/using-garden/configuration-files#git-submodules

[kvokka]

Just some additions from my side:

1. The issue initially was not about sub-modules. The project, which was used for the test does not contains git-submodules. It contains some paths, which are in .gitignore, which contains another repos. Like:

# .gitgingore
development/core

$ tree --prune -P 'garden.yaml'                
.
└── development
    ├── core
    │   └── garden.yaml
    ├── garden.yaml
    └── support
        ├── env_vars
        │   └── garden.yaml
        ├── postrgesql
        │   └── garden.yaml
        ├── pubsub_emulator
        │   └── garden.yaml
        ├── rbac
        │   └── garden.yaml
        └── redis
            └── garden.yaml

With this structure i can not just cd development && garden dev since there is no way to reach development/core/garden.yaml. The way which works is to put it in development/support/core/garden.yaml and set up remoteUrl.

2. Still do not understand, the case, when developer have only dotIgnoreFiles: [.gardenignore]. It looks like that this use only .gardenignore files. But actually, it still get files list from git and apply .gardenignore after. It is hidden dependency.
3. Developer can not manually add include that paths which are in.gitignore. They are ignored.
4. All .ignore files must follow the same semantics and ignore files, which are in this and only this type of ignore files.

[edvald]

Thanks for clarifying, I took the fact you had a nested git repository to mean you had submodules.

I can in turn clarify how we do our file scanning. Basically we always use git to scan for files. Git is a declared dependency of Garden, and Garden will fail to start if you don't have it. This is done for performance reasons, as well as to natively support .ignore files, including how they deal with multiple levels of ignore files (which is, turns out, the complicated part).

This means that we factor in .ignore files first, and then we apply include/exclude filters. This is mentioned in our docs, and is by design. So when you set e.g. dotIgnoreFiles: [.gardenignore], we do still use git to scan for files, but don't factor in . gitignore files, only .gardenignore files. Same semantics, just separately configured.

I hope that's more clear. Regarding your project structure, you do currently need to have the project config higher in your tree than your module configs, or at the same level. This was also a design choice, so that Garden can easily scan to find your project config, and have sane defaults as to where to look for files. Similar to many or most package managers etc.

That said, we can still look into allowing configuring relative paths outside of a config directory, but we'd need to make sure the paths are still within the same repository, and we could not allow it for all such fields because it could trigger cascading edge-cases to support. But I get where you're coming from :)

[kvokka]

Thank you @edvald for a such a quick and full response.

This limitation wire the hands for the projects with multiple repositories, since on one hand i should have the project level config on the top (where it was all the time), on the other hand i can not put another projects in the sub-directories under the project level config (moving all services in 1 repo is not applicable for my case) since each of them has own get repository.

I'm looking forward for adjusting the workflow process for the tool, but at this moment this case does not have the solution.

At this moment I may only ask to share this limitation in the docs with required project (repo) structure, since it may save time for other developers.

[edvald]

I'm still not quite sure I understand your overall layout, and why it's hard to accommodate, but it seems you've tried a lot of different options. I expect you tried the remote sources features (https://docs.garden.io/using-garden/using-remote-sources), which on the surface would seem to be more geared towards your setup, but I gather that also didn't work out?

[kvokka]

Yes, i tried it already. That gave the ability to start the service, but #1205. So does not actually works, plus lack of hot reload even for tests (since Garden does not detect the service as started).
Plus this solution makes the copy of the project with git --deep 1 in the .garden guts. Because of #1205 it was not possible to fully test it. Some poking around did not show the sync, but i may be because of failed start.

[edvald]

Understood. We'll fix #1205 for sure, I figure that'll be a blocker either way. But the shallow Git clone only applies if you don't use garden link. When you link a source, you attach directly to the local clone you have of the source repo, so you should see changes sync the same as with any other directory.

[kvokka]

Thank you for such a great support!

Will be happy to try out this approach later.