Duplicate resource when using .flux.yaml files

[AlessandroEmm]

I'm not sure if it is a feature request, bug or rather a support question, but I'll ask anyway.

Describe the feature
Flux should allow multiple .flux files in different git-folders. I currently get an error like

loading resources from repo: duplicate resource from configuration/xxx/.flux.yaml and configuration/xxx/.flux.yaml.

Looking at the code this seems to be a conscious behaviour, but I don't get why that would be since different git-paths are usually not related hence the use of different folders/path.

What would the new user story look like?
Will be able to tell when I get an answer to my question. 👍

Expected behavior
Flux should allow multiple .flux files in different git-folders.

[2opremio]

Flux should allow multiple .flux files in different git-folders. I currently get an error like

loading resources from repo: duplicate resource from configuration/xxx/.flux.yaml and configuration/xxx/.flux.yaml.

Could you please be more specific about what caused the problem and what you are trying to achieve? (Including the file layout of the git repo, the content of the .flux.yaml files and the --git-path flag passed to Flux, if any).

Flux does support using multiple .flux.yaml files, but you need to make sure not to generate duplicated resources as a result (just like when not using .flux.yaml files, where duplicated resources are not accepted).

[AlessandroEmm]

@2opremio thanks for getting back to me.

I have multiple git-paths like
configuration/xxx/.
configuration/yyy/
configuration/zzz/
Each of them contains an identical .flux.yaml.

version: 1 commandUpdated: generators: - command: "kustomize build"

I'm puzzled since flux is complaining about the .flux.yaml instead of an actual k8s resource files.

[2opremio]

I'm puzzled since flux is complaining about the .flux.yaml instead of an actual k8s resource files.

The resources are generated to stdout so the closest you have to a file is the configuration file which specifies how to generate them.

The example you provide should work just fine as long as there are no duplicate resources generated by the config files, so I would check that.

Unfortunately, I am afraid I cannot help you further without a full, self-contained example which allows me to reproduce the problem.

BTW, flux looks for .flux.yaml files upwards for each --git-path, so, instead of having an identical .flux.yaml file in each directory you could place a single one in the parent directory.

[AlessandroEmm]

@2opremio

If flux only cares about generated resources how come it complains about the .flux files then (as to the error)? Since i'm able to process all folders independently using adhoc kustomize + kubectl, making sure there are no duplicates, I think the problem must be somewhere else. From the source it looks like .flux.yaml are also checked for uniqueness.

I tried that with the parent approach but that didn't work either since I got a lot of errors when processing the resources, which didn't appear when I ran/generated resources for every directory independently.

Could you please elaborate how the parent dir .flux.yaml file executes its commands relatively to the (sub-)directories? I haven't found any documentation on that and I wasn't able to grasp the code of that portion.

Thanks!

[hiddeco]

@AlessandroEmm

If flux only cares about generated resources how come it complains about the .flux files then (as to the error)?

This is because the resource.ID Flux uses internally (and the logic around detecting duplicate resources) hasn't been adjusted to work with Kustomize resources which do not contain a name (kustomization/<no-name> == kustomization/<no-name>).

[2opremio]

Could you please elaborate how the parent dir .flux.yaml file executes its commands relatively to the (sub-)directories? I haven't found any documentation on that and I wasn't able to grasp the code of that portion.

The docs could be friendlier. Here are the important bits:

For every flux target path, Flux will look for a .flux.yaml file in the target path and all its parent directories

The working directory (also known as CWD) of the commands executed from a .flux.yaml file will be set to the target path (--git-path entry) used when finding that .flux.yaml file.

[AlessandroEmm]

Thanks, I understand better now. I think the text itself is ok, I even remember reading both sentences, but wasn't able to bring these together. I guess an example, showing that, would help greatly.

I put a single .flux.yaml in the parent of the subdirectory. And have the --git-path directive pointing to the subdirectories. Weird thing is I do now get this error:

"loading resources from repo: duplicate resource from configuration/cluster-base/dev-lab/.flux.yaml and configuration/cluster-base/dev-lab/.flux.yaml"

[2opremio]

( CC @squaremo , that message is pretty bad, I think we should include the resource ID and the --gitpath used to find each .flux.yaml file)

[AlessandroEmm]

I guess more logging would help generally. 👍

Any idea why it is failing now - struggling with the same path?

[2opremio]

@AlessandroEmm I am not sure, I am afraid can only ask you for a full example to debug it.

[AlessandroEmm]

@2opremio
Here we go: https://github.com/AlessandroEmm/flux-repo.git

fluxResource/base is the configuration for the flux instance
configuration contains the actual resource folders that flux reads.

[2opremio]

Thanks. Can you provide the --git-path(s) you are passing to flux in order to reproduce the problem in that repo? I will take a look once I have that.

[squaremo]

This suggests to me a few UX improvements, even if we don't know exactly what is the problem here:

• complain if a .flux.yaml file is found in a subdirectory of a --git-path -- in that case, it seems fairly clear the expectation is that the .flux.yaml will be interpreted, which is mistaken
• in general be a bit more discerning about which files are treated as manifests; certainly if a YAML doesn't have an apiVersion and a kind, we can't process it anyway, and should probably warn about it

[AlessandroEmm]

@2opremio they are in fluxResource/base/patch.yaml - I based my flux deployment on the base found in this repo here.

[2opremio]

• complain if a .flux.yaml file is found in a subdirectory of a --git-path -- in that case, it seems fairly clear the expectation is that the .flux.yaml will be interpreted, which is mistaken

This assumes --git-paths do not overlap, which may not be the case (nothing prevents the user from providing --git-path=dir1/dir2,dir2 which is perfectly valid in the case of using .flux.yaml files for both directories)

[2opremio]

@AlessandroEmm your setup produces a duplicated resource. To find it I created the following kustomization.yaml file and added it to the root of the repo:

bases:
- configuration/cluster-base/dev-lab/cluster-services
- configuration/cluster-base/dev-lab/devops/connect
- configuration/cluster-base/dev-lab/devops/iam
- configuration/cluster-base/dev-lab/devops/rail

When running kustomize build . at the root of the repo I get the following error:

Error: accumulating resources: recursed merging from path 'configuration/cluster-base/dev-lab/devops/connect': may not add resource with an already registered id: ~G_v1_Namespace|~X|flux

Which means that the flux namespace is produced in different --git-paths.

Flux should get better at reporting the error (including the --git-path and the resource ID which is duplicated and the .flux.yaml file which produced it) but, until then, you can use the propose approach to find duplicates (note that this doesn't take into account the potential duplicates from configuration/cluster-base-static)

[2opremio]

Closing since the root cause is found. I have created #2595 to improve the error message Flux gives.

@AlessandroEmm please reopen if you think the issue isn't solved.

[AlessandroEmm]

@2opremio
I'm not sure I understood exactly - as far as I can tell kustomize struggles because the unpatched resources are the same (they are mostly apart from the namespace and the URL for flux) I'm not sure how that is related to the .flux.yaml file being detected as duplicate? As far as I understood from the source the ID-check happens before running kustomize.

[2opremio]

Flux detects a problem related to duplicated resources. The error message is confusing but it refers to the generated resources by the .flux.yaml file, not the file itself.

The Kustomize problem is equivalent, solve it (by removing the duplicate) and Flux will also work

kustomize struggles because the unpatched resources are the same (they are mostly apart from the namespace and the URL for flux)

Kustomize struggles because the final resources are the same, including the namespace (two resources are the same if they have the same apiVersion, Kind, namespace (if applicable) and name).

As far as I understood from the source the ID-check happens before running kustomize

No, the check in Flux happens after generating the resources.

[squaremo]

This assumes --git-paths do not overlap, which may not be the case (nothing prevents the user from providing --git-path=dir1/dir2,dir2 which is perfectly valid in the case of using .flux.yaml files for both directories)

dir1/dir2 and dir2 aren't overlapping paths. But that aside: if fluxd finds a .flux.yaml in a subdirectory, it must be looking for plain YAMLs, since if it was processing a .flux.yaml file already, it would be doing that instead. So the scenario in which two paths with a .flux.yaml file are given, is not what I'm addressing here.

It is possible that you want to have a subdirectory that gets processed as "plain" YAMLs as well as being subject to a .flux.yaml; but it's much more likely that you've misunderstood how it works, and expected the .flux.yaml in the subdirectory to take effect. Hence the warning suggested.

[2opremio]

dir1/dir2 and dir2 aren't overlapping paths

I meant dir1/dir2 and dir1, sorry

Thanks for the explanation, now I know what you meant.