Concurrency issues with flux build without kustomization.yaml

[ansarhun]

With version v4 I noticed that the allenporter/flux-local/action/diff action is started to fail.

In the error there are errors regarding the generation of the kustomization.yaml e.g.:

DEBUG:flux_local.command:Command 'flux build ks crds-traefik --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/clusters/home --namespace flux-system' failed with return code 1
✗ kustomize build failed: kustomization.yaml is empty

DEBUG:flux_local.command:Command 'flux build ks flux-system --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/clusters/home --namespace flux-system' failed with return code 1
✗ failed to generate kustomization.yaml: failed to save original kustomization.yaml: open /Users/ansar/Projects/home-ops/clusters/home/kustomization.yaml: no such file or directory <nil> <nil>

As the error occurs in different places for each run my guess is that it's related to flux build generating the kustomization.yaml file but as multiple build commands are run parallel they will collide on the same file.

To verify I changed the concurrency level to 1 in here: https://github.com/allenporter/flux-local/blob/main/flux_local/command.py#L17
and it seems to "solved" the issue.

[allenporter]

Yeah, i've seen flux build write files while running which previously kind of surprised me. Good find.

Does this mean that both kustomizations crds-traefik and flux-system both have a path of clusters/home? This means then that there are two kustomizations building the same directory at the same time, which is not something expected to happen.

[ansarhun]

My setup is using the same setup as tests/testdata/cluster7
clusters/home contains the top-level kustomize.toolkit.fluxcd.io/v1 Kustomizations

the flux-system is in the clusters/home

but for the crds-traefik: clusters/home contains a crds.yaml

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: crds
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./flux/crds
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

which looks like this:

☸ k0s-cluster in home-ops on  master [$!?] via 🐍 v3.11.6 (venv)
❯ ls -lah flux/crds
total 0
drwxr-xr-x  6 ansar  staff   192B Dec 30 20:02 .
drwxr-xr-x  8 ansar  staff   256B Sep  8  2022 ..
drwxr-xr-x  3 ansar  staff    96B Dec 15 10:18 cert-manager
drwxr-xr-x  3 ansar  staff    96B Dec 21 16:47 kube-prometheus-stack
drwxr-xr-x  3 ansar  staff    96B Oct 14 08:55 stash
drwxr-xr-x  3 ansar  staff    96B Dec  9 09:00 traefik

☸ k0s-cluster in home-ops on  master [$!?] via 🐍 v3.11.6 (venv)
❯ ls -lah flux/crds/traefik
total 8
drwxr-xr-x  3 ansar  staff    96B Dec  9 09:00 .
drwxr-xr-x  6 ansar  staff   192B Dec 30 20:02 ..
-rw-r--r--  1 ansar  staff   618B Dec 30 19:46 crds.yaml

and the crds-traefik:

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: traefik-source
  namespace: flux-system
spec:
  interval: 12h
  url: https://github.com/traefik/traefik-helm-chart.git
  ref:
    # renovate: registryUrl=https://helm.traefik.io/traefik chart=traefik
    tag: v26.0.0
  ignore: |
    # exclude all
    /*
    # include crd directory
    !/traefik/crds
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: crds-traefik
  namespace: flux-system
spec:
  interval: 30m
  prune: false
  wait: true
  sourceRef:
    kind: GitRepository
    name: traefik-source

[allenporter]

Ah! So effectively what is happening then is the crds-traefik kustomzation is being built pointed at the local repo -- but really we should recognize it has a source ref for another repo (which is out of scope of what flux-local can build right now).

A workaround can be to use the --sources or sources tag and include flux-system so that it will ignore the traefik-source repo.

I see two potential issues to follow on here:

• do better at filtering fluxtomizations by git repository, which also means we need a way to pick which gitrepository to use. (we shouldn't try to build kustomization that in the first place)
• fix path canonicalization for why these two are seen as different directories, even though they are the same directory (this will fix the concurrency issue)

Also related is #360 which talks about more issues with external git repos

[ansarhun]

I can confirm if I completely remove the crds Kustomization, which using the other git sources, I do not see the issue

Trying to use the sources argument didn't seems to work (it's still flaky), the crds-kube-prometheus-stack is similar to the traefik one

☸ k0s-cluster in home-ops on  master [$!?] via 🐍 v3.11.6 (venv) took 2s
❯ flux-local diff hr --path ./clusters/home --all-namespaces --sources flux-system --skip-crds --skip-secrets

☸ k0s-cluster in home-ops on  master [$!?] via 🐍 v3.11.6 (venv) took 3s
❯ flux-local diff hr --path ./clusters/home --all-namespaces --sources flux-system --skip-crds --skip-secrets

☸ k0s-cluster in home-ops on  master [$!?] via 🐍 v3.11.6 (venv) took 2s
❯ flux-local diff hr --path ./clusters/home --all-namespaces --sources flux-system --skip-crds --skip-secrets
flux-local error:  Command 'flux build ks crds-kube-prometheus-stack --dry-run --kustomization-file /dev/stdin --path /private/var/folders/53/h59dr4xx68vfyzm7srn0b0hh0000gn/T/tmpyk__o2f0/clusters/home --namespace flux-system' failed with return code 1
✗ remove /private/var/folders/53/h59dr4xx68vfyzm7srn0b0hh0000gn/T/tmpyk__o2f0/clusters/home/kustomization.yaml: no such file or directory

[allenporter]

I made a PR to attempt to add a repo to reproduce the --sources issue but it seems to be filtering for me. Perhaps you can see if i'm missing something? #487

Perhaps try with flux get ks in your repo and see if that does the correct filtering or not.

Does crds-kube-prometheus-stack look roughly the same as the traefik one?

[ansarhun]

Yes the PR looks correct, it's the same as my setup.

---

Filtering with get ks works for me as well

❯ flux-local get ks --path ./clusters/home --all-namespaces --sources flux-system
NAMESPACE      NAME           PATH
flux-system    apps           flux/apps
flux-system    backups        flux/backups
flux-system    charts         flux/charts
flux-system    config         flux/config
flux-system    core           flux/core
flux-system    crds           flux/crds
flux-system    flux-system    clusters/home

vs without filtering

❯ flux-local get ks --path ./clusters/home --all-namespaces
flux-local error:  Command 'flux build ks crds-traefik --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/clusters/home --namespace flux-system' failed with return code 1
✗ kustomize build failed: unable to find one of 'kustomization.yaml', 'kustomization.yml' or 'Kustomization' in directory '/Users/ansar/Projects/home-ops/clusters/home'

---

Also checked diff ks but that also affected

❯ flux-local diff ks --path ./clusters/home --all-namespaces --sources flux-system --skip-crds --skip-secrets
flux-local error:  Command 'flux build ks crds-traefik --dry-run --kustomization-file /dev/stdin --path /private/var/folders/53/h59dr4xx68vfyzm7srn0b0hh0000gn/T/tmpv9m68257/clusters/home --namespace flux-system' failed with return code 1
✗ remove /private/var/folders/53/h59dr4xx68vfyzm7srn0b0hh0000gn/T/tmpv9m68257/clusters/home/kustomization.yaml: no such file or directory

---

Checking the executed flux commands

❯ flux-local --log-level DEBUG get ks --path ./clusters/home --all-namespaces --sources flux-system 2>&1 | grep "Running command: flux"
DEBUG:flux_local.command:Running command: flux build ks apps --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/flux/apps --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks backups --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/flux/backups --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks charts --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/flux/charts --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks config --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/flux/config --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks core --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/flux/core --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks crds --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/flux/crds --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks flux-system --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/clusters/home --namespace flux-system

vs for diff

❯ flux-local --log-level DEBUG diff ks --path ./clusters/home --all-namespaces --sources flux-system 2>&1 | grep "Running command: flux"
DEBUG:flux_local.command:Running command: flux build ks apps --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/flux/apps --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks backups --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/flux/backups --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks charts --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/flux/charts --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks config --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/flux/config --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks crds --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/flux/crds --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks core --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/flux/core --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks flux-system --dry-run --kustomization-file /dev/stdin --path /Users/ansar/Projects/home-ops/clusters/home --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks apps --dry-run --kustomization-file /dev/stdin --path /private/var/folders/53/h59dr4xx68vfyzm7srn0b0hh0000gn/T/tmpvp3ikwaj/flux/apps --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks config --dry-run --kustomization-file /dev/stdin --path /private/var/folders/53/h59dr4xx68vfyzm7srn0b0hh0000gn/T/tmpvp3ikwaj/flux/config --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks core --dry-run --kustomization-file /dev/stdin --path /private/var/folders/53/h59dr4xx68vfyzm7srn0b0hh0000gn/T/tmpvp3ikwaj/flux/core --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks crds --dry-run --kustomization-file /dev/stdin --path /private/var/folders/53/h59dr4xx68vfyzm7srn0b0hh0000gn/T/tmpvp3ikwaj/flux/crds --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks charts --dry-run --kustomization-file /dev/stdin --path /private/var/folders/53/h59dr4xx68vfyzm7srn0b0hh0000gn/T/tmpvp3ikwaj/flux/charts --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks backups --dry-run --kustomization-file /dev/stdin --path /private/var/folders/53/h59dr4xx68vfyzm7srn0b0hh0000gn/T/tmpvp3ikwaj/flux/backups --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks crds-kube-prometheus-stack --dry-run --kustomization-file /dev/stdin --path /private/var/folders/53/h59dr4xx68vfyzm7srn0b0hh0000gn/T/tmpvp3ikwaj/clusters/home --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks crds-stash --dry-run --kustomization-file /dev/stdin --path /private/var/folders/53/h59dr4xx68vfyzm7srn0b0hh0000gn/T/tmpvp3ikwaj/clusters/home --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks crds-traefik --dry-run --kustomization-file /dev/stdin --path /private/var/folders/53/h59dr4xx68vfyzm7srn0b0hh0000gn/T/tmpvp3ikwaj/clusters/home --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks flux-system --dry-run --kustomization-file /dev/stdin --path /private/var/folders/53/h59dr4xx68vfyzm7srn0b0hh0000gn/T/tmpvp3ikwaj/clusters/home --namespace flux-system

it will still try to process the crds-* targets with the sources filter

---

Does crds-kube-prometheus-stack look roughly the same as the traefik one?

Yes

[allenporter]

Added concurrency limits by kustomization path in aee9331

[ansarhun]

I was able to test it and it seems to be working fine now with the fix, thank you!