ServiceAccount/external-dns namespace not specified, error: the server could not find the requested resource

[abdullahskyc]

Hey, I'm new to Flux as I'm self learning it. I've recently upgraded to the latest version of flux using a bootstrap command:
flux bootstrap git \ --components-extra=image-reflector-controller,image-automation-controller \ --url=ssh://git@path_to_repo \ --branch=main \ --path=$REPO_PATH \ --private-key-file=$PRIVATE_SSH_KEY
This resolved a problem which fluxcd was not automatically changing tag names for manifest deployment files. Now it changes the tags and commits the change. For some reason the actual dns where the deployment is hosted is not updated, perhaps fluxcd bot doesn't apply the changes? Something to do with kustomize it seems like.

In the bootstrap command, upon waiting for waiting for Kustomization "flux-system/flux-system" to be reconciled,
I receive the error:
ServiceAccount/external-dns namespace not specified, error: the server could not find the requested resource which has also changed to:
Kustomization reconciliation failed: ConfigMap/my-cert namespace not specified, error: the server could not find the requested resource for some reason, maybe it resolved the serviceAccount but not the configMap?.

But I've checked in my external-dns.yaml and it does include namespace of default.
I have searched around but don't really know where I need to specify the namespace, any help would be great. Is this an issue with kustomize? Or something with the configMap? I'm also a noob at kubernetes :(

Kustomize version: {Version:kustomize/v4.4.0 GitCommit:63ec6bdb3d737a7c66901828c5743656c49b60e1 BuildDate:2021-09-27T16:24:12Z GoOs:linux GoArch:amd64}

Checking my flux logs, for my ImagePolicies, I keep seeing an error of 'Reconciler error version list argument cannot be empty'

[kingdonb]

The recent changes in Flux behavior have been centered around behavior when merging existing configurations with Flux configs.

https://fluxcd.io/docs/faq/#why-are-kubectl-edits-rolled-back-by-flux

Basically, if you just upgraded to Flux 0.26.x and found some config dropped out from your cluster, that config doesn't live in git. This FAQ was just recently added to address the expectations of users that could kubectl edit a resource and as long as those changes don't conflict with what is in Git, Flux would not overwrite them. It's no longer true, this idea conflicted with too many important expectations and it was finally an upgrade from Kubernetes into Server-Side Apply with some surprising behavior that forced a change.

I'm not sure how this upgrade could result in a "namespace not found" error but, you can usually inspect the output of Kustomize by using this command to kustomize build:

kustomize build --load-restrictor=LoadRestrictionsNone --reorder=legacy .

(as described here: https://fluxcd.io/docs/faq/#why-are-kubectl-edits-rolled-back-by-flux)

so you can see which resources still/now don't have a metadata.namespace after Kustomize has rendered them out, and make changes to test without committing and pushing then waiting for Flux to apply again and reading the error there. The link above also gives how to test a dry-run serverside apply as Flux does. (If you don't get the same results there, I'll be happy to try and diagnose it further with you, but this should provide the necessary transparency to help clarify what is going wrong.)

Also be aware of the new flux diff and flux build features which may help short-circuit some of the operational complexity that I just described and get you the answers quicker when testing a change.

Finally as I mentioned some things have changed drastically, and you mentioned an upgrade but you haven't noted what version of Flux you are using. Be sure you are on the latest version of Flux, or please include the version that you're using if possible in your reply! I do personally know what the latest version is now, but these posts are not always responded quickly, and we sometimes publish releases in quick succession, sometimes as quickly as once a day. So this will be hard to verify without including those specific details.

[abdullahskyc]

Current versions (running flux version command):

flux: v0.26.3
helm-controller: v0.16.0
image-automation-controller: v0.20.0
image-reflector-controller: v0.16.0
kustomize-controller: v0.20.2
notification-controller: v0.21.0
source-controller: v0.21.2

And kustomize version:
{Version:kustomize/v4.4.0 GitCommit:63ec6bdb3d737a7c66901828c5743656c49b60e1 BuildDate:2021-09-27T16:24:12Z GoOs:linux GoArch:amd64}

When I run that command kustomize build --load-restrictor=LoadRestrictionsNone --reorder=legacy ., I receive an error:

Error: accumulating resources: accumulation err='accumulating resources from 'gotk-
components.yaml': yaml: line 558: could not find expected ':'': got file 'gotk-
components.yaml', but '/root/kubernetes-flux/clusters/my-cluster/flux-system/gotk-
components.yaml' must be a directory to be a root

I assume this is a problem with the manifest file (flux-system/kustomization.yaml controller), on line 558/559? Strange since this is an autogenerated file by flux, though I did have to resolve a merge conflict initially. But this particular error seems to be haunting me when running any command related to flux:
flux reconcile kustomization flux-system --with-source OR kubectl apply -k flux-system/
But there seems to be no error on those lines:

556                        matchLabels:
557                          additionalProperties:
558                            type: string
559                          description: MatchLabels is a map of {key,value} pairs.
560                            A single {key,value} in the matchLabels map is equivalent
561                            to an element of matchExpressions, whose key field is
562                            "key", the operator is "In", and the values array contains
563                            only "value". The requirements are ANDed.

Maybe I should just rerun another installation but I don't know how these gotk-*.yaml files were generated and what command will generate such files, but seems like an indentation error/whitespace. But after installing flux again via bootstrap command, the original error has returned (only when reconciling with Kustomization):

◎ waiting for Kustomization reconciliation
✗ Kustomization reconciliation failed: ConfigMap/my-cert namespace not specified, error: the server could not find the requested resource

What I will try is to find a way to generate the same gotk-components.yaml file and see if I can run the kustomize build command in an attempt to troubleshoot this

The output from flux get kustomizations --watch:

NAME            READY   MESSAGE                                                                                                         REVISION                                          SUSPENDED 
flux-system     False   ConfigMap/my-cert namespace not specified, error: the server could not find the requested resource   main/8ec43da053c1dd8255779b1e6cf9110009f97911     False  

But this contradicts with the error from kustomize build --load-restrictor=LoadRestrictionsNone --reorder=legacy . doesn't it? Which yields the aforementioned line 558: could not find expected ':'': got file 'gotk-components.yaml'

Update 1: It was formatting for the gotk-components.yaml file shown by yamllint, now that error has been resolved using some auto formatting (working with nano makes it difficult).

Using that link you sent me, I run a command:

kustomize build --load-restrictor=LoadRestrictionsNone --reorder=legacy . 
| kubectl apply --server-side --dry-run=server -f-

Which states some errors with 'kustomize-controller':

Apply failed with 4 conflicts: conflicts with "kustomize-controller":
- .status.conditions
- .status.storedVersions
- .status.acceptedNames.kind
- .status.acceptedNames.plural
Please review the fields above--they currently have other managers. Here
are the ways you can resolve this warning:
* If you intend to manage all of these fields, please re-run the apply
  command with the `--force-conflicts` flag.
* If you do not intend to manage all of the fields, please edit your
  manifest to remove references to the fields that should keep their
  current managers.
* You may co-own fields by updating your manifest to match the existing
  value; in this case, you'll become the manager if the other manager(s)
  stop managing the field (remove it from their configuration).
See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts

These errors are stated again till another failure in kubectl apply:

Apply failed with 2 conflicts: conflicts with "flux":
- .spec.template.spec.containers[name="manager"].resources.limits.cpu
conflicts with "kubectl-client-side-apply" using apps/v1:
- .spec.template.spec.containers[name="manager"].resources.limits.cpu

Which also repeat for some time. Im confused at the fact that to resolve these warnings I have to edit the generated gotk-components.yaml file which itself states not to be edited. Where do I edit the 'flux' and 'kustomize-controller' configs anyway? gotk-components.yaml doesn't seem to have all the conflicting fields when I search for '.spec.template' or 'status.acceptedNames'.

** Update 2:**
Looking specifically at configMaps (where the issue arose), it seems that configMap/my-cert only existed for namespace default and not for 'flux-system', but even though I created an identical configMap (copy, paste) via:

kubectl get configmap my-cert --namespace=default -o yaml | sed 's/namespace: default/namespace: flux-system/' | kubectl create -f -

The reconcile still throws the same error where ConfigMap/my-cert namespace not specified, error: the server could not find the requested resource

Weirdly enough, I don't even know how this configMap got mixed up in all this, likely due to the flux bootstrap command which referenced a cert

[abdullahskyc]

Ok the issue was basically referenced here, #1939 but being a noob in kubernetes as well, I didn't realize sooner that the kustomize-controller introduced these changes which needed namespace added to alot of the manifest files. Since I had ALOT of deployments and as such; ingresses, services and other types (secret, cert, etc.) which needed me to update the metadata tag in each of their respective files.

So I just ran a find and sed to do this (recursively):

find  ./ -name '*.yaml' -type f -exec  sed -i '3s/metadata:/metadata:\
 namespace: default/g' {} \;

Then executed flux reconcile kustomization flux-system --with-source

Finally I saw all my deployments updated.

Thanks for the commands, they helped in debugging :)

[teocns]

1) Add namespace: default where missing:

import os
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path

import ruamel.yaml
from ruamel.yaml import YAML


def process_file(filepath):
    yaml = YAML()
    yaml.preserve_quotes = True

    docs = list(yaml.load_all(Path(filepath)))

    updated = False
    for doc in docs:
        # print('Type of doc: ', type(doc))
        if not doc:
            continue
        if (
            "metadata" in doc
            and "namespace" not in doc["metadata"]
            and "kind" in doc
            and "apiVersion" in doc
        ):
            doc["metadata"]["namespace"] = "default"
            updated = True

    if updated:
        yaml.dump_all(docs, Path(filepath))


# Walk through the directory
with ThreadPoolExecutor() as executor:
    for dirpath, dirnames, filenames in os.walk("./"):
        for filename in filenames:
            if filename.endswith(".yaml") or filename.endswith(".yml"):
                filepath = os.path.join(dirpath, filename)
                # process_file(filepath)
                executor.submit(process_file, filepath)

2) Only stage + namespace: default additions, ignoring trash reformats:

git diff -U0 | grepdiff 'namespace: default' --output-matching=hunk  | git apply --cached --unidiff-zero

brew install patchutils for grepdiff