Flux's interaction with resourceVersion?

[roy-work]

Describe the bug

Flux fails with the following error:

ts=2020-03-25T19:12:34.19054984Z caller=sync.go:165 component=daemon err="$NAMESPACE:certificate/$NAME: running kubectl: The certificates \"$NAME\" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update"

ts=2020-03-25T19:12:33.159305724Z caller=sync.go:550 method=Sync cmd="kubectl apply -f -" took=449.849893ms err="running kubectl: The certificates \"$NAME\" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update" output=

My understanding of this error is that there is a resourceVersion attribute on the manifest for this k8s objects. Certain, but not, types of k8s objects require you, when updating the object, to specify the version your update is based on. If that specified version == the current version, the update applies; if it != the current version, that implies that someone else has changes the resource since the version your changes were based on, and the update fails due to the conflict. It's there to prevent you from blowing your foot off, basically, which I can appreciate & understand.

To Reproduce

We're not entirely sure. This is in a setup where we have cert-manager 0.13.1 installed; in the repository Flux is reading from, we have the manifest for a cert-manager Certificate object.

Oddly, though, we have other k8s clusters with the same setup that don't seem to hit this error, so we're not sure what's going on there. (On theory I need to check is that this particular cluster has an update, and the others do not.)

Expected behavior

I'm not sure 😄; that'd be good. Ideally, we want Flux to manage this; if at all possible, I guess Flux should know what the last version it applied was (perhaps this could be tracked in a separate annotation?) and then send the appropriate resourceVersion.

Mostly, however, I am currently just filing this bug to get an understanding of what Flux / Flux's developers think the behavior here is supposed to be, and to find out if I'm doing something completely wrong. E.g., do we need to include this in the manifests ourselves?

Logs

If applicable, please provide logs of fluxd. In a standard stand-alone installation of Flux, you'd get this by running kubectl logs deploy/flux -n flux.

Additional context

• Flux version: We're unfortunately on a custom fork; we're branched off of this commit which is a bit after v1.17.1.
• Kubernetes version: v1.15.7
• Git provider: ? GitHub ?
• Container registry provider: ? Azure CR ?

[sjdweb]

Hey @roy-work I'm experiencing this also. On each apply there's a list of "Errors" in the Slackbot, all certificate resources.

[maelvls]

Hi. I was able to reproduce the issue with a certificate of the form:

# File "./cert-yaml" in the repo targetted by Flux 1:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example
  namespace: default
  resourceVersion: "0"
spec:
  dnsNames:
  - example.boring.mael-valais-gcp.jetstacker.net
  issuerRef:
    group: cert-manager.io
    kind: Issuer
    name: letsencrypt
  secretName: example
  usages:
  - digital signature
  - key encipherment

Detailed steps to reproduce

Prerequisites: brew, gh installed and a GITHUB_TOKEN.

brew install fluxctl
mkdir -p /tmp/test && pushd /tmp/test
gh repo create test --private
cat <<EOF > cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example
  namespace: default
  resourceVersion: "0"
spec:
  dnsNames:
  - example.boring.mael-valais-gcp.jetstacker.net
  issuerRef:
    group: cert-manager.io
    kind: Issuer
    name: letsencrypt
  secretName: example
  usages:
  - digital signature
  - key encipherment
EOF
git checkout -b main
git commit -a -m "Initial commit"
git push --set-upstream origin main
kubectl create ns flux
GITHUB_TOKEN=...
fluxctl install \
    --git-user=maelvls \
    --git-email=mael@vls.dev \
    --git-url=https://user:$GITHUB_TOKEN@github.com/maelvls/test.git \
    --git-path="" \
    --git-branch=main \
    --git-label="" \
    --namespace=flux | kubectl apply -f -
popd

The error is:

ts=2021-07-19T13:07:28.57591597Z caller=sync.go:606 method=Sync cmd="kubectl apply -f -" 
took=109.567153ms err="running kubectl: exit status 1, stderr: The certificates \"example\" is
invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update" output=

ts=2021-07-19T13:07:28.575956888Z caller=sync.go:234 component=daemon
err="default:certificate/example: running kubectl: exit status 1, stderr:
The certificates \"example\" is invalid: metadata.resourceVersion: Invalid value: 0x0: must
be specified for an update"  

Does the Certificate manifest in your GitHub repo contain a resourceVersion?

[kingdonb]

From the Kubernetes API Concepts guide:

Every Kubernetes object has a resourceVersion field representing the version of that resource as stored in the underlying database. When retrieving a collection of resources (either namespace or cluster scoped), the response from the server will contain a resourceVersion value that can be used to initiate a watch against the server. The server will return all changes (creates, deletes, and updates) that occur after the supplied resourceVersion. This allows a client to fetch the current state and then watch for changes without missing any updates.

It doesn't seem like resourceVersion is a value that clients are meant to set. I'm not aware of any behavior that will reject an update if it presents with an outdated resourceVersion, but if you can provide a link I'll be happy to learn something.

This is one of those values that I usually strip out before committing resources back to my cluster via Flux. I very recently heard of this kubectl/krew plugin called kubectl-neat that is meant for neatening up resources and removing overly verbose defaults that need not be in the "intent-style" or upstream version of the resource when it is submitted by Flux.

https://github.com/itaysk/kubectl-neat

@maelvls Thanks for resurrecting this issue and finding a repro. I'm happy to see some progress on it.

[kingdonb]

Is this blocking anyone, or is there a problem remaining here to solve?

I'm going to make a fast pass through all remaining issues soon again, and aiming to close at least 1/4 of the remaining issues on the second pass, I will start by closing issues that are stale or don't have current reports if nobody responds and they are not current or active. I'm not sure if there is a report of a bug here, but I have re-labeled the issue as question because I think there is likely no Flux change here that is likely. As #3320 indicates Flux v1 is in maintenance mode.

Issues requiring new feature development to resolve, or behavior changes, are also on the closing block, as Flux v1 maintenance mode means stronger backwards compatibility guarantees, there cannot be breaking changes during maintenance mode without violating our commitment to preserve the existing behavior for folks depending on stable maintenance. (If there is a feature request here, it would be better addressed to fluxcd/flux2.)

While we do try to answer all support requests, not every issue has gotten follow-up and the default support reply for Flux v1 has become to alert the reporter about Flux v2 and recommend upgrading. This is preferred since there is more activity and developer time allocated there, we want to put the most effort where it will benefit more people in the future.

[kingdonb]

Flux v1 is formally superseded since the GitOps Toolkit APIs have been declared stable:

https://fluxcd.io/docs/migration/timetable/

The repo will remain in maintenance for some time, but no new features can be accepted. Bugs can be addressed if they are critical and there is a PR to resolve it, but soon only CVEs can be addressed in Flux v1, and new users are all recommended to use Flux v2 for some time now.

Thanks for using Flux!