Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot undo ignore annotation #1211

Closed
tamirms opened this issue Jul 6, 2018 · 6 comments

Comments

@tamirms
Copy link

@tamirms tamirms commented Jul 6, 2018

I added an flux.weave.works/ignore: "true" annotation to some of my kubernetes deployments. In a subsequent commit to my kubernetes repo, I removed the annotations. However, flux still seems to ignore my deployments. I've tried killing the flux and flux memcached pods but that still hasn't fixed the problem. I've also tried renaming my manifest file.

some more info from slack:

in the weave flux agent logs I see
```ts=2018-07-06T17:41:26.352552842Z caller=sync.go:70 component=sync-loop resource=default:deployment/pb-production-accounts-deployment ignore=apply```
so for some reason the ignore tag is stuck to the deployment
I have triple checked our manifest file
and there are no annotations at all

stefan [7:47 PM]
what version are you running?

tamir [7:47 PM]
how can I find that info?

stefan [7:47 PM]
`kubectl get deployment/flux -oyaml`
check the image tag in the output (edited)

tamir [7:54 PM]
I ran `kubectl get deployment/weave-flux-agent -o yaml --namespace=weave`
@stefan the image tag is `quay.io/weaveworks/flux:1.4.2`

stefan [7:56 PM]
that's the latest version, it should work... maybe @mbridgen has any idea how to diagnose this

tamir [8:06 PM]
I’ve tried reinstalling as well
doesn’t seem to do anything

stefan [8:07 PM]
try renaming the deployment file

tamir [8:07 PM]
ok

tamir [8:14 PM]
it didn’t work
`ts=2018-07-06T18:11:55.108793942Z caller=sync.go:70 component=sync-loop resource=default:deployment/pb-production-accounts-deployment ignore=apply`
still shows up in the logs (edited)
I guess I’ll have to do things manually
@mbridgen @stefan let me know if you require any more info from me to diagnose this issue

stefan [8:16 PM]
one more try, can you please delete the flux pod and the memcache one

tamir [8:17 PM]
ok

tamir [8:29 PM]
still didn’t work :disappointed:
is there some volume memcached is using to persist its state?
maybe I can delete that as well?

stefan [8:37 PM]
Nop, my guess is that the git pull that Flux does gets somehow an older version. Can you check in Weave Cloud that you have the right branch

tamir [8:41 PM]
i see this
`ts=2018-07-06T18:37:57.633720149Z caller=loop.go:102 component=sync-loop event=refreshed url=git@github.com:... branch=master HEAD=5126c700b0ea20a1aa3677f9bcf700ad4b4f6200`
the git commit hash is correct
everything looks correctly configured in weave cloud as well

stefan [9:02 PM]
Please make an issue on GitHub with these logs
@bricef bricef added the bug label Jul 9, 2018
@squaremo

This comment has been minimized.

Copy link
Member

@squaremo squaremo commented Jul 9, 2018

Do the resources in the cluster carry the annotation? You will need to remove it from those, as well, because fluxd will ignore something if either the manifest, or the resource itself, has the annotation.

kubectl annotate <resource> "flux.weave.works/ignore"-
@tamirms

This comment has been minimized.

Copy link
Author

@tamirms tamirms commented Jul 9, 2018

@squaremo ah that looks like the cause! so if I want to undo the ignore annotation I should update the yaml configuration in my git repo and also manually run kubectl annotate <resource> "flux.weave.works/ignore"- ?

I would expect that updating the yaml configuration and pushing that to git would also remove the annotation from the resource itself. Also, it seems that running kubectl apply -f ... on the updated configuration does not remove the annotation.. is that to be expected?

@squaremo

This comment has been minimized.

Copy link
Member

@squaremo squaremo commented Jul 9, 2018

so if I want to undo the ignore annotation I should update the yaml configuration in my git repo and also manually run kubectl annotate "flux.weave.works/ignore"- ?

Yes, that should do it. If you want to use ignore in future, it's probably better to just annotate the resource in the cluster. (You might want to annotate the manifest if you didn't want something created in the first place, though.)

I would expect that updating the yaml configuration and pushing that to git would also remove the annotation from the resource itself.

Well .. it's ignoring it ..
Though you might ask "why does it look at the annotations of both the resource in the cluster, and the manifest in git"?

The answer is that the ignore mechanism tries to account for different scenarios, and there's not always a file corresponding to a resource, or a resource corresponding to a file. There was quite a discussion about it, back in #469 (comment).

Also, it seems that running kubectl apply -f ... on the updated configuration does not remove the annotation.. is that to be expected?

Yes; the way kubectl apply merges things can sometimes mean fields that are not mentioned in the update are not removed in the merged resource.

@tamirms

This comment has been minimized.

Copy link
Author

@tamirms tamirms commented Jul 9, 2018

ok if this is expected behavior feel free to close the issue

@squaremo

This comment has been minimized.

Copy link
Member

@squaremo squaremo commented Jul 9, 2018

Sorry if this held you up @tamirms. It's still a somewhat experimental feature, and I don't think it yet hits the simple v useful sweet spot.

@squaremo squaremo closed this Jul 9, 2018
@stefanprodan

This comment has been minimized.

Copy link
Member

@stefanprodan stefanprodan commented Jul 9, 2018

@dholbach please add the instructions on how to remove the annotation with kubectl to #1210 removing the annotation from Git is not going to work.

dholbach added a commit to dholbach/flux that referenced this issue Jul 9, 2018
dholbach added a commit to dholbach/flux that referenced this issue Jul 9, 2018
    document  ignore  annotation in the FAQ, take into
    account fluxcd#1211 and fluxcd#749

    closes fluxcd#955
@squaremo squaremo added the FAQ label Sep 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.