Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: make upgrade #255

Merged
merged 1 commit into from
Oct 5, 2021
Merged

fix: make upgrade #255

merged 1 commit into from
Oct 5, 2021

Conversation

phbelitz
Copy link
Member

@phbelitz phbelitz commented Aug 11, 2021

In ADR 1 we discussed the problem of Connaisseurs installation order and why we needed a bootstraping pod and the helm-hook image. This isn't necessary anymore. Now the webhook is part of the normal Connaisseur chart and won't be installed via the helm-hook image. Instead, an empty webhook configuration is applied in the normal installation phase next to the actual Connaisseur pods, which will be filled with the actual data in the post-install stage of the helm installation. This in it's own causes the problem, that everytime Connaisseur is upgraded, the webhook will have a new certificate, while the pods won't unless they are restarted. This is also fixed by taking an already existing certificate by using the helm lookup function and thus not changing the actual certificate.

fix #181, fix #165

Copy link
Collaborator

@xopham xopham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't work for me. I get into a loop where new pods get created and instantly terminated again.

Some further notes:

  • we should include the functionality in one of the integration tests (maybe one that already uninstalls and re-installs connaisseur)
  • please update the docs
  • after ci:add kube-linter #146 , we should be able to get rid of the kubectl config set-context --current --namespace $(NAMESPACE) and only reference the namespace in the helm command

@codecov-commenter
Copy link

codecov-commenter commented Aug 13, 2021

Codecov Report

Merging #255 (b4d1ddd) into develop (5697366) will increase coverage by 0.03%.
The diff coverage is 100.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #255      +/-   ##
===========================================
+ Coverage    96.65%   96.69%   +0.03%     
===========================================
  Files           22       22              
  Lines         1077     1058      -19     
===========================================
- Hits          1041     1023      -18     
+ Misses          36       35       -1     
Impacted Files Coverage Δ
connaisseur/alert.py 97.80% <ø> (+0.80%) ⬆️
connaisseur/flask_server.py 93.15% <100.00%> (-1.36%) ⬇️
connaisseur/util.py 95.34% <100.00%> (+1.06%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5697366...b4d1ddd. Read the comment docs.

@xopham xopham linked an issue Aug 13, 2021 that may be closed by this pull request
@xopham xopham linked an issue Aug 20, 2021 that may be closed by this pull request
Copy link
Collaborator

@xopham xopham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really awesome changes!! 🥳
A few notes:

docs/basics.md Outdated Show resolved Hide resolved
docs/adr/ADR-5_no-more-bootstrap.md Outdated Show resolved Hide resolved
docs/adr/ADR-5_no-more-bootstrap.md Outdated Show resolved Hide resolved
docs/adr/ADR-5_no-more-bootstrap.md Outdated Show resolved Hide resolved
@xopham xopham linked an issue Aug 20, 2021 that may be closed by this pull request
@phbelitz phbelitz force-pushed the fix/helm-upgrade branch 2 times, most recently from 42f3929 to 94a8516 Compare August 20, 2021 09:46
@phbelitz phbelitz requested a review from xopham August 20, 2021 10:02
Copy link
Collaborator

@xopham xopham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Further required changes:

  • Need to update the administration section in the docs/basics.md. Check for further docs that need changing.

connaisseur/flask_server.py Show resolved Hide resolved
helm/templates/_helpers.tpl Show resolved Hide resolved
helm/templates/certificate_webhook-conf.yaml Show resolved Hide resolved
helm/templates/certificate_webhook-conf.yaml Show resolved Hide resolved
helm/templates/certificate_webhook-conf.yaml Outdated Show resolved Hide resolved
helm/templates/certificate_webhook-conf.yaml Show resolved Hide resolved
helm/templates/certificate_webhook-conf.yaml Show resolved Hide resolved
helm/templates/certificate_webhook-conf.yaml Show resolved Hide resolved
helm/templates/certificate_webhook-conf.yaml Outdated Show resolved Hide resolved
@phbelitz phbelitz requested a review from xopham August 20, 2021 13:28
Copy link
Collaborator

@xopham xopham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should mention how to upgrade now in docs/basics.md

docs/adr/ADR-5_no-more-bootstrap.md Outdated Show resolved Hide resolved
docs/adr/ADR-5_no-more-bootstrap.md Outdated Show resolved Hide resolved
docs/adr/ADR-5_no-more-bootstrap.md Outdated Show resolved Hide resolved
docs/adr/ADR-5_no-more-bootstrap.md Outdated Show resolved Hide resolved
docs/adr/ADR-5_no-more-bootstrap.md Outdated Show resolved Hide resolved
docs/adr/ADR-5_no-more-bootstrap.md Outdated Show resolved Hide resolved
docs/adr/ADR-1_bootstrap-sentinel.md Outdated Show resolved Hide resolved
docs/adr/ADR-5_no-more-bootstrap.md Outdated Show resolved Hide resolved
docs/adr/ADR-5_no-more-bootstrap.md Show resolved Hide resolved
Copy link
Collaborator

@xopham xopham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really like the improvement!
I am not sure if it works. Fake upgrade to a same version v2.2.1 worked. Downgrade to v2.1.2 did not work bc readiness probe failed as it requires the webhook. Will we have to make notes for upgrading for the release?

On top of my comments, please fix:

  • there is several instances of the helm-hook remaining, e.g. in helm/values.yaml and some of the tests
  • I suspect there will also still be some notes on it in the docs
  • I would recommend to simply do a grep on the whole repo and check all instances
  • please do not rebase and only fixup, as I can otherwise not track changes
  • we should add an integration test to install from master and upgrade to develop. That will allow us to detect breaking changes as well. Opened ci: connaisseur upgrade integration test #298 for that

Should we wait with merging this for implementation of #280 bc currently it will be unintuitive that the config is not updated

docs/basics.md Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
helm/templates/deployment.yaml Outdated Show resolved Hide resolved
helm/values.yaml Outdated Show resolved Hide resolved
helm/values.yaml Outdated Show resolved Hide resolved
tests/integration/update.yaml Outdated Show resolved Hide resolved
connaisseur/util.py Outdated Show resolved Hide resolved
docs/adr/ADR-5_no-more-bootstrap.md Outdated Show resolved Hide resolved
docs/adr/ADR-5_no-more-bootstrap.md Outdated Show resolved Hide resolved
docs/adr/ADR-5_no-more-bootstrap.md Outdated Show resolved Hide resolved
docs/adr/ADR-5_no-more-bootstrap.md Outdated Show resolved Hide resolved
helm/templates/certificate_webhook-conf.yaml Outdated Show resolved Hide resolved
helm/values.yaml Show resolved Hide resolved
.github/workflows/release.yaml Show resolved Hide resolved
tests/integration/release-integration-test.sh Outdated Show resolved Hide resolved
tests/integration/release-integration-test.sh Outdated Show resolved Hide resolved
Copy link
Collaborator

@xopham xopham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

made some comments

Makefile Show resolved Hide resolved
docs/basics.md Outdated Show resolved Hide resolved
docs/basics.md Outdated Show resolved Hide resolved
@phbelitz phbelitz force-pushed the fix/helm-upgrade branch 2 times, most recently from 417b3db to 3d4b8d2 Compare October 1, 2021 14:55
@phbelitz
Copy link
Member Author

phbelitz commented Oct 1, 2021

changed the pipeline, so the normal integration test is always used (for both release and develop pipeline), but when running in the develop pipeline, the Connaisseur image will be whitelisted, though a manual change in the pipeline

Copy link
Collaborator

@xopham xopham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we are almost there.
Remaining changes are:

  • silently failing when trying to delete the old imagepolicy CRD
  • fixing the release.yaml: most beautiful solution would be to only overwrite that specific entry for the release case, but we would have to pick the entry from an array by number...whatever the case, it needs to be fixed to work

tests/integration/update.yaml Show resolved Hide resolved
xopham
xopham previously approved these changes Oct 4, 2021
Copy link
Collaborator

@xopham xopham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

The bootstrap sentinel and webhook installation job, introduced in ADR-1 were removed. The webhook now gets installed in the post-install phase. During upgrade, Connaisseur validates itself using RollingUpdates and for deletion an empty webhook get applied with a deletion policy set. Additionally the TLS certificates used for communication are only created, if no other already exist. Lastly the pipeline has been adjusted, so that the normal and namespaced integration tests have been merged, testing the upgrade of Connaisseur.

fix #181, fix #165
Copy link
Collaborator

@xopham xopham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@phbelitz phbelitz merged commit 9261dee into develop Oct 5, 2021
@phbelitz phbelitz deleted the fix/helm-upgrade branch October 5, 2021 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix helm upgrade / Connaisseur Update use 'startup probes' instead of sentinel change webhook api version
4 participants