Disable coreOS auto-updates #1241

jorge07 · 2018-04-16T11:19:01Z

jorge07 · 2018-04-16T11:34:46Z

Not tested yet

codecov-io · 2018-04-16T11:49:32Z

Codecov Report

Merging #1241 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1241   +/-   ##
=======================================
  Coverage   36.63%   36.63%           
=======================================
  Files          63       63           
  Lines        3882     3882           
=======================================
  Hits         1422     1422           
  Misses       2242     2242           
  Partials      218      218

Impacted Files	Coverage Δ
core/controlplane/config/config.go	`62.81% <ø> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5bb948d...1e8c5eb. Read the comment docs.

mumoshu

Thank you so much for this. I'm trying to understand... Perhaps you manually rebooted your node so that the update applied to your node?

I'm saying this because we already disable automatic "reboot" normally scheduled after the update has downloaded. And it practically allowed us to prevent automatic updates being actually applied:

#cloud-config
coreos:
  update:
    reboot-strategy: "off"

Anyway, your change LGTM overall. Please notify me once you've finished testing on your side. Thanks!

mumoshu · 2018-04-17T01:27:59Z

core/controlplane/config/templates/cluster.yaml

@@ -13,6 +13,9 @@ s3URI: {{.S3URI}}
 # The AMI ID of CoreOS.
 amiId: "{{.AmiId}}"

+# CoreOS has automatic updates https://coreos.com/os/docs/latest/update-strategies.html. This can be a risk in certain situations. Set this param to true to disable it in all instances.
+disableCoreOSUpdates: false


I'd prefer disableContainerLinuxAutomaticUpdates as it has officially renamed to Container Linux a year or so ago 😃

jorge07 · 2018-04-19T14:10:03Z

@mumoshu I updated the PR renaming this. I did not have time to test it yet, but I'll try to do it asap.

davidmccormick · 2018-04-23T08:50:20Z

I think that, once tested, this should be on by default - not behind a feature gate. :)

mumoshu · 2018-04-23T09:03:52Z

@davidmccormick Ah! That's definitely a good point. I agree.

jorge07 · 2018-04-24T22:07:04Z

Sorry for this waiting for a week. I've the remote team in the office and it's a super busy week. I hope find some time for rebase the branch and launch the cluster next week as I think it would be great have it ready for 0.9.10 final release

mumoshu · 2018-04-25T15:00:51Z

@jorge07 No worry but I just want to sync up with you that I am about to cut v0.9.10 this week! Do you have any reason you want this before #1233 which is I'm going to merge soon after cutting v0.9.10?

jorge07 · 2018-04-26T15:26:42Z

I did the QA today and this is not working at all.

The update-engine.service is running event is sysctemctl status show it as masked.

If I restart the service it fails, obviously because is masked. Any suggestions?

I followed this: https://coreos.com/os/docs/latest/update-strategies.html#disable-automatic-updates-daemon

jorge07 · 2018-04-27T12:21:39Z

With this last change I've it stopped... but:

Update Strategy: No Reboots
Failed Units: 1
  update-engine.service
core@xxx ~ $ systemctl status update-engine
● update-engine.service - Update Engine
   Loaded: loaded (/usr/lib/systemd/system/update-engine.service; disabled; vend
   Active: failed (Result: exit-code) since Fri 2018-04-27 12:14:01 UTC; 4min 27
 Main PID: 789 (code=exited, status=1/FAILURE)

Apr 27 12:13:44 localhost systemd[1]: Starting Update Engine...
Apr 27 12:13:44 localhost update_engine[789]: I0427 12:13:44.362311   789 main.cc:89] CoreOS Update Engine starting
Apr 27 12:13:44 localhost systemd[1]: Started Update Engine.
Apr 27 12:13:44 localhost update_engine[789]: I0427 12:13:44.378746   789 update_check_scheduler.cc:74] Next update check in 11m58s
Apr 27 12:14:01 ip-xxx.eu-west-1.compute.internal systemd[1]: Stopping Update Engine...
Apr 27 12:14:01 ip-xxx.eu-west-1.compute.internal systemd[1]: update-engine.service: Main process exited, code=exited, status=1/FAILURE
Apr 27 12:14:01 ip-xxx.eu-west-1.compute.internal systemd[1]: update-engine.service: Failed with result 'exit-code'.
Apr 27 12:14:01 ip-xxx.eu-west-1.compute.internal systemd[1]: Stopped Update Engine.

Help wanted here...

mumoshu · 2018-04-27T13:27:26Z

@jorge07 Thanks for your efforts. Unfortunately, it seems like we need ignition to actually disable update-engine according to coreos/bugs#1646 and linked issues...

An alternative idea is that to automatically remove update-engine related systemd unit file somewhere in the filesystem so that update-engine would never start in case of a reboot.

mumoshu · 2018-04-27T13:28:46Z

Hm, or maybe just specifying both command: stop and mask: true would do the job?

ref: coreos/bugs#1982 (comment)

jorge07 · 2018-04-27T15:51:32Z

I tried masking the update engine with the stop option too. Same issue, it ends as failed. The cluster its created and it works, but having a failed unit its not nice. Should I start considering refactor into ignition? Sounds like a huge epic

mumoshu · 2018-04-29T13:02:13Z

@jorge07 Thanks for your efforts. If you still want to make it in v0.9.10, I'd recommend my suggestion of removing update-engine(and possibly locksmithd related units).

It would look something like introducing disable-automatic-update.service which would basically run:

for u in update-engine locksmithd; do
  find / | grep lib64/systemd | grep $u | xargs rm
  systemctl stop ${u}.service
  systemctl disable ${u}.service
fi

You probably need this to clear failed units:

systemctl daemon-reload
systemctl reset-failed

But beware that this is completely untested!

mumoshu · 2018-04-29T13:02:59Z

Of course, I'd appreciate it if you could work on the migration to ignition 👍

jorge07 · 2018-04-29T13:11:54Z

I thought the same, I think that create a work around unit that can stop it and clean the failures it's the best option meanwhile migrate into ignition. I'll work on this next week. Thanks for the suggestions!

jorge07 · 2018-04-30T15:23:43Z

It works

ssh core@...

Container Linux by CoreOS stable (1688.5.3)
Update Strategy: No Reboots
core@ip-w-e-b-a ~ $ systemctl status update-engine
● update-engine.service
   Loaded: masked (/dev/null; bad)
   Active: inactive (dead) since Mon 2018-04-30 15:22:00 UTC; 42s ago

mumoshu · 2018-05-01T11:36:27Z

@jorge07 LGTM. Thank you very much for your effort 🍻

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 16, 2018

mumoshu reviewed Apr 17, 2018

View reviewed changes

mumoshu mentioned this pull request Apr 17, 2018

[Proposal] Disable CoreOS update-engine #1240

Closed

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 19, 2018

Disable coreOS auto-updates

1e8c5eb

mumoshu merged commit 185ccc5 into kubernetes-retired:master May 1, 2018

mumoshu added this to the v0.9.10 milestone May 1, 2018

mumoshu mentioned this pull request Jun 12, 2018

General discussion about alternative node provisioner and maybe alternative host OSes #1359

Closed

jorge07 mentioned this pull request Jul 1, 2020

[v0.15.0] Switch CoreOS to Flatcar #1879

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable coreOS auto-updates #1241

Disable coreOS auto-updates #1241

jorge07 commented Apr 16, 2018

jorge07 commented Apr 16, 2018

codecov-io commented Apr 16, 2018 •

edited

mumoshu left a comment

mumoshu Apr 17, 2018

jorge07 commented Apr 19, 2018

davidmccormick commented Apr 23, 2018

mumoshu commented Apr 23, 2018

jorge07 commented Apr 24, 2018

mumoshu commented Apr 25, 2018

jorge07 commented Apr 26, 2018 •

edited

jorge07 commented Apr 27, 2018 •

edited

mumoshu commented Apr 27, 2018

mumoshu commented Apr 27, 2018

jorge07 commented Apr 27, 2018

mumoshu commented Apr 29, 2018

mumoshu commented Apr 29, 2018

jorge07 commented Apr 29, 2018

jorge07 commented Apr 30, 2018

mumoshu commented May 1, 2018

Disable coreOS auto-updates #1241

Disable coreOS auto-updates #1241

Conversation

jorge07 commented Apr 16, 2018

jorge07 commented Apr 16, 2018

codecov-io commented Apr 16, 2018 • edited

Codecov Report

mumoshu left a comment

Choose a reason for hiding this comment

mumoshu Apr 17, 2018

Choose a reason for hiding this comment

jorge07 commented Apr 19, 2018

davidmccormick commented Apr 23, 2018

mumoshu commented Apr 23, 2018

jorge07 commented Apr 24, 2018

mumoshu commented Apr 25, 2018

jorge07 commented Apr 26, 2018 • edited

jorge07 commented Apr 27, 2018 • edited

mumoshu commented Apr 27, 2018

mumoshu commented Apr 27, 2018

jorge07 commented Apr 27, 2018

mumoshu commented Apr 29, 2018

mumoshu commented Apr 29, 2018

jorge07 commented Apr 29, 2018

jorge07 commented Apr 30, 2018

mumoshu commented May 1, 2018

codecov-io commented Apr 16, 2018 •

edited

jorge07 commented Apr 26, 2018 •

edited

jorge07 commented Apr 27, 2018 •

edited