Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[next] auto-updates are disabled on 32.20200416.1.0 #473

Closed
lucab opened this issue May 2, 2020 · 6 comments
Closed

[next] auto-updates are disabled on 32.20200416.1.0 #473

lucab opened this issue May 2, 2020 · 6 comments
Labels
area/updates kind/bug

Comments

@lucab
Copy link
Contributor

lucab commented May 2, 2020

OS version: 32.20200416.1.0 ("next" stream) 
$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora:fedora/x86_64/coreos/next
                 Version: 32.20200416.1.0 (2020-04-16T17:48:33Z)
                  Commit: 19a2e3dc4932d6043d8b437ed981d658bab019cfcde0c4cfe2b8c7d4b5df574a
            GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

ostree://fedora:fedora/x86_64/coreos/testing
                 Version: 31.20200420.2.0 (2020-04-21T16:48:43Z)
                  Commit: 0098d20fe3cea0fbee2ecd5f6e9e7d3b1c45846b61b389bce8f37ddcba690bf5
            GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

On 32.20200416.1.0 (first release on the next stream), Zincati defaults are overridden with an injected configuration fragment which disables auto-updates:

$ cat /etc/zincati/config.d/90-disable-on-non-production-stream.toml 
# https://github.com/coreos/fedora-coreos-tracker/issues/163
updates.enabled = false

Resulting in:

$ sudo socat - UNIX-CONNECT:/run/zincati/public/metrics.promsock | grep 'enabled'
# HELP zincati_update_agent_updates_enabled Whether auto-updates logic is enabled.
# TYPE zincati_update_agent_updates_enabled gauge
zincati_update_agent_updates_enabled 0

$ sudo journalctl -b 0 -u zincati | grep 'auto-updates'
May 02 12:03:11 localhost zincati[733]: [WARN ] initialization complete, auto-updates logic disabled by configuration

However, next is a production stream which should auto-update by default.
dustymabe added a commit to dustymabe/fedora-coreos-config that referenced this issue May 2, 2020
@dustymabe
next: enable automatic updates for next stream
f13d308 
Our next stream is a production stream that we want to have automatic
updates enabled on.

Fixes: coreos/fedora-coreos-tracker#473
@dustymabe dustymabe mentioned this issue May 2, 2020
Merged
@dustymabe
Copy link
Member

coreos/fedora-coreos-config#376 should fix this.

lucab pushed a commit to coreos/fedora-coreos-config that referenced this issue May 3, 2020
@dustymabe
next: enable automatic updates for next stream
99eab31 
Our next stream is a production stream that we want to have automatic
updates enabled on.

Fixes: coreos/fedora-coreos-tracker#473
jlebon added a commit to jlebon/coreos-assembler that referenced this issue May 4, 2020
@jlebon
fcos.upgrade.basic: only force updates on if required
65702a1 
On production streams, we should never have to force Zincati updates to
be on since that should already be the default. So remove that from the
Ignition config and dynamically check this when testing.

We check twice here because we need to account for the case where we're
first rebasing from e.g. `testing-devel` to `next` (where we do have to
force updates) and then an update on `next` itself (where we shouldn't
have to force updates).

This would have caught a regression in `next` where Zincati updates were
disabled: coreos/fedora-coreos-tracker#473.
@jlebon jlebon mentioned this issue May 4, 2020
Merged
@jlebon
Copy link
Member

jlebon commented May 4, 2020

I fixed the upgrade tests to catch this in the future: coreos/coreos-assembler#1422. I verified that the exact next build we have here fails the test:

$ kola -p qemu-unpriv --output-dir tmp/kola-upgrade run-upgrade -v --qemu-image builds/31.20200504.dev.0/x86_64/fedora-coreos-31.20200504.dev.0-qemu.x86_64.qcow2
2020-05-04T16:39:18Z cli: Started logging at level INFO
2020-05-04T16:39:18Z cli: Started logging at level INFO
=== RUN   fcos.upgrade.basic
=== RUN   fcos.upgrade.basic/setup
=== RUN   fcos.upgrade.basic/upgrade-from-previous
=== RUN   fcos.upgrade.basic/upgrade-from-current
--- FAIL: fcos.upgrade.basic (229.47s)
    --- PASS: fcos.upgrade.basic/setup (12.05s)
    --- PASS: fcos.upgrade.basic/upgrade-from-previous (71.49s)
            cluster.go:141: Running as unit: run-r7c3f5367f2a247beb8473b88186c9889.service
    --- FAIL: fcos.upgrade.basic/upgrade-from-current (121.13s)
            basic.go:282: failed waiting for machine reboot: timed out after 2m0s waiting for machine to reboot
FAIL, output in tmp/kola-upgrade
Error: harness: test suite failed
2020-05-04T16:43:07Z cli: harness: test suite failed

@dustymabe
Copy link
Member

workaround for affected systems (on version 32.20200416.1.0):

sudo rm -f /etc/fedora-coreos-pinger/config.d/90-disable-on-non-production-stream.toml
sudo rm -f /etc/zincati/config.d/90-disable-on-non-production-stream.toml
sudo systemctl restart zincati.service

@dustymabe
Copy link
Member

Link to the proposed text for the coreos-status post announcing the next stream and also referring to this issue: https://hackmd.io/Ti1lfql_RX2mwTQYwkAgHA

jlebon added a commit to jlebon/coreos-assembler that referenced this issue May 5, 2020
@jlebon
fcos.upgrade.basic: only force updates on if required
21b1316 
On production streams, we should never have to force Zincati updates to
be on since that should already be the default. So remove that from the
Ignition config and dynamically check this when testing.

We check twice here because we need to account for the case where we're
first rebasing from e.g. `testing-devel` to `next` (where we do have to
force updates) and then an update on `next` itself (where we shouldn't
have to force updates).

This would have caught a regression in `next` where Zincati updates were
disabled: coreos/fedora-coreos-tracker#473.
openshift-merge-robot pushed a commit to coreos/coreos-assembler that referenced this issue May 5, 2020
@jlebon @openshift-merge-robot
fcos.upgrade.basic: only force updates on if required
349ca20 
On production streams, we should never have to force Zincati updates to
be on since that should already be the default. So remove that from the
Ignition config and dynamically check this when testing.

We check twice here because we need to account for the case where we're
first rebasing from e.g. `testing-devel` to `next` (where we do have to
force updates) and then an update on `next` itself (where we shouldn't
have to force updates).

This would have caught a regression in `next` where Zincati updates were
disabled: coreos/fedora-coreos-tracker#473.
@dustymabe dustymabe added the meeting topics for meetings label May 6, 2020
@jlebon jlebon mentioned this issue May 6, 2020
Merged
jlebon added a commit to jlebon/fedora-coreos-pipeline that referenced this issue May 6, 2020
@jlebon
pipeline: add temporary knob to allow upgrade tests to fail
60dd635 
Upgrade tests on the `next` stream are going to fail right now because
of a known regression:

coreos/fedora-coreos-tracker#473

Add a temporary knob to allow continuing the build without it.

I initially tried just blacklisting it, but because it's the only test
in the suite right now, kola errors out due to no tests being available
to run.
@jlebon jlebon mentioned this issue May 6, 2020
Merged
@dustymabe dustymabe removed the meeting topics for meetings label May 7, 2020
@dustymabe
Copy link
Member

sent the email to the coreos-status list: https://lists.fedoraproject.org/archives/list/coreos-status@lists.fedoraproject.org/thread/KHHM3XLMLFMATLBE54RMOBVLXEUF2IFO/

@dustymabe
Copy link
Member

The fix for this went into next stream release 32.20200505.1.0. Please try out the new release and report issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/updates kind/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lucab @dustymabe @jlebon