behavior if no Ignition is provided #279

cgwalters · 2019-09-20T13:58:06Z

Today, we happily boot and sit there if the user doesn't provide Ignition. I didn't read the docs and assumed I could paste Ignition into the GCP "Startup script" (Since that's how AWS works).

Also tangentially related, afterburn seems to choke on ed25519 keys, I need to debug that.

Certainly if afterburn doesn't handle SSH keys from the provider (e.g. bare metal) there's no reason to boot and sit there inaccessible, doing nothing - right?

The text was updated successfully, but these errors were encountered:

jlebon · 2019-09-20T14:02:28Z

Today, we happily boot and sit there if the user doesn't provide Ignition.

Hmm, interesting. Are you able to access the logs? It might be that Ignition errored out but OnFailure=emergency.target isn't kicking in correctly.

ajeddeloh · 2019-09-20T17:11:26Z

Ignition uses the GCE "user data" not the startup script iirc. See: coreos/bugs#2558 for a CL analogy.

CL will boot with no ignition config and just add SSH keys from the cloud (via afterburn), which I presume is what we want for FCOS?

bgilbert · 2019-09-20T22:04:25Z

Certainly if afterburn doesn't handle SSH keys from the provider (e.g. bare metal) there's no reason to boot and sit there inaccessible, doing nothing - right?

Afterburn generally does handle provider SSH keys. Also, live ISO counts as bare metal, and we autologin there if we don't get an Ignition config. We could define a set of cases where it'd be reasonable to fail the boot without an Ignition config, but... does it matter? Either way you get a useless machine.

cgwalters · 2019-09-20T22:32:18Z

Either way you get a useless machine.

The thing I got very frustrated about with GCP is it was not very obvious that my ssh key wasn't added. So a fix here could just be making clear in the serial console log if:

We didn't fetch Ignition
No ssh keys were added by afterburn

dustymabe · 2019-09-23T13:44:27Z

Maybe we can build on the work robert did. Here is what we see now on the serial console:

Fedora 30.20190918.dev.3 (CoreOS preview)
Kernel 5.2.14-200.fc30.x86_64 on an x86_64 (ttyS0)

SSH host key: SHA256:OuECn8bBnGxibsCsfN0nR+YrDm+rCRvnCmGe8Edz6Kg (RSA)
SSH host key: SHA256:m8P9DoCKuhKQ8E7loJwF1haGRVIDDRlbaZ1+TcoNQbQ (ED25519)
SSH host key: SHA256:l6ucbtJCX/0SwKzmBzzaDY2qVATKIKk3XnUy1obfPVY (ECDSA)
eth0: 192.168.122.12 fe80::5054:ff:fec0:7829
localhost login:

We could possibly add some info there by adding a new file into /run/console-login-helper-messages/issue.d

ajeddeloh · 2019-12-11T17:02:28Z

I'm in favor of the issue.d method. Ignition can't know if afterburn won't find keys, so it can't fail the boot. Afterburn runs too late to fail the boot in a non-racy way.

cgwalters · 2019-12-11T23:18:55Z

Yep, I have no issue with that approach.

dustymabe · 2019-12-13T00:26:43Z

we discussed this in the meeting yesterday. There was mostly consensus that the issue.d approach would be a good path forward. There was a lot of discussion about how we detect if ignition ran or not. I believe there was an RFE for ignition that came out of that that @jlebon was going to open.

jlebon · 2019-12-13T22:03:34Z

I believe there was an RFE for ignition that came out of that that @jlebon was going to open.

coreos/ignition#903

lucab · 2020-04-02T13:04:01Z

@jlebon your coreos/ignition#903 goes in the direction of structured journal entries. I think it's a good API to have.
Do we still want to have the issue.d/ bits on top? If so, where would that logic live?

dustymabe · 2020-04-02T13:46:20Z

Do we still want to have the issue.d/ bits on top? If so, where would that logic live?

I think the answer is yes. I'm working with Sohan on this part of the puzzle as well as issue.d bits that will let users know if SSH auth keys exist (which is really the information that would have helped Colin understand there was a problem (see original description of this issue)). Where will the script live that generates the issue.d snippets? I think probably either a systemd service/script in FCOS configs OR ignition-dracut. WDYT?

jlebon · 2020-04-02T18:17:26Z

Yeah, agreed we still want a banner notification for both Ignition config provided and SSH keys found. IMO that glue should live in the FCOS configs.

lucab · 2020-04-22T13:19:33Z

I've some doubts on how coreos/afterburn#397 integrates here.

afterburn-sshkeys@core.service may run quite late in the boot process, so I fear that the serial console banner may be printed before keys are fetched.

Is the plan to delay the serial console, or to accommodate false negatives by not printing anything in the "no keys" case?

cgwalters · 2020-04-22T13:46:07Z

afterburn-sshkeys@core.service may run quite late in the boot process, so I fear that the serial console banner may be printed before keys are fetched.

Probably...if it doesn't already do this today, the best fix would be to extend CLHM to re-render if something changes.

dustymabe · 2020-04-22T19:18:00Z

I've some doubts on how coreos/afterburn#397 integrates here.

afterburn-sshkeys@core.service may run quite late in the boot process, so I fear that the serial console banner may be printed before keys are fetched.

Is the plan to delay the serial console, or to accommodate false negatives by not printing anything in the "no keys" case?

How bad is it to delay the console login? I think if it only delays it by tens of seconds then we should just pop Before=systemd-user-sessions.service (I think that will work) on it and call it a day.

Probably...if it doesn't already do this today, the best fix would be to extend CLHM to re-render if something changes.

That would work for motd, but I don't think the issue gets regenerated on the live console if the source files get updated does it?

cgwalters · 2020-04-22T19:19:42Z

I think if it only delays it by tens of seconds then we should just pop Before=systemd-user-sessions.service (I think that will work) on it and call it a day.

s/tens/tenths/? Yeah, I agree.

That would work for motd, but I don't think the issue gets regenerated on the live console if the source files get updated does it?

Well, with a serial console we can only append. If we have an actual video card we can redraw the screen.

rfairley · 2020-05-21T19:29:49Z

Sorry to have missed discussion here before.

afterburn-sshkeys@core.service may run quite late in the boot process, so I fear that the serial console banner may be printed before keys are fetched.

Probably...if it doesn't already do this today, the best fix would be to extend CLHM to re-render if something changes.

Yes, currently CLHM issuegen will only re-generate when it is started on boot, before systemd-user-sessions.service. If services that generate info during boot do Before=console-login-helper-messages-issuegen.service, snippets dropped in by the service into /run/console-login-helper-messages/issue.d should be displayed reliably.

It'll be more reliable though specifying console-login-helper-messages-issuegen.path with the .path unit added (coreos/console-login-helper-messages#47) which will cause CLHM issuegen to regenerate whenever something is dropped into the CLHM issue.d directory. This will also enable writing out issues to the CLHM issue.d directory any time after boot, and the generated issue message will be refreshed.

util-linux/util-linux#1041 was implemented - beginning util-linux v2.36 we'll be able to configure agetty so that /run/issue.d will be searched upon opening the login console with agetty - so no need to rely on c-l-h-m regeneration. But for now, dropping snippets during boot into /run/console-login-helper-messages/issue.d and having Before=console-login-helper-messages-issuegen.service would work if the information is sourced quickly (before issuegen finishes).

That would work for motd, but I don't think the issue gets regenerated on the live console if the source files get updated does it?

Well, with a serial console we can only append. If we have an actual video card we can redraw the screen.

I hadn't thought about refreshing the console after already opening it - maybe CLHM should be calling agetty --reload when regenerating the issue file?

…rized keys This PR addresses the concern raised in coreos/fedora-coreos-tracker#279 which talks about systems behavior when no igntion is provided. Currently, we're tracking ignitionConfig messages(coreos/fedora-coreos-tracker#279) and ssh-authorized keys info (coreos/afterburn#397) by sending the structured entry into journald log. Here, the systemd units are written to scrape through that information to display meaningful data to users.

dustymabe · 2020-05-22T18:59:11Z

The solution to this problem was implemented via changes to various upstream projects but ultimately will now be shown to the user because of the work @sohankunkerkar did in coreos/fedora-coreos-config#344.

dustymabe · 2020-05-22T18:59:17Z

The fix for this landed upstream. It is now pending a testing stream release.

dustymabe · 2020-06-04T14:08:47Z

The fix for this went into testing stream release 32.20200601.2.1. Please try out the new release and report issues.

dustymabe · 2020-06-19T16:26:56Z

The fix for this went into stable stream release 32.20200601.3.0.

Fixes: coreos#53 Also,related to https://bugzilla.redhat.com/show_bug.cgi?id=1851103 It will also help to address this comment: coreos/fedora-coreos-tracker#279 (comment)

dustymabe added the meeting topics for meetings label Dec 11, 2019

jlebon mentioned this issue Dec 13, 2019

RFE: Provide a way for apps to determine whether a user config was provided coreos/ignition#903

Closed

lucab removed the meeting topics for meetings label Dec 17, 2019

dustymabe assigned sohankunkerkar Mar 23, 2020

dustymabe added the jira for syncing to jira label Mar 23, 2020

cgwalters mentioned this issue Mar 24, 2020

RFC: Adding a Network TUI #427

Closed

dustymabe added the status/pending-testing-release Fixed upstream. Waiting on a testing release. label May 22, 2020

dustymabe closed this as completed May 22, 2020

dustymabe added status/pending-stable-release Fixed upstream and in testing. Waiting on stable release. and removed status/pending-testing-release Fixed upstream. Waiting on a testing release. labels Jun 4, 2020

dustymabe removed the status/pending-stable-release Fixed upstream and in testing. Waiting on stable release. label Jun 19, 2020

rfairley mentioned this issue Jul 16, 2020

agetty console should be reloaded upon generation of new issue snippet coreos/console-login-helper-messages#53

Closed

sohankunkerkar mentioned this issue Jul 27, 2020

issuegen: reload the agetty console upon generation of new issue snippet coreos/console-login-helper-messages#62

Merged

cgwalters mentioned this issue Sep 15, 2020

openstack: metadata fetcher may stop retrying before network comes up coreos/ignition#1081

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

behavior if no Ignition is provided #279

behavior if no Ignition is provided #279

cgwalters commented Sep 20, 2019

jlebon commented Sep 20, 2019

ajeddeloh commented Sep 20, 2019

bgilbert commented Sep 20, 2019

cgwalters commented Sep 20, 2019

dustymabe commented Sep 23, 2019

ajeddeloh commented Dec 11, 2019

cgwalters commented Dec 11, 2019

dustymabe commented Dec 13, 2019

jlebon commented Dec 13, 2019

lucab commented Apr 2, 2020 •

edited

Loading

dustymabe commented Apr 2, 2020

jlebon commented Apr 2, 2020

lucab commented Apr 22, 2020

cgwalters commented Apr 22, 2020

dustymabe commented Apr 22, 2020

cgwalters commented Apr 22, 2020

rfairley commented May 21, 2020

dustymabe commented May 22, 2020

dustymabe commented May 22, 2020

dustymabe commented Jun 4, 2020

dustymabe commented Jun 19, 2020

behavior if no Ignition is provided #279

behavior if no Ignition is provided #279

Comments

cgwalters commented Sep 20, 2019

jlebon commented Sep 20, 2019

ajeddeloh commented Sep 20, 2019

bgilbert commented Sep 20, 2019

cgwalters commented Sep 20, 2019

dustymabe commented Sep 23, 2019

ajeddeloh commented Dec 11, 2019

cgwalters commented Dec 11, 2019

dustymabe commented Dec 13, 2019

jlebon commented Dec 13, 2019

lucab commented Apr 2, 2020 • edited Loading

dustymabe commented Apr 2, 2020

jlebon commented Apr 2, 2020

lucab commented Apr 22, 2020

cgwalters commented Apr 22, 2020

dustymabe commented Apr 22, 2020

cgwalters commented Apr 22, 2020

rfairley commented May 21, 2020

dustymabe commented May 22, 2020

dustymabe commented May 22, 2020

dustymabe commented Jun 4, 2020

dustymabe commented Jun 19, 2020

lucab commented Apr 2, 2020 •

edited

Loading