Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression in udev handling when partitions are created by ignition #1194

Closed
jepio opened this issue Sep 29, 2023 · 5 comments · Fixed by flatcar/scripts#1202
Closed

Regression in udev handling when partitions are created by ignition #1194

jepio opened this issue Sep 29, 2023 · 5 comments · Fixed by flatcar/scripts#1202
Labels
kind/bug Something isn't working

Comments

@jepio
Copy link
Member

jepio commented Sep 29, 2023

Description

The following clc snippet (or the equivalent butane or ignition v3) doesn't work:

storage:
  disks:
  - device: /dev/vda
    partitions:
    - label: VAR
      number: 10
      start: '9GiB'
  filesystems:
  - name: var
    mount:
      device: /dev/disk/by-partlabel/VAR
      format: xfs
      label: var
  files:
  - filesystem: root
    path: /etc/fstab
    mode: 0644
    contents:
      inline: |
        /dev/disk/by-label/var /var xfs defaults 0 0

Ignition gets stuck waiting for device to appear and eventually times out and drops to an emergency shell.

Impact

Can't create new partitions in ignition (does it work for a secondary disk?)

Environment and steps to reproduce

  1. fetch flatcar stable disk image
  2. resize: qemu-img resize <disk.img> 20G
  3. run flatcar_production_qemu.sh with ignition

Expected behavior

Success

Additional information

This works on LTS and stable up to major 3374. 3510 and above fails like this:

[    5.780823] systemd[1]: Starting ignition-disks.service...
[    5.832946] ignition[718]: Ignition 2.14.0
[    5.838141] ignition[718]: Stage: disks
[    5.845725] ignition[718]: no configs at "/usr/lib/ignition/base.d"
[    5.854225] ignition[718]: no config dir at "/usr/lib/ignition/base.platform.d/qemu"
[    5.863847] ignition[718]: disks: createPartitions: op(1): [started]  waiting for devices [/dev/disk/by-id/virtio-primary-disk]
[    5.867880] ignition[718]: disks: createPartitions: op(1): [finished] waiting for devices [/dev/disk/by-id/virtio-primary-disk]
[    5.872746] ignition[718]: disks: createPartitions: created device alias for "/dev/disk/by-id/virtio-primary-disk": "/run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk" -> "/dev/vda"
[    5.883482] ignition[718]: disks: createPartitions: op(2): [started]  partitioning "/run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk"
[    5.893657] ignition[718]: disks: createPartitions: op(2): op(3): [started]  reading partition table of "/run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk"
[    5.902335] ignition[718]: disks: createPartitions: op(2): op(3): [finished] reading partition table of "/run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk"
[    5.906493] ignition[718]: disks: createPartitions: op(2): running sgdisk with options: [--pretend --new=10:18874368:+0 /run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk]
[    5.936738] ignition[718]: disks: createPartitions: op(2): running sgdisk with options: [--new=10:18874368:+0 --change-name=10:VAR /run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk]
[    5.940711] ignition[718]: disks: createPartitions: op(2): op(4): [started]  deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk"
[    6.154788] BTRFS: device label OEM devid 1 transid 13 /dev/vda6 scanned by systemd-udevd (446)
[    6.958227] ignition[718]: disks: createPartitions: op(2): op(4): [finished] deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk"
[    6.972495] ignition[718]: disks: createPartitions: op(2): op(5): [started]  waiting for triggered uevent
[    7.172562] ignition[718]: disks: createPartitions: op(2): op(5): [finished] waiting for triggered uevent
[    7.178744] ignition[718]: disks: createPartitions: op(2): [finished] partitioning "/run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk"
[    7.186823] ignition[718]: disks: createFilesystems: op(6): [started]  waiting for devices [/dev/disk/by-partlabel/VAR]
[    7.194799] systemd-networkd[685]: eth0: Gained IPv6LL
[ESC[0mESC[0;31m*     ESC[0m] (1 of 2) Job dev-disk-by\x2dpartlab…vice/start running (2s / no limit)
ESCM^MESC[K[ESC[0;1;31m*ESC[0mESC[0;31m*    ESC[0m] (1 of 2) Job dev-disk-by\x2dpartlab…vice/start running (2s / no limit)
ESCM^MESC[K[ESC[0;31m*ESC[0;1;31m*ESC[0mESC[0;31m*   ESC[0m] (1 of 2) Job dev-disk-by\x2dpartlab…vice/start running (3s / no limit)
ESCM^MESC[K[ ESC[0;31m*ESC[0;1;31m*ESC[0mESC[0;31m*  ESC[0m] (2 of 2) Job ignition-disks.service/start running (9s / no limit)
ESCM^MESC[K[  ESC[0;31m*ESC[0;1;31m*ESC[0mESC[0;31m* ESC[0m] (2 of 2) Job ignition-disks.service/start running (10s / no limit)
ESCM^MESC[K[   ESC[0;31m*ESC[0;1;31m*ESC[0mESC[0;31m*ESC[0m] (2 of 2) Job ignition-disks.service/start running (10s / no limit)
ESCM^MESC[K[    ESC[0;31m*ESC[0;1;31m*ESC[0m] (1 of 2) Job dev-disk-by\x2dpartlab…vice/start running (5s / no limit)
ESCM^MESC[K[     ESC[0;31m*ESC[0m] (1 of 2) Job dev-disk-by\x2dpartlab…vice/start running (6s / no limit)
ESCM^MESC[K[    ESC[0;31m*ESC[0;1;31m*ESC[0m] (1 of 2) Job dev-disk-by\x2dpartlab…vice/start running (6s / no limit)
ESCM^MESC[K[   ESC[0;31m*ESC[0;1;31m*ESC[0mESC[0;31m*ESC[0m] (2 of 2) Job ignition-disks.service/start running (13s / no limit)
ESCM^MESC[K[  ESC[0;31m*ESC[0;1;31m*ESC[0mESC[0;31m* ESC[0m] (2 of 2) Job ignition-disks.service/start running (13s / no limit)

3374 works, the output looks like this:

         Starting ESC[0;1;39mIgnition (disks)ESC[0m...
[    6.825113] systemd[1]: Starting Ignition (disks)...
[    6.869640] ignition[520]: Ignition 2.14.0
[    6.872458] ignition[520]: Stage: disks
[    6.874437] ignition[520]: no configs at "/usr/lib/ignition/base.d"
[    6.878769] ignition[520]: no config dir at "/usr/lib/ignition/base.platform.d/qemu"
[    6.881837] ignition[520]: disks: createPartitions: op(1): [started]  waiting for devices [/dev/disk/by-id/virtio-primary-disk]
[    6.903665] ignition[520]: disks: createPartitions: op(1): [finished] waiting for devices [/dev/disk/by-id/virtio-primary-disk]
[    6.922524] ignition[520]: disks: createPartitions: created device alias for "/dev/disk/by-id/virtio-primary-disk": "/run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk" -> "/dev/vda"
[    6.929480] ignition[520]: disks: createPartitions: op(2): [started]  partitioning "/run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk"
[    6.947332] ignition[520]: disks: createPartitions: op(2): op(3): [started]  reading partition table of "/run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk"
[    6.953799] ignition[520]: di[    7.194624]  vda: vda1 vda2 vda3 vda4 vda6 vda7 vda9
sks: createPartitions: op(2): op(3): [finished] reading partition table of "/run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk"
[    6.966963] ignition[520]: disks: createPartitions: op(2): running sgdisk with options: [--pretend --new=10:18874368:+0 /run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk]
[    6.985414] ignition[520]: disks: createPartitions: op(2): running sgdisk with options: [--new=10:18874368:+0 --change-name=10:VAR /run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk]
[    6.994395] ignition[520]: disks: createPartitions: op(2): op(4): [started]  deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk"
[    7.454212] systemd-networkd[347]: eth0: Gained IPv6LL
[    8.248506]  vda: vda1 vda2 vda3 vda4 vda6 vda7 vda9 vda10
[    8.039379] ignition[520]: disks: createPartitions: op(2): op(4): [finished] deleting 0 partitions and creating 1 partitions on "/run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk"
[    8.050373] ignition[520]: disks: createPartitions: op(2): op(5): [started]  waiting for triggered uevent
[    8.385328] ignition[520]: disks: createPartitions: op(2): op(5): [finished] waiting for triggered uevent
[    8.392767] ignition[520]: disks: createPartitions: op(2): [finished] partitioning "/run/ignition/dev_aliases/dev/disk/by-id/virtio-primary-disk"
[    8.402272] ignition[520]: disks: createFilesystems: op(6): [started]  waiting for devices [/dev/disk/by-partlabel/VAR]
[    8.442391] ignition[520]: disks: createFilesystems: op(6): [finished] waiting for devices [/dev/disk/by-partlabel/VAR]
[    8.451259] ignition[520]: disks: createFilesystems: created device alias for "/dev/disk/by-partlabel/VAR": "/run/ignition/dev_aliases/dev/disk/by-partlabel/VAR" -> "/dev/vda10"
[    8.458365] ignition[520]: disks: createFilesystems: op(7): [started]  determining filesystem type of "/dev/disk/by-partlabel/VAR"
[    8.484467] ignition[520]: disks: createFilesystems: op(7): [finished] determining filesystem type of "/dev/disk/by-partlabel/VAR"
[    8.491105] ignition[520]: disks: createFilesystems: found  filesystem at "/dev/disk/by-partlabel/VAR" with uuid "" and label ""
[    8.493757] ignition[520]: disks: createFilesystems: op(8): [started]  wiping filesystem signatures from "/run/ignition/dev_aliases/dev/disk/by-partlabel/VAR"
[    8.497372] ignition[520]: disks: createFilesystems: op(8): [finished] wiping filesystem signatures from "/run/ignition/dev_aliases/dev/disk/by-partlabel/VAR"
[    8.502561] ignition[520]: disks: createFilesystems: op(9): [started]  creating "xfs" filesystem on "/run/ignition/dev_aliases/dev/disk/by-partlabel/VAR"
[    8.592445] ignition[520]: disks: createFilesystems: op(9): [finished] creating "xfs" filesystem on "/run/ignition/dev_aliases/dev/disk/by-partlabel/VAR"
[    8.598342] ignition[520]: disks: createFilesystems: op(a): [started]  waiting for triggered uevent
[    8.634420] ignition[520]: disks: createFilesystems: op(a): [finished] waiting for triggered uevent
[    8.639457] ignition[520]: disks: disks passed
[ESC[0;32m  OK  ESC[0m] Finished ESC[0;1;39mIgnition (disks)ESC[0m.

Please add any information here that does not fit the above format.
@jepio jepio added the kind/bug Something isn't working label Sep 29, 2023
@pothos
Copy link
Member

pothos commented Sep 29, 2023

PR in flatcar/scripts#1202 - currently waiting for a new nightly SDK

@dongsupark dongsupark moved this from 📝 Needs Triage to ⚒️ In Progress in Flatcar tactical, release planning, and roadmap Oct 2, 2023
@dongsupark dongsupark moved this from ⚒️ In Progress to ✅ Testing / in Review in Flatcar tactical, release planning, and roadmap Oct 2, 2023
pothos added a commit to flatcar/scripts that referenced this issue Oct 2, 2023
@pothos
Run partprobe after Ignition partition changes
eee657b 
This adds coreos/ignition#1717 as downstream
patch to fix flatcar/Flatcar#1194
@pothos pothos mentioned this issue Oct 2, 2023
Merged
2 tasks
pothos added a commit to flatcar/scripts that referenced this issue Oct 2, 2023
@pothos
Run partprobe after Ignition partition changes
2acaf32 
This adds coreos/ignition#1717 as downstream
patch to fix flatcar/Flatcar#1194
@github-project-automation github-project-automation bot moved this from ✅ Testing / in Review to Implemented in Flatcar tactical, release planning, and roadmap Oct 2, 2023
pothos added a commit to flatcar/scripts that referenced this issue Oct 2, 2023
@pothos
Run partprobe after Ignition partition changes
7b96fa2 
This adds coreos/ignition#1717 as downstream
patch to fix flatcar/Flatcar#1194
pothos added a commit to flatcar/scripts that referenced this issue Oct 2, 2023
@pothos
Run partprobe after Ignition partition changes
cdf7db4 
This adds coreos/ignition#1717 as downstream
patch to fix flatcar/Flatcar#1194
@pothos
Copy link
Member

pothos commented Oct 2, 2023

Fix will be part of the next releases (down to version 3602.2.x)

@jepio
Copy link
Member Author

jepio commented Oct 2, 2023

What about LTS-3510?

@pothos
Copy link
Member

pothos commented Oct 2, 2023

It uses an older Ignition version but maybe the patch will still apply - have to check (bootengine also needs a backport).

pothos added a commit to flatcar/scripts that referenced this issue Oct 4, 2023
@pothos
Run partprobe after Ignition partition changes
00bd16b 
This adds coreos/ignition#1717 as downstream
patch to fix flatcar/Flatcar#1194
@pothos pothos mentioned this issue Oct 4, 2023
Merged
@pothos
Copy link
Member

pothos commented Oct 5, 2023

Done

@github-actions github-actions bot mentioned this issue Nov 9, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Run partprobe after Ignition partition changes flatcar/scripts
2 participants
@pothos @jepio