New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignition should support filesystem reuse semantics for partitioning #2388

Closed
jmreicha opened this Issue Mar 26, 2018 · 11 comments

Comments

Projects
None yet
5 participants
@jmreicha

jmreicha commented Mar 26, 2018

Issue Report

Bug

Container Linux Version

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1632.3.0
VERSION_ID=1632.3.0
BUILD_ID=2018-02-14-0338
PRETTY_NAME="Container Linux by CoreOS 1632.3.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

AWS with EBS data volume

Expected Behavior

Ignition should skip trying to partition the disk if it has already been partitioned but it is not doing this.

Per another issue I tried the wipeTable: false and wipeFilesystem: false flags but they don't seem to have any effect.

Actual Behavior

The first time Ignition runs, it creates the partition and everything works correctly.

The problem is when attempting to attach the EBS volume to a new instance after creating the partition table and directories, etc and then rerunning Ignition on the new server.

systemd[1]: Starting Ignition (disks)...

ignition[405]: Ignition v0.20.1
ignition[405]: reading system config file "/usr/lib/ignition/base.ign"
ignition[405]: no config URL provided
ignition[405]: reading system config file "/usr/lib/ignition/user.ign"
ignition[405]: no config at "/usr/lib/ignition/user.ign"
ignition[405]: disks: createPartitions: op(1): [started]  waiting for devices [/dev/xvdb]
ignition[405]: disks: createPartitions: op(1): [finished] waiting for devices [/dev/xvdb]
ignition[405]: disks: createPartitions: created device alias for "/dev/xvdb": "/dev_aliases/dev/xvdb" -> "/dev/xvdb"
ignition[405]: [    2.678373]  xvdb: xvdb1
disks: createPartitions: op(2): [started]  partitioning "/dev_aliases/dev/xvdb"FAILED Failed to start Ignition (disks).
See 'systemctl status ignition-disks.service' for details.
Dependency failed for Ignition (record completion).
Dependency failed for Initrd Default Target.
Dependency failed for Ignition (files).

and later in the logs.

systemd[1]: ignition-disks.service: Main process exited, code=exited, status=1/FAILURE
ignition[405]: disks: createPartitions: op(2): op(3): [started]  creating 1 partitions on "/dev_aliases/dev/xvdb"
systemd[1]: ignition-disks.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Ignition (disks).
ignition[405]: disks: createPartitions: op(2): op(3): [failed]   creating 1 partitions on "/dev_aliases/dev/xvdb": exit status 4: Cmd: "/usr/sbin/sgdisk" "--new=1:0:+0" "--change-name=1:STATEFUL" "/dev_aliases/dev/xvdb" Stdout: "Setting name!\npartNum is 0\nREALLY setting name!\n" Stderr: "Could not create partition 1 from 34 to 2047\nError encountered; not saving changes.\n"
systemd[1]: Dependency failed for Ignition (record completion).
ignition[405]: disks: createPartitions: op(2): [failed]   partitioning "/dev_aliases/dev/xvdb": commit failure: create partitions failed: exit status 4: Cmd: "/usr/sbin/sgdisk" "--new=1:0:+0" "--change-name=1:STATEFUL" "/dev_aliases/dev/xvdb" Stdout: "Setting name!\npartNum is 0\nREALLY setting name!\n" Stderr: "Could not create partition 1 from 34 to 2047\nError encountered; not saving changes.\n"
systemd[1]: Dependency failed for Initrd Default Target.
ignition[405]: disks: create partitions failed: commit failure: create partitions failed: exit status 4: Cmd: "/usr/sbin/sgdisk" "--new=1:0:+0" "--change-name=1:STATEFUL" "/dev_aliases/dev/xvdb" Stdout: "Setting name!\npartNum is 0\nREALLY setting name!\n" Stderr: "Could not create partition 1 from 34 to 2047\nError encountered; not saving changes.\n"
systemd[1]: initrd.target: Job initrd.target/start failed with result 'dependency'.
systemd[1]: initrd.target: Triggering OnFailure= dependencies.
systemd[1]: ignition-quench.service: Job ignition-quench.service/start failed with result 'dependency'.
systemd[1]: Dependency failed for Ignition (files).

Reproduction Steps

  1. Run Ignition to create the initial disk configuration with the floating EBS volume to be reused across servers
  2. Destroy original server and reattach volume to new servers
  3. Run Ignition on the new server

Other Information

I have been able to get the server to boot correctly by removing everything in the storage.disks declaration, but am hoping to avoid this because it adds extra manual steps after provisioning once.

Here's basically what the configuration looks like.

{
    "ignition": {
      "config": {},
      "timeouts": {},
      "version": "2.1.0"
    },
    "networkd": {},
    "passwd": {},
    "storage": {
      "directories": [
        {
          "filesystem": "data",
          "group": {
            "id": 500
          },
          "path": "/path1",
          "user": {
            "id": 500
          },
          "mode": 511
        },
        {
          "filesystem": "data",
          "group": {
            "id": 500
          },
          "path": "/path2",
          "user": {
            "id": 500
          }
        }
      ],
      "disks": [
        {
          "device": "/dev/xvdb",
          "wipeTable": false,
          "partitions": [
            {
              "label": "STATEFUL",
              "number": 1
            }
          ]
        }
      ],
      "filesystems": [
        {
          "mount": {
            "device": "/dev/xvdb1",
            "format": "ext4",
            "wipeFilesystem": false,
            "label": "STATEFUL"
          },
          "name": "data"
        }
      ]
    }
}
@lucab

This comment has been minimized.

Member

lucab commented Mar 26, 2018

I have been able to get the server to boot correctly by removing everything in the storage.disks declaration, but am hoping to avoid this because it adds extra manual steps after provisioning once.

I think this is the expected current behavior, but it's probably under-documented.
wipeTable is specifically related to the disk table; if your configuration specifies partition entries, those will be created all the times.

I'm not sure if @dgonyeo or @ajeddeloh have any plans to change this behavior.

@jmreicha

This comment has been minimized.

jmreicha commented Mar 26, 2018

@lucab Thanks for the response. Do you know then if there's any way to do what I want? Maybe something with Systemd?

@ajeddeloh

This comment has been minimized.

ajeddeloh commented Mar 26, 2018

Yup, working on this now. The short explanation is we want to add the filesystem reuse symantics to the partitioning code on a partition by partition level, so you'll be able to say "create this if it doesn't exist and leave it alone if it does".

In the mean time, since it looks like you're not using it for the root partition, you could probably create do it with a systemd oneshot with ConditionFirstBoot with a script that manually implements the logic.

@jmreicha

This comment has been minimized.

jmreicha commented Mar 27, 2018

@ajeddeloh Cool. Any kind of an ETA for the file system reuse stuff?

@ajeddeloh

This comment has been minimized.

ajeddeloh commented Mar 27, 2018

I'm hoping to have it up for review by end of week, maybe early next week. I'm pushing changes to my partition-230 branch if you want to take a peek. The first commit on that branch has the new docs for the feature if you want to get an idea of what it will look like from a user perspective. If you have feedback on it, feel free to comment on the branch.

Most of the implementation work is done, but there's still a lot of cleanup and testing work that needs to be done before we can merge it.

@jmreicha

This comment has been minimized.

jmreicha commented Mar 27, 2018

I will keep an eye on the developments. Should I close this issue or keep open until that branch is released?

@ajeddeloh

This comment has been minimized.

ajeddeloh commented Mar 28, 2018

I thought we had an issue for the partitioning update, but it turns out we don't. I'm renaming this issue and will close it once the change makes it into coreos-overlay.

@ajeddeloh ajeddeloh changed the title from Issues attaching previously formatted/partitioned disk to Ignition should support filesystem reuse semantics for partitioning Mar 28, 2018

@cgwalters

This comment has been minimized.

Member

cgwalters commented May 16, 2018

If you're planning to reuse an EBS volume between different instances, personally I'd partition it "upfront" (perhaps manually, but you could script it). Then have ignition inject a systemd .mount unit that just looks for its well-known UUID. Basically it feels to me like this EBS volume is a "pet", not something you maybe want created dynamically or destroyed. This is just a drive-by comment; it's hard to say without knowing more about your use case though.

@bgilbert

This comment has been minimized.

Member

bgilbert commented May 16, 2018

@cgwalters A slightly cleaner approach is a service unit that conditionally formats the disk. Those sorts of workarounds often point to missing functionality in Ignition, though. Ignition should be able to provision the machine in entirety, not just be a piece of the provisioning workflow.

@ajeddeloh

This comment has been minimized.

ajeddeloh commented May 16, 2018

Adding on to what @bgilbert said, we also need to consider baremetal cases, including PXE cases where Ignition runs every boot. In those cases we need a way of allowing Ignition to partition something to be used as persistent storage but not scramble the guids every boot.

@bgilbert

This comment has been minimized.

Member

bgilbert commented Jun 16, 2018

Done in Ignition 0.26.0, which will be in alpha 1814.0.0.

@bgilbert bgilbert closed this Jun 16, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment