coreos-metadata.service: Caused by: Couldn't find 'coreos.oem.id' flag in cmdline file (/proc/cmdline)

[jforman]

Issue Report

Bug

Container Linux Version

core@coreA11 ~ $ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1590.2.0
VERSION_ID=1590.2.0
BUILD_ID=2017-12-06-0313
PRETTY_NAME="Container Linux by CoreOS 1590.2.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

Hardware: Bare-metal Ubuntu running Libvirt/Kvm with CoreOS as the guest VM

Expected Behavior

coreos-metadata.service agent should start

Actual Behavior

jforman@desktop1:~$ ssh core@10.10.0.226
Container Linux by CoreOS beta (1590.2.0)
Failed Units: 1
coreos-metadata.service

core@coreA11 ~ $ systemctl status coreos-metadata.service
● coreos-metadata.service - CoreOS Metadata Agent
Loaded: loaded (/usr/lib/systemd/system/coreos-metadata.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2017-12-25 14:08:28 UTC; 36s ago
Process: 701 ExecStart=/usr/bin/coreos-metadata ${COREOS_METADATA_OPT_PROVIDER} --attributes=/run/metadata/coreos (code=exited, status=1/FAILURE)
Main PID: 701 (code=exited, status=1/FAILURE)

Dec 25 14:08:28 coreA11 systemd[1]: Starting CoreOS Metadata Agent...
Dec 25 14:08:28 coreA11 systemd[1]: coreos-metadata.service: Main process exited, code=exited, status=1/FAILURE
Dec 25 14:08:28 coreA11 systemd[1]: coreos-metadata.service: Failed with result 'exit-code'.
Dec 25 14:08:28 coreA11 systemd[1]: Failed to start CoreOS Metadata Agent.

core@coreA11 ~ $ cat /usr/lib/systemd/system/coreos-metadata.service
[Unit]
Description=CoreOS Metadata Agent

[Service]
Type=oneshot
Environment=COREOS_METADATA_OPT_PROVIDER=--cmdline
ExecStart=/usr/bin/coreos-metadata ${COREOS_METADATA_OPT_PROVIDER} --attributes=/run/metadata/coreos

[Install]
RequiredBy=metadata.target
core@coreA11 ~ $ /usr/bin/coreos-metadata --cmdline --attributes=/run/metadata/coreos
Error: initialization
Caused by: Couldn't find 'coreos.oem.id' flag in cmdline file (/proc/cmdline)

Reproduction Steps

Reboot the VM. Any login. The service just doesn't start.

Other Information

I used the vagrant-virtualbox config transpiler platform for creating my ignition config.

I plan to try to hack on the coreos-metadata repo to add support for libvirt as a provider (vagrant-virtualbox does not work as a provider since it expects eth1 as the network interface, where as libvirt provides eth0 by default).

[crawford]

I don't expect coreos-metadata to support any platforms without a metadata service. The intention of coreos-metadata is just to consolidate all of the garbage logic needed to fetch metadata from each of the cloud providers and make it accessible to systemd services. In your scenario, a call to ip a will suffice. And, in fact, coreos-metadata can't know which interface corresponds to which role.

You can use something like this example as your service. It fundamentally the same way as coreos-metadata, it just doesn't have any fancy code. I hope that makes sense.

[jforman]

Oh interesting, I see what you mean @crawford. I just need to make sure this oneshot service starts before 'everything' else, yes?

[crawford]

Exactly. coreos-metadata.service is typically explicitly added as a dependency of (Requires=coreos-metadata.service) and ordered before (Before=coreos-metadata.service) each of the services which need its output. This ensures that everything starts lazily (if it's needed at all) and systemd will just take care of the rest.

[lucab]

I missed this initial bug-report, but I've left a similar comment when reviewing the resulting PR: coreos/afterburn#81 (comment). I'd guess this ticket can be closed then.

[nickgrealy]

Hi @crawford,

Apologies up front if these questions are "stupid", I'm just getting up to speed and had a few questions:

---

1) Do I need to manually add a coreos.oem.id=custom flag to the /proc/cmdline?

Manually sounds bad, how should the value be populated normally? There is currently no coreos.oem.id flag at all in my /proc/cmdline.

(I don't understand how a custom metadata agent will fix this issue.)

e.g.

core@localhost ~ $ cat /proc/cmdline
rootflags=rw mount.usrflags=ro BOOT_IMAGE=/coreos/vmlinuz-a mount.usr=/dev/mapper/usr verity.usr=PARTUUID=<SOME_ID> rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200n8 console=tty0 coreos.config.url=oem:///coreos-install.json verity.usrhash=<SOME_HASH>

---

2) Does it matter what filename the custom metadata agent writes to?

Can I call it foobar, as long as it aligns with the EnvironmentFile=/run/metadata/foobar?

---

3) What triggers the use of coreos-metadata?

Is it the presence of a variable in the etcd section, or in the systemd.units section, or both? (when I remove all variables from my Ignition config, I don't get this issue).

ignition.yml

# setup shared config
etcd:
  version:                     3.1.6
  name:                        "{HOSTNAME}"
  advertise_client_urls:       http://{PRIVATE_IPV4}:2375,http://{PRIVATE_IPV4}:9000
  discovery:                   https://discovery.etcd.io/<token>

...

systemd:
  units:
    # install portainer
    - name: portainer.service
      enable: true
      contents: |
        [Unit]
        Description=Portainer.io
        After=docker.service
        Requires=docker.service

        [Service]
        TimeoutStartSec=0
        ExecStartPre=-/usr/bin/docker stop %n
        ExecStartPre=-/usr/bin/docker rm %n
        ExecStartPre=/usr/bin/docker volume create portainer_data
        ExecStartPre=/usr/bin/docker pull portainer/portainer
        ExecStart=/usr/bin/docker run --rm --name %n -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer
        ExecStartPost=/usr/bin/etcdctl set /domains/server.com/{PRIVATE_IPV4}:9000 running
        ExecStop=/usr/bin/docker stop %n
        ExecStopPost=/usr/bin/etcdctl rm /domains/server.com/{PRIVATE_IPV4}:9000

        [Install]
        WantedBy=multi-user.target

Thanks

UPDATE: I've just read this document on setting up a custom metadata provider, it's a much better resource than this article, and (if it works) answers my questions.

(If it works...) Is there any chance we can update the coreos.com docs?

[lucab]

@nickgrealy trying to answer most of your questions here, but if you keep having doubts on your journey please prefer the ML for discussion.

1. No, you don't need to set coreos.oem.id manually. if you are on a plain bare-metal node (i.e. without any external metadata or cloud-service service), it is correct for that to be missing/empty.

2. It matters if you plan to use other tools/units of ours (e.g. ct) which expects our standard path. This seems to be your case.

3. When processing your YAML configuration, templated variables are inspected and coreos-metadata dependencies (After+Requires) are added to the resulting units. I think you may have some confusion here, because the config you pasted is not the final ignition config. This is just the input for container-linux-config-transpiler (i.e. ct), but the final result will be a JSON config file for ignition with those additional details and dependencies.

4. it is hard for us to strike a good balance in user doc details, so we'll gladly take any hints from your side to iron out that article. The source for that article is here, and you can just freely send a PR with your suggestions.