Error using coreos-metadata.service file in 2765.2.0

[obourdon]

Description

There seems to be a strange side-effect upgrading our Flatcar based Packer images for AWS from 2605.12.0 to 2765.2.0

We have some rspec script which validates our final image but this now fails with the following error:

     ×  Command: `sudo systemd-analyze verify /usr/lib/systemd/system/coreos-metadata.service` exit_status is expected to eq 0
     
     expected: 0
          got: 1
     
     (compared using ==)

This is very strange as nothing seems to have really changed for the script /usr/lib/systemd/system/coreos-metadata.service between those 2 releases. Installing from ISO and checking the command by hand gives:

# systemd-analyze verify /usr/lib/systemd/system/coreos-metadata.service
/run/systemd/system/docker.socket:8: ListenStream= references a path below legacy directory /var/run/, updating /var/run/docker.sock → /run/docker.sock; please update the unit file accordingly.
/usr/lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
coreos-metadata.service: Command ln is not executable: No such file or directory

and return code is indeed 1 and not 0 as expected. Changing the line:

ExecStartPost=ln -fs /run/metadata/flatcar /run/metadata/coreos

into

ExecStartPost=/usr/bin/ln -fs /run/metadata/flatcar /run/metadata/coreos

seems to make the problem disapear, however as this line is exactly the same between the 2 releases
it should also have failed in 2605.12.0

Impact

Not able to validate our Packer images based on Flatcar anymore

Environment and steps to reproduce

See all above

Expected behavior

same behaviour expected between 2605.12.0 and 2765.2.0 (either both failed or both succeed depending on fullpath of ln command usage or not)

Additional information

None

[margamanterola]

Hi, thanks for reporting.

I've created a PR to address this issue (and fix the containerd unit that had the same problem). This should be fixed in the next set of releases, which will likely happen later this week.

Regarding why this is failing now and not before, the systemd version changed from 246 to 247. I quickly scanned the systemd changelog and I couldn't find something about this, but I guess that the way the verification works changed in between and while the unit still runs successfully, the verification no longer passes.

[obourdon]

@marga-kinvolk many thanks for this quick response, really appreciated

strangely enough I remember working on another issue (not at all related with this one) on our side early february when upgrading to Flatcar 2605.12.0 where systemd was already upgraded to 247 but seems that also strangely it reverted to 246. I must admit that this gets a lot of confusion in my head but I am pretty sure working 2 full days on this ...

On another side, may be the default PATH has been changed or is not taken into account anymore which could explain this, but could not find any info confirming this

As far as the other PR you are mentioning above, could you please share link. Many thanks again

[obourdon]

My bad, looking deeper into my story of changes on our side, the systemd was indeed 246 but we were migrating from 2605.8.0 where this was 245. Sorry for the confusion, and hopefully, my brain is not that deficient in the end

[margamanterola]

GitHub automatically linked the PR when I mentioned this bug, it's right above my comment. In any case, this is the link: flatcar-archive/coreos-overlay#888

[obourdon]

thanks for the link and PR. I was looking inside the body of the comments and not a the top of the description of my issue therefore not seing the link which was already there as you mentioned. Many thanks again

[margamanterola]

The fix has been cherry picked and will be present in the next stable release.