Flatcar on EC2 sets a incorrect os hostname #707

jkroepke · 2022-04-06T10:49:59Z

Description

By default, on flatcar (Alpha?), the hostname contains dots and systemd-networkd is not set hostname received through DHCP.

Readings the docs at https://www.freedesktop.org/software/systemd/man/hostnamectl.html, a system hostname should not contain any dots and should be a single dns label.

Also Debian follows this pattern.

On Flatcar, the /etc/hostname is populated by coreos-metadata. For AWS, the change is introduced in flatcar/bootengine#21, but also mention that systemd-hostname will set the hostname, which does not happen.

I also dump a DHCP traffic which confirms that the hostname is delivered by DHCP.

tcpdump DHCP traffic

# tcpdump -i eth0 -pvn port 67 and port 68
dropped privs to tcpdump
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:25:00.866515 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 0a:5c:80:ef:c6:c0, length 300, xid 0x9ea9fc3e, Flags [none]
    Client-Ethernet-Address 0a:5c:80:ef:c6:c0
    Vendor-rfc1048 Extensions
      Magic Cookie 0x63825363
      DHCP-Message Option 53, length 1: Discover
      Parameter-Request Option 55, length 13:
        Subnet-Mask, BR, Time-Zone, Classless-Static-Route
        Domain-Name, Domain-Name-Server, Hostname, YD
        YS, NTP, MTU, Option 119
        Default-Gateway
09:25:00.866565 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 576)
    10.110.64.1.67 > 10.110.71.54.68: BOOTP/DHCP, Reply, length 548, xid 0x9ea9fc3e, Flags [none]
    Your-IP 10.110.71.54
    Client-Ethernet-Address 0a:5c:80:ef:c6:c0
    Vendor-rfc1048 Extensions
      Magic Cookie 0x63825363
      DHCP-Message Option 53, length 1: Offer
      Server-ID Option 54, length 4: 10.110.64.1
      Lease-Time Option 51, length 4: 3600
      Subnet-Mask Option 1, length 4: 255.255.224.0
      BR Option 28, length 4: 10.110.95.255
      Domain-Name Option 15, length 29: "eu-central-1.compute.internal"
      Domain-Name-Server Option 6, length 4: 10.110.0.2
      Hostname Option 12, length 19: "i-0b4cb6bd27739f893"
      MTU Option 26, length 2: 9001
      Default-Gateway Option 3, length 4: 10.110.64.1
09:25:00.866818 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 0a:5c:80:ef:c6:c0, length 300, xid 0x9ea9fc3e, Flags [none]
    Client-Ethernet-Address 0a:5c:80:ef:c6:c0
    Vendor-rfc1048 Extensions
      Magic Cookie 0x63825363
      DHCP-Message Option 53, length 1: Request
      Server-ID Option 54, length 4: 10.110.64.1
      Requested-IP Option 50, length 4: 10.110.71.54
      Parameter-Request Option 55, length 13:
        Subnet-Mask, BR, Time-Zone, Classless-Static-Route
        Domain-Name, Domain-Name-Server, Hostname, YD
        YS, NTP, MTU, Option 119
        Default-Gateway
09:25:00.866855 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 576)
    10.110.64.1.67 > 10.110.71.54.68: BOOTP/DHCP, Reply, length 548, xid 0x9ea9fc3e, Flags [none]
    Your-IP 10.110.71.54
    Client-Ethernet-Address 0a:5c:80:ef:c6:c0
    Vendor-rfc1048 Extensions
      Magic Cookie 0x63825363
      DHCP-Message Option 53, length 1: ACK
      Server-ID Option 54, length 4: 10.110.64.1
      Lease-Time Option 51, length 4: 3600
      Subnet-Mask Option 1, length 4: 255.255.224.0
      BR Option 28, length 4: 10.110.95.255
      Domain-Name Option 15, length 29: "eu-central-1.compute.internal"
      Domain-Name-Server Option 6, length 4: 10.110.0.2
      Hostname Option 12, length 19: "i-0b4cb6bd27739f893"
      MTU Option 26, length 2: 9001
      Default-Gateway Option 3, length 4: 10.110.64.1

If systemd-networkd would set the hostname delivered by DHCP, than the hostname would be correctly set.

Impact

After a successful boot, the hostname of the machine contains dots, e.g. is a FQDN hostname.

# hostname -f && hostname && cat /etc/hostname
i-00c64910e1b14a8ef.eu-central-1.compute.internal
i-00c64910e1b14a8ef.eu-central-1.compute.internal
i-00c64910e1b14a8ef.eu-central-1.compute.internal

Since kubelet use this hostname as default node name, kubelet will register nodes also as an FQDN.
Starting with RBN hostname, AWS related tools in the kubernetes environment (e.g. AWS Cloud Provider) expect the short hostname as node name. See: kubernetes/cloud-provider-aws#335

As workaround, I have to set --hostname-override on all kubernetes components, but this should not be the case on a Container optimized linux system.

Environment and steps to reproduce

Set-up: Setup EC2 instance with Flatcar
Task: Boot and login into it.
Action(s): [ sequence of actions that triggered the bug, see example below ]
a. cat /etc/hostname
b. run hostname
Error: Contains FQDN instead a dns single-label hostname

Expected behavior

Taken from an Ubuntu instance:

ubuntu@ip-172-16-0-60:~$ hostname -f && hostname && cat /etc/hostname
ip-172-16-0-60.eu-central-1.compute.internal
ip-172-16-0-60
ip-172-16-0-60

Additional information

I'm running flatcar Alpha

Flatcar Container Linux by Kinvolk alpha 3185.0.0 for Amazon EC2

The upstream coreos afterburn will not set the hostname service, if the platform is AWS. See: https://github.com/coreos/afterburn/blob/main/dracut/30afterburn/afterburn-hostname.service#L4

No clue, if systemd will set a transient hostname, if a static hostname is already defined.

The text was updated successfully, but these errors were encountered:

jkroepke · 2022-04-06T10:54:48Z

I also enable debug log of networkd https://superuser.com/a/1234599

# cat /etc/systemd/system/systemd-networkd.service.d/10-debug.conf
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

I can verify, that transient hostname is still set by systemd-networkd, but since the /etc/hostname is populated through coreos metadata, the transient hostname does not matters.

Apr 06 10:46:37 i-08115ab723fbe0417.eu-central-1.compute.internal systemd-networkd[184971]: Setting transient hostname: 'i-08115ab723fbe0417'

I guess rollback flatcar/bootengine#21 would resolve the issue.

Otherwise, afterburn should correctly set the hostname from IMDS.

jkroepke · 2022-04-06T11:19:55Z

Hi,

running

hostnamectl set-hostname ""
systemctl restart systemd-networkd

Resolve the issue.

# hostname -f && hostname && cat /etc/hostname
i-0f443d994d87d9648.eu-central-1.compute.internal
i-0f443d994d87d9648
cat: /etc/hostname: No such file or directory

This is the current workaround, that I use for now.

systemd:
  units:
    - name: "fix-hostname.service"
      enabled: true
      # https://superuser.com/a/1005804
      contents: |
        [Unit]
        Description=Fix Hostname
        Before=network-pre.target
        Wants=network-pre.target
        
        DefaultDependencies=no
        Requires=local-fs.target
        After=local-fs.target
        
        [Service]
        Type=oneshot
        ExecStart=/usr/bin/rm /etc/hostname
        RemainAfterExit=yes
        
        [Install]
        WantedBy=network.target

tormath1 · 2022-04-06T13:35:06Z

Hi,

Thanks for raising this issue - I think it should be solved with this commit: coreos/afterburn@860db5c which is about to be merged (see: flatcar-archive/coreos-overlay#1769).

The hostname will be truncated to the first dot, so in your case: i-00c64910e1b14a8ef.eu-central-1.compute.internal will become i-00c64910e1b14a8ef.

EDIT: In the meantime, you could maybe try to disable flatcar-metadata-hostname.service from the Ignition config ? Let us know.

jkroepke · 2022-04-06T17:12:02Z

EDIT: In the meantime, you could maybe try to disable flatcar-metadata-hostname.service from the Ignition config ? Let us know.

How I can disable services that runs inside initramfs? I would expect, that ignition configure only unit files in /sysroot, not configure services inside initramfs.

On a bootet system, there is such no service.

$ systemctl status flatcar-metadata-hostname.service
Unit flatcar-metadata-hostname.service could not be found.

The ability to disable flatcar-metadata-hostname.service would be great.

jkroepke · 2022-04-06T19:42:58Z

Hi @tormath1

i looked into coreos/afterburn@860db5c . As I understand, the hostname is truncated only if the FQDN have more than 64 bytes. This is not the case here. Even on us-east-1, since the FQDN does not include the region.

in conclusion, flatcar-archive/coreos-overlay#1769 would not help here, EXCEPT https://github.com/coreos/afterburn/blob/main/dracut/30afterburn/afterburn-hostname.service is used for the flatcar-metadata-hostname service, which is not running on AWS by default.

pothos · 2022-04-12T07:50:50Z

Sounds like we should either exclude ec2 again or throw away the existing shell logic and adapt the new afterburn unit to set the hostname?

The hostname afterburn got from the EC2 metadata was a FQDN and too long. We actually don't need afterburn to set the hostname because it is also set up via DHCP. Also, the upstream afterburn-hostname.service already has conditions to run on the right platforms. We can use these instead of custom shell logic. Align flatcar-metadata-hostname.service with upstream afterburn and don't use it for EC2. Adapt the upstream unit to fit our needs with a different cmdline arg naming and skip the relabling workaround (for now). Fixes flatcar/Flatcar#707

pothos · 2022-04-14T06:50:22Z

PRs ready for review: flatcar/bootengine#43 flatcar-archive/coreos-overlay#1817

jkroepke added the kind/bug Something isn't working label Apr 6, 2022

jkroepke changed the title ~~systemd-networkd is not set hostname received through DHCP and results into an incorrect os hostname~~ Flatcar on EC2 sets a incorrect os hostname Apr 6, 2022

tormath1 mentioned this issue Apr 6, 2022

coreos-base/afterburn: sync with 5.2.0 flatcar-archive/coreos-overlay#1769

Merged

1 task

pothos mentioned this issue Apr 13, 2022

Remove afterburn /etc/hostname setup for EC2 and align it with upstream flatcar/bootengine#43

Merged

1 task

pothos self-assigned this Apr 13, 2022

pothos closed this as completed in flatcar/bootengine#43 Apr 14, 2022

gabriel-samfira mentioned this issue May 9, 2023

Flatcar doesn't boot on OpenStack kubernetes/kops#15385

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flatcar on EC2 sets a incorrect os hostname #707

Flatcar on EC2 sets a incorrect os hostname #707

jkroepke commented Apr 6, 2022

jkroepke commented Apr 6, 2022 •

edited

Loading

jkroepke commented Apr 6, 2022

tormath1 commented Apr 6, 2022 •

edited

Loading

jkroepke commented Apr 6, 2022

jkroepke commented Apr 6, 2022 •

edited

Loading

pothos commented Apr 12, 2022

pothos commented Apr 14, 2022

Flatcar on EC2 sets a incorrect os hostname #707

Flatcar on EC2 sets a incorrect os hostname #707

Comments

jkroepke commented Apr 6, 2022

Description

Impact

Environment and steps to reproduce

Expected behavior

Additional information

jkroepke commented Apr 6, 2022 • edited Loading

jkroepke commented Apr 6, 2022

tormath1 commented Apr 6, 2022 • edited Loading

jkroepke commented Apr 6, 2022

jkroepke commented Apr 6, 2022 • edited Loading

pothos commented Apr 12, 2022

pothos commented Apr 14, 2022

jkroepke commented Apr 6, 2022 •

edited

Loading

tormath1 commented Apr 6, 2022 •

edited

Loading

jkroepke commented Apr 6, 2022 •

edited

Loading