Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Installing ubuntu-desktop with Packer breaks the VM #1612

Closed
mikkoc opened this issue Aug 22, 2019 · 4 comments
Closed

[BUG] Installing ubuntu-desktop with Packer breaks the VM #1612

mikkoc opened this issue Aug 22, 2019 · 4 comments
Assignees
Labels

Comments

@mikkoc
Copy link

mikkoc commented Aug 22, 2019

Describe the bug
Installing ubuntu-desktop on 18.04 LTS with Packer, creates a broken image that is not able to boot. I'm not sure this is an issue with WALinuxAgent / Cloud-Init / Azure.

It seems like network interface eth0 is unable to go UP, and everything else breaks.

Packer file:

{
  "builders": [
    {
      "type": "azure-arm",
      "subscription_id": "xxx",
      "managed_image_resource_group_name": "temp",
      "managed_image_name": "temp-test-{{timestamp}}",
      "managed_image_storage_account_type": "Premium_LRS",
      "shared_image_gallery_destination": {
        "resource_group": "temp-test",
        "gallery_name": "temp-test",
        "image_name": "ubuntu-bionic-18.04-amd64-server",
        "image_version": "0.0.1",
        "replication_regions": [
          "northeurope"
        ]
      },
      "os_type": "Linux",
      "image_publisher": "Canonical",
      "image_offer": "UbuntuServer",
      "image_sku": "18.04-LTS",
      "location": "northeurope",
      "vm_size": "Standard_DS2_v2",
      "virtual_network_name": "tf-vnet-test",
      "virtual_network_resource_group_name": "temp-test-01",
      "virtual_network_subnet_name": "tf-subnet-test"
    }
  ],
  "provisioners": [
    {
      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh '{{ .Path }}'",
      "inline": [
        "apt-get update -y",
        "apt install ubuntu-desktop -y",
        "waagent -version",
        "/usr/sbin/waagent -force -deprovision+user && export HISTSIZE=0 && sync"
      ],
      "inline_shebang": "/bin/sh -x",
      "type": "shell"
    }
  ]
}

Distro and WALinuxAgent details (please complete the following information):

  • Distro and Version: [e.g. Ubuntu 18.04]
  • WALinuxAgent version:
    azure-arm: WALinuxAgent-2.2.40 running on ubuntu 18.04
    azure-arm: Python: 3.6.8
    azure-arm: Goal state agent: 2.2.42

Additional context
Booting a VM created from that image:


Welcome to Ubuntu 18.04.3 LTS!

[   12.135114] systemd[1]: Set hostname to <localhost.localdomain>.
[   15.212889] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[   15.230535] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[  OK  ] Set up automount Arbitrary Executab…rmats File System Automount Point.
[   15.247519] systemd[1]: Created slice User and Session Slice.
[  OK  ] Created slice User and Session Slice.
[   15.260195] systemd[1]: Created slice System Slice.
[  OK  ] Created slice System Slice.
[   15.269390] systemd[1]: Listening on udev Control Socket.
[  OK  ] Listening on udev Control Socket.
[   15.280795] systemd[1]: Created slice system-systemd\x2dfsck.slice.
[  OK  ] Created slice system-systemd\x2dfsck.slice.
[  OK  ] Listening on fsck to fsckd communication Socket.
[  OK  ] Reached target Slices.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
         Mounting POSIX Message Queue File System...
         Starting Remount Root and Kernel File Systems...
         Starting Set the console keyboard layout...
         Mounting Huge Pages File System...
         Starting Load Kernel Modules...
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Reached target Swap.
         [   15.380044] EXT4-fs (sda1): re-mounted. Opts: discard
Mounting Kernel Debug File System...
[  OK  ] Listening on udev Kernel Socket.
         Starting udev Coldplug all Devices...
[  OK  ] Listening on Journal Audit Socket.
[  OK  ] Listening on Network Service Netlink Socket.
         Starting Create list of required st…ce nodes for the current kernel...
[  OK  ] Listening on LVM2 poll daemon socket.
[  OK  ] Created slice system-serial\x2dgetty.slice.
[  OK  ] Listening on Syslog Socket.
         Starting Journal Service...
         Starting Uncomplicated firewall...
[  OK  ] Reached target User and Group Name Lookups.
[  OK  ] Listening on LVM2 metadata daemon socket.
[  OK  ] Listening on Device-mapper event daemon FIFOs.
         Starting Monitoring of LVM2 mirrors…ng dmeventd or progress polling...
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Started Remount Root and Kernel File Systems.[   15.530554] lp: driver loaded but no devices found

[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted Kernel Debug File System.
[  OK  ] Started Create list of required sta…vice nodes for the current kernel.
[  OK  ] Started Uncomplicated firewall.
         Starting Create Static Device Nodes in /dev...
[   15.592207] ppdev: user-space parallel port driver
[  OK  ] Started Hyper-V KVP Protocol Daemon.
         Starting Load/Save Random Seed...
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Load/Save Random Seed.
[   15.705717] Loading iSCSI transport class v2.0-870.
[   15.712121] systemd-journald[497]: Received request to flush runtime journal from PID 1
[  OK  ] Started udev Coldplug all Devices.[   15.786850] iscsi: registered transport (tcp)

[  OK  ] Started Set the console keyboard layout.
[  OK  ] Started Create Static Device Nodes in /dev.
         Starting udev Kernel Device Manager...
[  OK  ] Started LVM2 metadata daemon.
[   16.613311] iscsi: registered transport (iser)
[  OK  ] Started Load Kernel Modules.
         Mounting Kernel Configuration File System...
         Starting Apply Kernel Variables...
         Mounting FUSE Control File System...
[  OK  ] Started Monitoring of LVM2 mirrors,…sing dmeventd or progress polling.
[  OK  ] Started Flush Journal to Persistent Storage.
[  OK  ] Mounted Kernel Configuration File System.
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Started Apply Kernel Variables.
[  OK  ] Started udev Kernel Device Manager.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Found device /dev/ttyS0.
[  OK  ] Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
[  OK  ] Found device Virtual_Disk UEFI.
[  OK  ] Found device Virtual_Disk Temporary_Storage.
         Starting File System Check on /dev/disk/cloud/azure_resource-part1...
         Mounting /boot/efi...
[  OK  ] Started File System Check on /dev/disk/cloud/azure_resource-part1.
[  OK  ] Mounted /boot/efi.
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
         Starting Set console font and keymap...
         Starting Clean up any mess left by 0dns-up...
         Starting ebtables ruleset management...
         Starting AppArmor initialization...
         Starting Tell Plymouth To Write Out Runtime Data...
[  OK  ] Started Set console font and keymap.
[  OK  ] Started Tell Plymouth To Write Out Runtime Data.
[  OK  ] Started ebtables ruleset management.
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
         Starting Network Time Synchronization...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Started Clean up any mess left by 0dns-up.
[  OK  ] Started Network Time Synchronization.
[  OK  ] Reached target System Time Synchronized.
[  OK  ] Started AppArmor initialization.
         Starting Initial cloud-init job (pre-networking)...
[   23.376523] sh[903]: + [ -e /var/lib/cloud/instance/obj.pkl ]
[   23.387079] sh[903]: + echo cleaning persistent cloud-init object
[   23.399580] sh[903]: cleaning persistent cloud-init object
[   23.411571] sh[903]: + rm /var/lib/cloud/instance/obj.pkl
[   23.432076] sh[903]: + exit 0
[   39.650829] cloud-init[915]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'init-local' at Thu, 22 Aug 2019 15:36:41+0000. Up 35.15 seconds.
[  OK  ] Started Initial cloud-init job (pre-networking).
[  OK  ] Reached target Network (Pre).
         Starting Raise network interfaces...
         Starting Network Service...
[  OK  ] Started Raise network interfaces.
[  OK  ] Started Network Service.
         Starting Wait for Network to be Configured...
         Starting Network Name Resolution...
[  OK  ] Started Wait for Network to be Configured.
         Starting Initial cloud-init job (metadata service crawler)...
[  OK  ] Started Network Name Resolution.
[  OK  ] Reached target Host and Network Name Lookups.
[   41.361679] cloud-init[1088]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'init' at Thu, 22 Aug 2019 15:36:47 +0000. Up 40.95 seconds.
[   41.381469] cloud-init[1088]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
[   41.390282] cloud-init[1088]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[   41.397905] cloud-init[1088]: ci-info: | Device |   Up  |  Address  |    Mask   | Scope |     Hw-Address    |
[   41.405445] cloud-init[1088]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[   41.414044] cloud-init[1088]: ci-info: |  eth0  | False |     .     |     .     |   .   | 00:0d:3a:b6:04:20 |
[   41.422715] cloud-init[1088]: ci-info: |   lo   |  True | 127.0.0.1 | 255.0.0.0 |  host |         .         |
[   41.430874] cloud-init[1088]: ci-info: |   lo   |  True |  ::1/128  |     .     |  host |         .         |
[   41.438724] cloud-init[1088]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[   41.447037] cloud-init[1088]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[   41.454499] cloud-init[1088]: ci-info: +-------+-------------+---------+-----------+-------+
[   41.461273] cloud-init[1088]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[   41.467841] cloud-init[1088]: ci-info: +-------+-------------+---------+-----------+-------+
[   41.474507] cloud-init[1088]: ci-info: +-------+-------------+---------+-----------+-------+
[   41.481832] cloud-init[1088]: 2019-08-22 15:36:47,905 - azure.py[ERROR]: Failed to read /var/lib/dhcp/dhclient.eth0.leases: [Errno 2] No such file or directory: '/var/lib/dhcp/dhclient.eth0.leases'
[   41.493706] cloud-init[1088]: 2019-08-22 15:36:47,919 - azure.py[WARNING]: No lease found; using default endpoint
[  207.088889] cloud-init[1088]: 2019-08-22 15:39:33,323 - DataSourceAzure.py[WARNING]: Error communicating with Azure fabric; You may experience.connectivity issues.
[  207.102122] cloud-init[1088]: Traceback (most recent call last):
[  207.108037] cloud-init[1088]:   File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceAzure.py", line 668, in _negotiate
[  207.119921] cloud-init[1088]:     fabric_data = metadata_func()
[  207.133184] cloud-init[1088]:   File "/usr/lib/python3/dist-packages/cloudinit/sources/helpers/azure.py", line 39, in impl
[  207.160672] cloud-init[1088]:     return func(*args, **kwargs)
[  207.171589] cloud-init[1088]:   File "/usr/lib/python3/dist-packages/cloudinit/sources/helpers/azure.py", line 466, in get_metadata_from_fabric
[  207.191615] cloud-init[1088]:     return shim.register_with_azure_and_fetch_data(pubkey_info=pubkey_info)
[  207.208575] cloud-init[1088]:   File "/usr/lib/python3/dist-packages/cloudinit/sources/helpers/azure.py", line 39, in impl
[  207.218418] cloud-init[1088]:     return func(*args, **kwargs)
[  207.224445] cloud-init[1088]:   File "/usr/lib/python3/dist-packages/cloudinit/sources/helpers/azure.py", line 401, in register_with_azure_and_fetch_data
[  207.236279] cloud-init[1088]:     'http://{0}/machine/?comp=goalstate'.format(self.endpoint))
[  207.244197] cloud-init[1088]:   File "/usr/lib/python3/dist-packages/cloudinit/sources/helpers/azure.py", line 79, in get
[  207.254458] cloud-init[1088]:     return url_helper.read_file_or_url(url, headers=headers)
[  207.272256] cloud-init[1088]:   File "/usr/lib/python3/dist-packages/cloudinit/url_helper.py", line 107, in read_file_or_url
[  207.291144] cloud-init[1088]:     exception_cb=exception_cb)
[  207.312184] cloud-init[1088]:   File "/usr/lib/python3/dist-packages/cloudinit/url_helper.py", line 306, in readurl
[  207.323619] cloud-init[1088]:     raise excps[-1]
[  207.329153] cloud-init[1088]: cloudinit.url_helper.UrlError: HTTPConnectionPool(host='168.63.129.16', port=80): Max retries exceeded with url: /machine/?comp=goalstate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f349fa7df28>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
[  OK  ] Stopped File System Check on /dev/disk/cloud/azure_resource-part1.
         Starting Tell Plymouth To Write Out Runtime Data...
         Starting File System Check on /dev/disk/cloud/azure_resource-part1...
[  OK  ] Started Tell Plymouth To Write Out Runtime Data.
[  OK  ] Started File System Check on /dev/disk/cloud/azure_resource-part1.
[  216.919814] cloud-init[1088]: Generating public/private rsa key pair.
[  216.932689] cloud-init[1088]: Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
[  216.951648] cloud-init[1088]: Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
[  216.960411] cloud-init[1088]: The key fingerprint is:
[  216.971723] cloud-init[1088]: SHA256:9o+qtwCS5P5jERSXuRH55ytT2kG68+XxqmzlkRI4FeQ root@mikko-test002
[  216.988716] cloud-init[1088]: The key's randomart image is:
[  216.998219] cloud-init[1088]: +---[RSA 2048]----+
[  217.011644] cloud-init[1088]: |    ..o= .o.     |
[  217.020636] cloud-init[1088]: |    ..=  ..      |
[  217.029859] cloud-init[1088]: |  ..   + oE      |
[  217.039040] [  OK  ] cloud-init[1088]: | o .. . + +      |Started Initial cloud-init job (metadata service crawler).

[  OK  ] Reached target System Initialization.
[  217.065412] cloud-init[1088]: |  + ..  S* . .   |
[  217.075656] cloud-init[1088]: | . ... ...= +    |
[  217.084539] cloud-init[1088]: |  .  ..  =.*o.   |
[  217.095391] cloud-init[1088]: |   .o  .B.+=.o   |
[  217.103613] cloud-init[1088]: |   ....ooO=.+..  |
[  OK  ] Started CUPS Scheduler.
[[  217.120278] cloud-init[1088]: +----[SHA256]-----+  OK  ] Listening on Open-iSCSI iscsid Socket.
[  OK  ]
[  217.136519] cloud-init[1088]: Generating public/private dsa key pair.Started ACPI Events Check.
[  OK  ] Reached target Paths.

[  OK  ] Listening on UUID daemon activation socket.[  217.157215] cloud-init[1088]: Your identification has been saved in /etc/ssh/ssh_host_dsa_key.
[  OK  ] Listening on CUPS Scheduler.
[  217.169623] cloud-init
[1088]: Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub.[  OK  ] Started Daily apt download activities.
[  217.181258] cloud-init[1088]:
[  OK  ] Started Daily Cleanup of Temporary Directories.The key fingerprint is:
[  217.191647] cloud-init[1088]: SHA256:Vq/BUd+Zytsq6EXkTFTnnQyQO33B4BMysO7dlhZOsC4 root@mikko-test002
[  OK  ] Started Discard unused blocks once a week.

[  217.204527] cloud-init[1088]: The key's randomart image is:[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK
[  217.217540] cloud-init[1088]: ] Listening on ACPID Listen Socket.
+---[DSA 1024]----+
[  217.226410]          Starting LXD - unix socket.
cloud-init[1088]: |         ..===o. |
[  OK  ] Started Trigger anacron every hour.
[  OK  [  217.248521] cloud-init[1088]: |          oo+.*+=|] Started Daily apt upgrade and clean activities.
         Starting Socket activation for snappy daemon.
[  217.266762]
[  OK  ] cloud-init[1088]: Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
|         .o+oo.*+|
[  OK  ] Started Message of the Day.
[  217.289485] cloud-init[1088]: [  OK  ] Reached target Timers.|        .o==+.o. |

[  OK  [  217.303880] cloud-init[1088]: |        S.o=o=.  |] Reached target Cloud-config availability.

[  217.313002] cloud-init[1088]: [  OK  ] Listening on LXD - unix socket.|       .. +o+ =  |
[  217.322790]
[  OK  ] cloud-init[1088]: |         Eo+ B . |
Listening on Socket activation for snappy daemon.
[  217.335327] cloud-init[1088]: |         .o.o .  |[  OK  ] Reached target Sockets.
[  217.345624] cloud-init
[  OK  ] Reached target Basic System.[1088]: |        ..  ..   |

[  217.356652] cloud-init[1088]:          Starting Avahi mDNS/DNS-SD Stack...+----[SHA256]-----+
[  217.368770] cloud-init[1088]:
[  OK  ] Started FUSE filesystem for LXC.Generating public/private ecdsa key pair.
[  217.387054]
         Starting LXD - container startup/shutdown...
[  OK  ] Started Hyper-V VSS Protocol Daemon.cloud-init[1088]: Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key.
[  OK  ] Started ACPI event daemon.
         Starting Modem Manager...

[  217.443770] cloud-init[1088]: Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub.         Starting System Logging Service...
         Starting LSB: Record successful boot for GRUB...

[  217.490225] cloud-init[1088]: The key fingerprint is:
[  217.499632] cloud-init[1088]: SHA256:7GVgdeRxEa2ASrd7DUVFGiI5g4xSQqdJaS+szwu+Vu0 root@mikko-test002
[  217.519681] cloud-init[1088]: The key's randomart image is:
[  217.535798] cloud-init[1088]: +---[ECDSA 256]---+
[  217.548779] cloud-init[1088]: |   .+ooo .o+=.**o|
[  217.559641] cloud-init[1088]: |   .+=. +.*+.+.o.|
[  217.571356] cloud-init[1088]: |   ooo .oo +.o.. |
[  217.581587] cloud-init[1088]: |    o .o... . .  |
[  217.591571] cloud-init[1088]: |   . o  S o. o   |
[  217.601899] cloud-init[1088]:          Starting Snappy daemon...
|  . . .. o. . .  |
[  OK  ] Started Run anacron jobs.
[  217.622690] cloud-init[1088]: |  .+ .  .  .     |[  OK  ] Started Deferred execution scheduler.

         [  217.648293] cloud-init[1088]: | ...o E          |Starting Restore /etc/resolv.conf i…fore the ppp link was shut down...
[  OK  ] Started irqbalance daemon.

[  217.674967] [  OK  ] cloud-init[1088]: Started D-Bus System Message Bus.
| .o...           |
[  217.699051] cloud-init[1088]: +----[SHA256]-----+
[  217.711823] cloud-init[1088]: Generating public/private ed25519 key pair.
[  217.726391] cloud-init[1088]: Your identification has been saved in /etc/ssh/ssh_host_ed25519_key.
[  217.745083] cloud-init[1088]: Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub.
[  217.759396] cloud-init[1088]: The key fingerprint is:
[  217.774177] cloud-init[1088]: SHA256:JBvQDguVfUIZY2Zo1700IJwqvnr1X26W8cna1Dg/va8 root@mikko-test002
[  217.789860] cloud-init[1088]: The key's randomart image is:
[  217.801814] cloud-init[1088]: +--[ED25519 256]--+
[  217.815718] cloud-init[1088]: |   .oBO=.o       |
[  217.825919] cloud-init[1088]: |  . ==Ooo +      |
[  217.840993] cloud-init[1088]: |   o *oo.. o     |
[  217.851274] cloud-init[1088]: |  . o .=  .      |
[  217.863652] cloud-init[1088]: | . .  . S        |
[  217.879607] cloud-init[1088]: |  . .     .  o   |
[  217.888823] cloud-init[1088]: |   o .    .=+... |
[  217.895851] cloud-init[1088]: |  o   .  o+o+o. .|
[  217.902077] cloud-init[1088]: |.o     ..oo.. E++|
[  217.907879] cloud-init[1088]: +----[SHA256]-----+
         Starting WPA supplicant...
         Starting Login Service...
         Starting Accounts Service...
[  OK  ] Started CUPS Scheduler.
         Starting Detect the available GPUs and deal with any system changes...
[  OK  ] Started Hyper-V File Copy Protocol Daemon.
[  OK  ] Started Regular background program processing daemon.
         Starting Dispatcher daemon for systemd-networkd...
         Starting Disk Manager...
[  OK  ] Started System Logging Service.
[  OK  ] Started Restore /etc/resolv.conf if…before the ppp link was shut down.
[  OK  ] Started Avahi mDNS/DNS-SD Stack.
         Starting Authorization Manager...
[  OK  ] Started Make remote CUPS printers available locally.
[  OK  ] Started Detect the available GPUs and deal with any system changes.
[  OK  ] Started Login Service.
[  OK  ] Started LSB: Record successful boot for GRUB.
[  OK  ] Started Authorization Manager.
[  OK  ] Started Accounts Service.
[  OK  ] Started WPA supplicant.
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.
         Starting OpenBSD Secure Shell server...
[  OK  ] Started crash report submission daemon.
[  OK  ] Started Azure Linux Agent.
         Starting Availability of block devices...
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Remote File Systems.
         Starting LSB: Speech Dispatcher...
         Starting LSB: automatic crash report generation...
         Starting Permit User Sessions...
         Starting Tool to automatically coll… submit kernel crash signatures...
[  OK  ] Started Unattended Upgrades Shutdown.
[  OK  ] Started Dispatcher daemon for systemd-networkd.
[  OK  ] Started Availability of block devices.
[  OK  ] Started Permit User Sessions.
         Starting GNOME Display Manager...
         Starting Hold until boot process finishes up...
[  OK  ] Started Hold until boot process finishes up.
[  OK  ] Started LSB: Speech Dispatcher.
[  OK  ] Started Serial Getty on ttyS0.
[  OK  ] Reached target Login Prompts.
         Starting Set console scheme...
[  OK  ] Started LSB: automatic crash report generation.
[  OK  ] Started Tool to automatically colle…nd submit kernel crash signatures.
[  OK  ] Started Set console scheme.
[  OK  ] Created slice system-getty.slice.
[  OK  ] Started OpenBSD Secure Shell server.
[  OK  ] Started Snappy daemon.
         Starting Wait until snapd is fully seeded...
[  OK  ] Started Wait until snapd is fully seeded.
         Starting Apply the settings specified in cloud-config...
[  OK  ] Started GNOME Display Manager.
[  OK  ] Started LXD - container startup/shutdown.
[  OK  ] Started Modem Manager.
[  OK  ] Started Disk Manager.
2019/08/22 15:39:50.349356 INFO Daemon Azure Linux Agent Version:2.2.40
2019/08/22 15:39:50.357905 INFO Daemon OS: ubuntu 18.04
2019/08/22 15:39:50.361875 INFO Daemon Python: 3.6.8
2019/08/22 15:39:50.374345 INFO Daemon Daemon process id 1472 is tracked in systemd cgroup walinuxagent.service
2019/08/22 15:39:50.382360 INFO Daemon CGroups: successfully set up agent cgroup
2019/08/22 15:39:50.388640 INFO Daemon Run daemon
2019/08/22 15:39:50.392988 INFO Daemon Clean protocol
2019/08/22 15:39:50.399400 INFO Daemon Running CloudInit provisioning handler
2019/08/22 15:39:50.412053 INFO Daemon Detect protocol endpoints
2019/08/22 15:39:50.424927 INFO Daemon Clean protocol
2019/08/22 15:39:50.431279 INFO Daemon WireServer endpoint is not found. Rerun dhcp handler
2019/08/22 15:39:50.437344 INFO Daemon Test for route to 168.63.129.16
2019/08/22 15:39:50.444159 WARNING Daemon No route exists to 168.63.129.16
2019/08/22 15:39:50.448997 INFO Daemon Checking for dhcp lease cache
2019/08/22 15:39:50.454175 INFO Daemon looking for leases in path [/var/lib/dhcp/dhclient.*.leases]
2019/08/22 15:39:50.460309 INFO Daemon cached endpoint not found
2019/08/22 15:39:50.466963 INFO Daemon Cache exists [False]
2019/08/22 15:39:50.471559 INFO Daemon Send dhcp request
2019/08/22 15:39:50.475409 INFO Daemon Examine /proc/net/route for primary interface
2019/08/22 15:39:50.480314 ERROR Daemon /proc/net/route contains no routes
2019/08/22 15:39:50.488583 WARNING Daemon Could not determine primary interface, please ensure /proc/net/route is correct
2019/08/22 15:39:50.495551 WARNING Daemon Contents of /proc/net/route:
Iface   Destination     Gateway         Flags   RefCnt  Use     Metric  Mask            MTU     Window  IRTT

2019/08/22 15:39:50.510502 WARNING Daemon Primary interface examination will retry silently
2019/08/22 15:39:52.717454 ERROR Daemon /proc/net/route contains no routes

2019/08/22 15:39:54.731630 ERROR Daemon /proc/net/route contains no routes
Ubuntu 18.04.3 LTS mikko-test002 ttyS0

mikko-test002 login: [  229.153348] cloud-init[1523]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'modules:config' atThu, 22 Aug 2019 15:39:50 +0000. Up 223.53 seconds.
ci-info: no authorized ssh keys fingerprints found for user mikko.
<14>Aug 22 15:39:56 ec2:
<14>Aug 22 15:39:56 ec2: #############################################################
<14>Aug 22 15:39:56 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
<14>Aug 22 15:39:56 ec2: 1024 SHA256:Vq/BUd+Zytsq6EXkTFTnnQyQO33B4BMysO7dlhZOsC4 root@mikko-test002 (DSA)
<14>Aug 22 15:39:56 ec2: 256 SHA256:7GVgdeRxEa2ASrd7DUVFGiI5g4xSQqdJaS+szwu+Vu0 root@mikko-test002 (ECDSA)
<14>Aug 22 15:39:56 ec2: 256 SHA256:JBvQDguVfUIZY2Zo1700IJwqvnr1X26W8cna1Dg/va8 root@mikko-test002 (ED25519)
<14>Aug 22 15:39:56 ec2: 2048 SHA256:9o+qtwCS5P5jERSXuRH55ytT2kG68+XxqmzlkRI4FeQ root@mikko-test002 (RSA)
<14>Aug 22 15:39:56 ec2: -----END SSH HOST KEY FINGERPRINTS-----
<14>Aug 22 15:39:56 ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBR4y3Ianlc9dXl7EyUZdHBolTYJed3mWcy7Tt57wMs6dLRT28xteGxKhrDYFDICb+g4KdfcVvaHEpQLw8xNMEg= root@mikko-test002
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHMTIUvMQwsG/AQiTedtdvJzmrldWQW1oCb/EWhYFhs4 root@mikko-test002
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCezwMCWVzMlp+Epf6gymoE0M0clvgwXbgGLbqjB5bpLCnkvTsj54PwK0GfNbFhAJE+CV4/z8L6RXv+nnECcuN77wDADs28MsZpDhbbzfDoWNnlfY83iL9PlGefr0V/za3MbUjwvmM/4kv/xpxasJWl32rpNqmsu67PuEJz1m9QKA2Oh3QuFm92AT+8dB5MMQfk3Z7GSch3+e014Z350zXjN3mBl2lTolIl6R+j8Th8/0/Xi7xy1K/3N/WBY4F5OJP9Y0ZP5I92i53iN6qF4WWE5vdsFsedOGPWvqJNY5/nSpQgLKB2IqaFP0PslfT3d0Vio+h+7EnT8W7/UqWlTR7/ root@mikko-test002
-----END SSH HOST KEY KEYS-----
2019/08/22 15:39:56.743434 ERROR Daemon /proc/net/route contains no routes
[  230.260191] cloud-init[1726]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'modules:final' at Thu, 22 Aug 2019 15:39:56 +0000. Up 229.87 seconds.
[  230.284276] cloud-init[1726]: ci-info: no authorized ssh keys fingerprints found for user mikko.
[  230.296728] cloud-init[1726]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 finished at Thu, 22 Aug 2019 15:39:56 +0000. Datasource DataSourceAzure [seed=/dev/sr0].  Up 230.24 seconds
2019/08/22 15:39:58.825203 ERROR Daemon /proc/net/route contains no routes
2019/08/22 15:40:00.837439 ERROR Daemon /proc/net/route contains no routes
2019/08/22 15:40:02.849542 ERROR Daemon /proc/net/route contains no routes
2019/08/22 15:40:04.862518 ERROR Daemon /proc/net/route contains no routes
2019/08/22 15:40:06.875307 ERROR Daemon /proc/net/route contains no routes
2019/08/22 15:40:08.890871 ERROR Daemon /proc/net/route contains no routes
@mikkoc mikkoc added the triage Needs Triaging label Aug 22, 2019
@vrdmr vrdmr added triaged and removed triage Needs Triaging labels Sep 26, 2019
@yinghli
Copy link

yinghli commented Oct 30, 2019

I got the same error message when Ubuntu 18.04 boot up in VNET that enable IPv6.
My workaround is access serial console from portal and run:
dhclient -4
dhclient -6

@agjohnson
Copy link

To narrow this report down even further, the Ubuntu package ifupdown is the underlying culprit. I get the same console errors from cloud-init above using the packer configuration below.

It's not clear to me if this is a bug with waagent, cloud-init, or should be resolved in ifupdown. The cause seems to be that ifupdown installs some sysv/systemd files that conflict with whatever Azure is expecting the instance to do on boot. We build images on AWS and Azure, and there are no issues with the AWS images on Ubuntu 18.04.

I'll let package maintainers here or on cloud-init speculate what the best fix is, but as a user, I can say I dived really deep tracking this down and never would have guessed ifupdown was the problem. The only way I was able to track this down was to diff /etc/ at each Packer provisioning step, as I stepped through image creation in debug mode. I noticed the sysv/systemd files for networking were created, and the console log line Started Raise network interfaces. was missing from the console output of an instance created from a known good image.

I suspect that if you install ifupdown and try imaging your instance, your image will not be bootable by Azure. So, fallout from this bug could be wider than just users building new images -- though I haven't confirmed this.

A potential fix

As part of your image clean up steps, you could uninstall ifupdown if you don't need it (this was brought in by salt-minion in my case, and wasn't needed anymore):

    ...
    {
      "type": "shell",
      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo {{ .Path }}",
      "inline": [
        "apt-get remove -yq ifupdown",
        "apt-get autoremove -yq"
      ]
    },
    ...

Packer config

This config will reproduce the bug if you add the necessary environment variables for your subscription. The resulting image cannot be used to boot an image -- the NIC will never come.

{
  "variables": {
    "instance_type": "Standard_B1ms",
    "resource_group": "{{ env `AZURE_RESOURCE_GROUP` }}",
    "location": "{{ env `AZURE_LOCATION` }}",
    "client_id": "{{ env `ARM_CLIENT_ID` }}",
    "client_secret": "{{ env `ARM_CLIENT_SECRET` }}",
    "subscription_id": "{{ env `ARM_SUBSCRIPTION_ID` }}",
    "tenant_id": "{{ env `ARM_TENANT_ID` }}"
  },
  "builders": [
    {
      "type": "azure-arm",

      "client_id": "{{ user `client_id` }}",
      "client_secret": "{{ user `client_secret` }}",
      "subscription_id": "{{ user `subscription_id` }}",
      "tenant_id": "{{ user `tenant_id` }}",

      "os_type": "Linux",
      "os_disk_size_gb": 80,
      "image_publisher": "Canonical",
      "image_offer": "UbuntuServer",
      "image_sku": "18.04-LTS",

      "managed_image_name": "test-failure-{{ isotime \"2006-01-02-1504\" }}",
      "managed_image_resource_group_name": "{{ user `resource_group` }}",

      "location": "West US 2",
      "vm_size": "{{ user `instance_type` }}"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo {{ .Path }}",
      "inline": [
          "set -x",
          "/usr/bin/cloud-init status --wait",
          "systemctl stop apt-daily.timer",
          "systemctl stop apt-daily.service",
          "systemctl stop apt-daily-upgrade.timer",
          "systemctl stop apt-daily-upgrade.service",
          "systemctl stop unattended-upgrades.service",
          "apt-get install -yq ifupdown",
          "/usr/sbin/waagent -force -deprovision+user && export HISTSIZE=0 && sync"
      ]
    }
  ]
}

@narrieta narrieta assigned trstringer and unassigned narrieta Mar 10, 2020
@agjohnson
Copy link

agjohnson commented Mar 11, 2020

Update: I hit an identical bug on AWS yesterday as well, so seems like this is a conflict between ifupdown package and cloud-init? AWS behaves differently than Azure on boot, and the AWS instance was able to start, but Packer timed out consistently waiting for an SSH connection afterwards. I built an image that had ifupdown in it first, then tried to use that image as a base in another Packer build, which is where I hit the SSH timeout.

I'll open an issue against cloud-init

Edit: https://bugs.launchpad.net/cloud-init/+bug/1867029

@pgombar
Copy link
Contributor

pgombar commented Jul 13, 2020

Tracked in https://bugs.launchpad.net/cloud-init/+bug/1867029 and seems to be fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

8 participants