IoTEdge Service fails to start on Windows 10 after docker update to 18.06.0-ce

[MPapst]

After Update to Docker 18.06.0-ce the IoTEdge service fails to start.

Expected Behavior

Start-Service iotedge should start the iotedge service.

Current Behavior

The Service fails to start

PS C:\WINDOWS\system32> start-service iotedge
start-service : Failed to start service 'iotedge (iotedge)'.

Context (Environment)

Device (Host) Operating System

Windows 10 Enterprise 1803

Architecture

amd64

Container Operating System

Windows

Runtime Versions

iotedged

iotedge 1.0.0 (52ef77db24126bf473265fc09c53d35290a2dd6b)

Edge Agent

mcr.microsoft.com/azureiotedge-agent 1.0 101f624f0ba3 4 weeks ago 441MB

Edge Hub

mcr.microsoft.com/azureiotedge-hub 1.0 061c26928cd5 4 weeks ago 454MB

Docker

PS C:\WINDOWS\system32> docker version
Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:05:28 2018
 OS/Arch:           windows/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.24)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:23:19 2018
  OS/Arch:          windows/amd64
  Experimental:     false

Logs

PS C:\WINDOWS\system32> Get-WinEvent -ea SilentlyContinue `
>> -FilterHashtable @{ProviderName= "iotedged";
>>   LogName = "application"; StartTime = [datetime]::Now.AddMinutes(-5.0)} |
>> select TimeCreated, Message |
>> sort-object @{Expression="TimeCreated";Descending=$false}

TimeCreated          Message
-----------          -------
7/26/2018 7:50:24 AM info: iotedged::app -- Using config file: C:\ProgramData\iotedge\config.yaml
7/26/2018 7:50:24 AM info: iotedged::app -- Version - 1.0.0 (52ef77db24126bf473265fc09c53d35290a2dd6b)
7/26/2018 7:50:24 AM info: iotedged::app -- Starting Azure IoT Edge Security Daemon
7/26/2018 7:50:24 AM info: iotedged -- Initializing the module runtime...
7/26/2018 7:50:24 AM info: iotedged -- Using runtime network id nat
7/26/2018 7:50:24 AM info: iotedged::windows -- Starting iotedged service.
7/26/2018 7:50:24 AM info: iotedged -- Configuring C:\ProgramData\iotedge as the home directory.
7/26/2018 7:50:24 AM info: iotedged -- Finished initializing the module runtime.
7/26/2018 7:50:24 AM info: iotedged -- Finished configuring certificates.
7/26/2018 7:50:24 AM info: iotedged -- Transparent gateway certificates not found, operating in quick start mode...
7/26/2018 7:50:24 AM info: iotedged -- Configuring certificates...
7/26/2018 7:50:24 AM info: iotedged -- Initializing hsm...
7/26/2018 7:50:24 AM info: iotedged -- Finished initializing hsm.
7/26/2018 7:50:24 AM info: iotedged -- Detecting if configuration file has changed...
7/26/2018 7:50:24 AM info: iotedged -- Provisioning edge device...
7/26/2018 7:50:24 AM info: iotedged -- No change to configuration file detected.
7/26/2018 7:50:24 AM info: iotedged -- Starting management API...
7/26/2018 7:50:24 AM info: iotedged::windows -- Stopped iotedged service.
7/26/2018 7:50:24 AM info: provisioning::provisioning -- Manually provisioning device "IoTEdge-0001" in hub "iothubsunbelt.azure-devices.net"
7/26/2018 7:50:24 AM info: iotedged -- Finished provisioning edge device.
7/26/2018 7:50:24 AM error: edgelet_utils::logging -- An http error occurred.
7/26/2018 7:50:24 AM error: iotedged::windows -- Error while running service. Quitting.

[gauravagarwal28]

I have a similar issue. Downgrading to older docker version didn't work either. Any pointers?

[veyalla]

@MPapst @gauravagarwal28 Can you share the contents of config.yaml and output of ipconfig from your systems?

[JMayrbaeurl]

Just checked on my machine. On starting Docker I've received an update notification and installed the latest update. Test with latest version 18.06.0-ce-win72 (19098), released today, did work without any problems.

[ilyas-it83]

#                      IoT Edge Daemon configuration
###############################################################################
#
# This file configures the IoT Edge daemon. The daemon must be restarted to
# pick up any configuration changes.
#
# Note - this file is yaml. Learn more here: http://yaml.org/refcard.html
#
###############################################################################

###############################################################################
# Provisioning mode and settings
###############################################################################
#
# Configures the identity provisioning mode of the daemon.
#
# Supported modes:
#     manual - using an iothub connection string
#     dps    - using dps for provisioning
#
###############################################################################

provisioning:
  source: "manual"
  device_connection_string: "HostName=xxxxxx.azure-devices.net;DeviceId=mydesktopEdge;SharedAccessKey=Iwontdisclose"

# provisioning:
#   source: "dps"
#   global_endpoint: "https://global.azure-devices-provisioning.net"
#   scope_id: "{scope_id}"
#   registration_id: "{registration_id}"

###############################################################################
# Certificate settings
###############################################################################
#
# Configures the certificates required to operate the IoT Edge
# runtime as a gateway which enables external leaf devices to securely
# communicate with the Edge Hub. If not specified, the required certificates
# are auto generated for quick start scenarios which are not intended for
# production environments.
#
# Settings:
#     device_ca_cert   - path to the device ca certificate and its chain
#     device_ca_pk     - path to the device ca private key file
#     trusted_ca_certs - path to a file containing all the trusted CA
#                        certificates required for Edge module communication
#
###############################################################################

# certificates:
#   device_ca_cert: "<ADD PATH TO DEVICE CA CERTIFICATE HERE>"
#   device_ca_pk: "<ADD PATH TO DEVICE CA PRIVATE KEY HERE>"
#   trusted_ca_certs: "<ADD PATH TO TRUSTED CA CERTIFICATES HERE>"

###############################################################################
# Edge Agent module spec
###############################################################################
#
# Configures the initial Edge Agent module.
#
# The daemon uses this definition to bootstrap the system. The Edge Agent can
# then update itself based on the Edge Agent module definition present in the
# deployment in IoT Hub.
#
###############################################################################

agent:
  name: "edgeAgent"
  type: "docker"
  env: {}
  config:
    image: "mcr.microsoft.com/azureiotedge-agent:1.0"
    auth: {}

###############################################################################
# Edge device hostname
###############################################################################
#
# Configures the environment variable 'IOTEDGE_GATEWAYHOSTNAME' injected into
# modules.
#
###############################################################################

hostname: "IN-L-7010516"

###############################################################################
# Connect settings
###############################################################################
#
#
# Configures URIs used by clients of the management and workload APIs
#     management_uri - used by the Edge Agent and 'iotedge' CLI to start,
#                      stop, and manage modules
#     workload_uri   - used by modules to retrieve tokens and certificates
#
# The following uri schemes are supported:
#     http - connect over TCP
#
###############################################################################

connect:
  management_uri: "http://10.0.75.1:15580"
  workload_uri: "http://10.0.75.1:15581"

###############################################################################
# Listen settings
###############################################################################
#
# Configures the listen addresses for the daemon.
#     management_uri - used by the Edge Agent and 'iotedge' CLI to start,
#                      stop, and manage modules
#     workload_uri   - used by modules to retrieve tokens and certificates
#
# The following uri schemes are supported:
#     http - listen over TCP
#
###############################################################################

listen:
  management_uri: "http://10.0.75.1:15580"
  workload_uri: "http://10.0.75.1:15581"

###############################################################################
# Home Directory
###############################################################################
#
# This configures the home directory for the daemon.
#
###############################################################################

homedir: "C:\\ProgramData\\iotedge"

###############################################################################
# Moby Container Runtime settings
###############################################################################
#
# uri - configures the uri for the container runtime.
# network - configures the network on which the containers will be created.
#
###############################################################################

moby_runtime:
  uri: "npipe://./pipe/docker_engine"
    network: "nat"

[MPapst]

@veyalla: I'll check the IP's tomorrow and post it here.

[veyalla]

@MPapst Sounds good, another to check if Docker default to Linux containers after upgrade. https://docs.docker.com/docker-for-windows/#switch-between-windows-and-linux-containers

@ilyas-it83 Looks like you are working with Janice on the issues you are facing, I'll let her handle that.

[myagley]

@ilyas-it83 one thing to note about your config file is that it is yaml and whitespace is significant. It looks like there are too many spaces before the network: "nat" which is mostly likely causing the configuration errors you are seeing. The moby_runtime section should look like the following:

moby_runtime:
  uri: "npipe://./pipe/docker_engine"
  network: "nat"

[MPapst]

@veyalla, you were right about the config.yaml. The IP of the host inside the nat network has changed with the update from 10.0.75.1 to 172.26.48.1.
Isn't there any way to detect that automatically when the iotedge service is starting?
Correcting the IP and the service works again.

Shall I close the issue or do you want to use as kind of feature request?

[gauravagarwal28]

@veyalla

PS C:\WINDOWS\system32> ipconfig

Windows IP Configuration

Ethernet adapter vEthernet (DockerNAT):

Connection-specific DNS Suffix . :
Link-local IPv6 Address . . . . . : fe80::3902:ec8b:9a31:9b8a%4
IPv4 Address. . . . . . . . . . . : 10.0.75.1
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . :

Ethernet adapter Ethernet:

Connection-specific DNS Suffix . :
Link-local IPv6 Address . . . . . : fe80::a8f2:5021:f32b:93e0%18
IPv4 Address. . . . . . . . . . . : 10.9.43.202
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.9.43.1

Ethernet adapter vEthernet (Default Switch):

Connection-specific DNS Suffix . :
Link-local IPv6 Address . . . . . : fe80::496c:26f8:8f39:337e%10
IPv4 Address. . . . . . . . . . . : 172.31.89.161
Subnet Mask . . . . . . . . . . . : 255.255.255.240
Default Gateway . . . . . . . . . :
PS C:\WINDOWS\system32> hostname
GauravA-PC

My Config.yaml file:

config.txt

((I copied it to desktop and renamed it to config.txt so that it could be uploaded here))

[veyalla]

@gauravagarwal28 can you confirm you have executed this command per the instructions (you can do it again to be sure):

Invoke-WebRequest -useb https://download.microsoft.com/download/0/6/4/064F84EA-D1DB-4EAA-9A5C-CC2F0FF6A638/vc_redist.x64.exe -o vc_redist.exe 
.\vc_redist.exe /quiet /norestart

[gauravagarwal28]

@veyalla
Yes, I have executed this command too.

[veyalla]

@MPapst You are probably confusing the "nat" network used for Windows containers with "DockerNAT" network used for Linux containers. I'd guess that after the Docker for Windows update you were inadvertently switched from using Windows containers to Linux containers (or vice versa).

I am glad you were able to get it working. Please close the issue, and feel free to open a new one if you need any additional assistance.