Module Client using GatewayName prevents NetworkingConfig

[nanowireUK]

We are deploying IoT Edge to a host with existing Docker containers bound to an existing Docker network 'main-network'. We want to be able to deploy IoT Edge and have these containers communicate with the existing containers whilst retaining edgeHub connectivity.

Expected Behavior

Using the createOptions to pass a NetworkingConfig>EndpointsConfig object that contains 'main-network'. We do this for all containers within the deployment manifest using the object below. Expected behaviour is that containers can communicate within this 'main-network' and with the edgeHub / IoT Hub.

"createOptions": {
  "NetworkingConfig": {
    "EndpointsConfig": {
      "main-network": {
        "IPAMConfig": {},
        "Links": [],
        "Aliases": []
    }
  }
}

Current Behavior

The configuration is applied as expected and all containers can communicate with one another on a network level.

The critical issue is that any containers using the IoT Hub Device SD to initiatialise a Module Client are unable to communicate with the EdgeHub. I suspect this also explains some problems in issue #1275

We have done our own RCA and have found out that the issue is the reliance on an alias on the EdgeHub within the standard azure-iot-edge network. This is presumably in place to facilitate code reuse within the SDK for clients connecting to the edgeHub as a transparent gateway. The env $GATEWAYHOSTNAME is used as the hostname and this is not resolvable within the docker environment. The fix for us is to put the hostname in the Aliases for the edgeHub and after this everything works as expected.

"createOptions": {
  "NetworkingConfig": {
    "EndpointsConfig": {
      "main-network": {
        "IPAMConfig": {},
        "Links": [],
        "Aliases": [
          "<gateway-hostname>"
        ]
    }
  }
}

The obvious problem with this approach is that this means the manifest is now specific to a particular host and cannot be deployed to other instances without adapting the Aliases array.

Desired behaviour would be that the Module Client instead connects to the 'edgeHub' hostname since this should always be reachalbe within an IoT Edge environment.

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

1. Deploy latest version of IoT Edge to Windows or Linux
2. Create a new Docker network on the host
3. Deploy a manifest where all modules have createOptions that includes the above NetworkingConfig
4. Try initialising a Module Client within a module. This will raise an unknown host exception as the Module Client uses the $GATEWAYHOSTNAME env and cannot resolve this

Context (Environment)

Output of iotedge check

Not available currently since this was on an environment I no longer have access to. No issues beyond the standard production readiness warnings were issued. All connectivity checks passed (since edgeHub can still communicate with IoT Hub).

Device Information

• Host OS [e.g. Ubuntu 16.04, Ubuntu 18.04, Windows IoT Core]: Windows 10 LTSB
• Architecture [e.g. amd64, arm32, arm64]: amd64
• Container OS [e.g. Linux containers, Windows containers]: Linux Containers

[and-rewsmith]

The $GATEWAYHOSTNAME env var represents the hostname of the edge device. EdgeHub then binds to the ports of the host machine so that external devices can interact with it using this gateway hostname. I am guessing that you overwrote the default CreateOptions for EdgeHub? You can see an example here:
https://docs.microsoft.com/en-us/azure/iot-edge/module-composition

You won't need to declare an alias if the EdgeHub binds its ports to the host. Once the ports are bound, then the containers should be able to communicate with EdgeHub using the gateway host name.

Let me know if you have any questions.

[and-rewsmith]

@nanowireUK Any updates?

[nanowireUK]

Sorry wasn't getting notifications on messages - just when you closed it.

The problem isn't that the EdgeHub isn't accessible. The problem is that a module within the Edge deployment cannot resolve the $GATEWAYHOSTNAME without that alias which severely impacts the ability to use CreateOptions to control network.

[and-rewsmith]

We just had a call where we confirmed that this problem is not related to port bindings. It is a symptom of overwriting the networking configuration for edgeHub as @nanowireUK described initially.

[and-rewsmith]

I think that this scenario can work on linux if you change the default network in the config.yaml. This way, you won't need to edit the create options (eliminating the needed alias which is dynamically generated).

Also I have a few notes on your scenario:

• You are using linux containers on windows, which is not supported
• There is currently a bug on windows where if you are running containers on a custom network (either defined in the config yaml or through create options), then containers won't start back up when you restart the device

Here is the relevant line in the config.yaml:
https://github.com/Azure/iotedge/blob/master/edgelet/contrib/config/linux/config.yaml#L351