edgeHub System.Net.Sockets.SocketException (13): Permission denied #9

Siddius · 2018-07-05T13:07:25Z

Expected Behavior

EdgeHub should start

Current Behavior

edgeHub is crashing

Steps to Reproduce

Deploy a service

Context (Environment)

Device (Host) Operating System

Ubuntu 16.04 aarch64

Architecture

aarch64

Container Operating System

Linux Docker version

Runtime Versions

iotedged

<6>2018-07-05T13:00:16Z [INFO] - Starting Azure IoT Edge Security Daemon
<6>2018-07-05T13:00:16Z [INFO] - Version - 1.0.0 (52ef77db24126bf473265fc09c53d35290a2dd6b)
iotedged 0.1.0

Edge Agent

mcr.microsoft.com/azureiotedge-agent:1.0.0-linux-arm32v7

Edge Hub

mcr.microsoft.com/azureiotedge-hub:1.0.0-linux-arm32v7

Docker

18.03.1-ce

Logs

2018-06-28 11:22:15.776 +00:00 [FTL] - Unable to start Kestrel.
System.Net.Sockets.SocketException (13): Permission denied
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransport.BindAsync()
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.<>c__DisplayClass22_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.BindAsync(AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.EndpointsStrategy.BindAsync(AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(IServerAddressesFeature addresses, KestrelServerOptions serverOptions, ILogger logger, Func`2 createBinding)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
2018-06-28 11:22:15.916 +00:00 [INF] - Starting MQTT head
2018-06-28 11:22:16.071 +00:00 [INF] - Initializing TLS endpoint on port 8883 for MQTT head.
2018-06-28 11:22:16.266 +00:00 [INF] - Starting AMQP head
2018-06-28 11:22:16.292 +00:00 [INF] - Started MQTT head
2018-06-28 11:22:16.778 +00:00 [INF] - Started AMQP head
2018-06-28 11:22:16.848 +00:00 [INF] - Stopping
2018-06-28 11:22:31.981 +00:00 [INF] - Stopped

Unhandled Exception: System.AggregateException: One or more errors occurred. (Permission denied) ---> System.Net.Sockets.SocketException: Permission denied
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransport.BindAsync()
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.<>c__DisplayClass22_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.BindAsync(AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.EndpointsStrategy.BindAsync(AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(IServerAddressesFeature addresses, KestrelServerOptions serverOptions, ILogger logger, Func`2 createBinding)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Edge.Hub.Http.HttpProtocolHead.StartAsync() in /opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Http/HttpProtocolHead.cs:line 25
   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 119
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /opt/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 31

Additional Information

Except ssh and docker no other service is running on the host system

The text was updated successfully, but these errors were encountered:

myagley · 2018-07-09T16:04:58Z

Hello. Thanks for the issue report.

Do you have any other services on the device that are running on port 443? This could be a port conflict. Can you please curl -v https://localhost:443 just to be sure?

The Edge Hub is also run as an unprivileged user. In order to make this work, we need to set the CAP_NET_BIND_SERVICE capability to bind to a sub-1024 port. There could be something going on there, although we haven't run into anything here or heard any reports yet.

Siddius · 2018-07-10T07:54:36Z

Have checked once more that on host system nothing is running on port 443 ( checked with netstat and and curl -v https://localhost:443 which failed )
Also this should not be a problem cause the port is opened inside the container which should always be available. If the port on the host is occupied docker would throw an error, not the program in the container.

CAP_NET_BIND_SERVICE was also my first guess but i thought this would be managed by your daemon.

docker inspect edgeHub that neither Privileged nor CAP_NET_BIND_SERVICE is granted. Here an excerpt :

[
    {
        "Id": "ebe23cbe09af2f308531c6c19a85e3f467ee454ffc6ccdcbaf04f0eae874b7fd",
        "Created": "2018-07-10T07:29:59.923346091Z",
        "Path": "/bin/sh",
        "Args": [
            "-c",
            "echo \"$(date --utc +\"[%Y-%m-%d %H:%M:%S %:z]\"): Starting Edge Hub\" &&     exec /usr/bin/dotnet Microsoft.Azure.Devices.Edge.Hub.Service.dll"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 132,
            "Error": "",
            "StartedAt": "2018-07-10T07:30:01.479638513Z",
            "FinishedAt": "2018-07-10T07:30:35.303768219Z"
        },
        "Image": "sha256:b5637d573d56d73ba61b2b6ecbac862dcc0f0a7754ec3097c227c05b1bfe9a1a",
        "ResolvConfPath": "/var/lib/docker/containers/ebe23cbe09af2f308531c6c19a85e3f467ee454ffc6ccdcbaf04f0eae874b7fd/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/ebe23cbe09af2f308531c6c19a85e3f467ee454ffc6ccdcbaf04f0eae874b7fd/hostname",
        "HostsPath": "/var/lib/docker/containers/ebe23cbe09af2f308531c6c19a85e3f467ee454ffc6ccdcbaf04f0eae874b7fd/hosts",
        "LogPath": "/var/lib/docker/containers/ebe23cbe09af2f308531c6c19a85e3f467ee454ffc6ccdcbaf04f0eae874b7fd/ebe23cbe09af2f308531c6c19a85e3f467ee454ffc6ccdcbaf04f0eae874b7fd-json.log",
        "Name": "/edgeHub",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/var/run/iotedge/workload.sock:/var/run/iotedge/workload.sock"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {
                "443/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "443"
                    }
                ],
                "5671/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "5671"
                    }
                ],
                "8883/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "8883"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },

...

myagley · 2018-07-10T16:39:52Z

We actually setcap the binary when creating the image: https://github.com/Azure/iotedge/blob/master/edge-hub/docker/linux/arm32v7/base/Dockerfile#L31

I believe setcap requires filesystem support. Can you please provide a little more info about your system? Run docker info

Siddius · 2018-07-11T05:49:27Z

ocker info
Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 12
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.0-yocto-standard
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 3.682GiB
Name: RCAR
ID: FXIG:6E75:GASX:73Q4:P5EP:FXWK:RUJO:OOAX:7AEX:5AWB:FEQI:LXFD
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

myagley · 2018-07-11T16:55:12Z

Does this system have any other security modules installed and enabled, for instance SELinux? How was the OS built/obtained? I notice a yocto kernel, with Ubuntu 16.04 distro, on aarch64, which seems non-standard. Just trying to understand what could be different here.

For reference, this is the list of officially supported platforms. This setup is a little outside anything we've tested.

Siddius · 2018-07-12T13:34:38Z

Its an Ubuntu 16.04 from scratch (roofts) with special kernel build with yocto, which is needed for this board.

Seems like the kernel is missing something, i know there is an EXT3_FS_XATTR but i am using ext4 and this should not be a problem, i guess. But both setcap and getcap give:

# getcap /usr/share/dotnet/dotnet
Failed to get capabilities of file `/usr/share/dotnet/dotnet' (Operation not supported)

Siddius · 2018-07-13T11:26:06Z

Okay i did it :)

In kernel you need the XATTR and SECURITY feature enabled for the underlying filesystem in my case
CONFIG_EXT4_FS_SECURITY=y
after this you have to delete the edgeHub image and start iotedge.service again.

SimonSimCity · 2021-07-13T09:58:20Z

I was brought here because the file-system I mounted as root - which is NFS v3 - seems not to be supported using Linux 5.4.72 😢 Switching to ext4 works - but sadly doesn't allow me to boot from network (as a thin client would do). If someone has a solution, I'm thrilled to hear about it! I've heard support for XATTR landed in Linux 5.9 (https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.9-NFS-Server-User-Xattr) but haven't tried.

To those using other file systems - maybe your filesystem doesn't support extended attributes (Linux calls the feature XATTR) or MAC labels - MAC as MandatoryAccessControl - (Linux calls the feature SECURITY) at all? Wikipedia has a list of filesystems and what metadata they support: https://en.wikipedia.org/wiki/Comparison_of_file_systems#Metadata

Siddius mentioned this issue Jul 11, 2018

EdgeAgent can't connect Azure/iot-edge-v1#631

Closed

Siddius closed this as completed Jul 13, 2018

HenrikBach1 mentioned this issue Sep 14, 2018

V2 - Why does the edgeHub module stop? #228

Closed

matsujirushi mentioned this issue Dec 29, 2018

edgeHub System.Net.Sockets.SocketException (13): Permission denied #671

Closed

varunpuranik mentioned this issue Feb 11, 2020

Add developer docs for new features in IoT Edge #2332

Merged

tony979 mentioned this issue Feb 24, 2020

transport.connect raised error with send_message_to_output.py Azure/azure-iot-sdk-python#462

Closed

yophilav added customer-reported iotedge labels Mar 17, 2020

allenzt mentioned this issue Aug 13, 2020

edgeHub start up failed due to setcap not take effect in Container #3401

Closed

This was referenced Nov 16, 2020

module start up issue with NXP yocto port of iotedge #3845

Closed

1.2.0-rc1 edgeHub failure on tier two ARM64 device #3974

Closed

mschwan-phytec mentioned this issue Jan 21, 2021

edgeHub fails with permission error, CONFIG_EXT4_FS_SECURITY does not solve issue #4262

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

edgeHub System.Net.Sockets.SocketException (13): Permission denied #9

edgeHub System.Net.Sockets.SocketException (13): Permission denied #9

Siddius commented Jul 5, 2018

myagley commented Jul 9, 2018

Siddius commented Jul 10, 2018

myagley commented Jul 10, 2018

Siddius commented Jul 11, 2018

myagley commented Jul 11, 2018

Siddius commented Jul 12, 2018 •

edited

Loading

Siddius commented Jul 13, 2018 •

edited

Loading

SimonSimCity commented Jul 13, 2021 •

edited

Loading

edgeHub System.Net.Sockets.SocketException (13): Permission denied #9

edgeHub System.Net.Sockets.SocketException (13): Permission denied #9

Comments

Siddius commented Jul 5, 2018

Expected Behavior

Current Behavior

Steps to Reproduce

Context (Environment)

Device (Host) Operating System

Architecture

Container Operating System

Runtime Versions

iotedged

Edge Agent

Edge Hub

Docker

Logs

Additional Information

myagley commented Jul 9, 2018

Siddius commented Jul 10, 2018

myagley commented Jul 10, 2018

Siddius commented Jul 11, 2018

myagley commented Jul 11, 2018

Siddius commented Jul 12, 2018 • edited Loading

Siddius commented Jul 13, 2018 • edited Loading

SimonSimCity commented Jul 13, 2021 • edited Loading

Siddius commented Jul 12, 2018 •

edited

Loading

Siddius commented Jul 13, 2018 •

edited

Loading

SimonSimCity commented Jul 13, 2021 •

edited

Loading