SCTP connections between Docker containers hangs

[chrisawad]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

After starting up the containers with docker-compose (or just docker run, individually), I expect the server container to listen for incoming connections on sctp port 3868, the client container to initiate a connection to the server container on the same port, and for the server container to accept the connection.

$ docker-compose up
Starting server ... done
Starting client ... done
Attaching to server, client
server | Opening SctpServerChannel...
server | Binding to: server/172.18.0.2:3868
server | Waiting for new connection....
client | Opening SctpChannel...
client | Binding to: client/172.18.0.3:0
client | Connecting to: server/172.18.0.2:3868
> server | Connected: [ client/172.18.0.3:8888 ]
> client | Connected!

Actual behavior

The server container comes up and waits for connections as expected and the client initiates the connection as expected, but the server never accepts the connection and the client blocks indefinitely.

$ docker-compose up
Starting server ... done
Starting client ... done
Attaching to server, client
server | Opening SctpServerChannel...
server | Binding to: server/172.18.0.2:3868
server | Waiting for new connection....
client | Opening SctpChannel...
client | Binding to: client/172.18.0.3:0
client | Connecting to: server/172.18.0.2:3868

Steps to reproduce the behavior

src/SctpClient.java

import java.io.IOException;
import java.net.InetSocketAddress;

import com.sun.nio.sctp.SctpChannel;

public class SctpClient 
{
    public static void main(String[] args) throws IOException 
    {
        String[] parts = args[0].split(":");
        InetSocketAddress local = new InetSocketAddress(parts[0], Integer.parseInt(parts[1]));

        parts = args[1].split(":");
        InetSocketAddress remote = new InetSocketAddress(parts[0], Integer.parseInt(parts[1]));

        System.out.println("Opening SctpChannel...");
        SctpChannel client = SctpChannel.open();

        System.out.println("Binding to: "+local);
        client.bind(local);

        System.out.println("Connecting to: "+remote);
        if (client.connect(remote))
            System.out.println("Connected!");           
        else
            System.out.println("Connection failed!");
    }
}

src/SctpServer.java

import java.io.IOException;
import java.net.InetSocketAddress;

import com.sun.nio.sctp.SctpChannel;
import com.sun.nio.sctp.SctpServerChannel;

public class SctpServer 
{
    public static void main(String[] args) throws IOException 
    {
        String[] parts = args[0].split(":");
        InetSocketAddress local = new InetSocketAddress(parts[0], Integer.parseInt(parts[1]));

        System.out.println("Opening SctpServerChannel...");
        SctpServerChannel server = SctpServerChannel.open();

        System.out.println("Binding to: "+local);
        server.bind(local);

        System.out.println("Waiting for new connection....");
        while (true)
        {
            SctpChannel client = server.accept();
            if (client == null)
                System.out.println("Connection failed!");
            else
                System.out.println("Connected: "+client.getRemoteAddresses());
        }
    }
}

docker-compose.yml

version: '3.7'
services:

        server:
                build:
                        context: .
                        dockerfile: Dockerfile-Server
                image: server
                container_name: server
                restart: unless-stopped
                environment:
                        - LOCAL_IFACE=server:3868
                ports:
                        - "3868:3868/sctp"
                networks:
                        - sctp

        client:
                build:
                        context: .
                        dockerfile: Dockerfile-Client
                image: client
                container_name: client
                restart: unless-stopped
                environment:
                        - LOCAL_IFACE=client:0
                        - REMOTE_IFACE=server:3868
                networks:
                        - sctp
                depends_on:
                        - server
                stdin_open: true
                tty: true

networks:
        sctp:
                name: sctp

Dockerfile-Client

FROM openjdk:11-jdk
RUN apt-get update -y && apt-get install lksctp-tools -y && apt-get clean
WORKDIR /opt
COPY src/SctpClient.java .
CMD java SctpClient.java $LOCAL_IFACE $REMOTE_IFACE

Dockerfile-Server

FROM openjdk:11-jdk
RUN apt-get update -y && apt-get install lksctp-tools -y && apt-get clean
WORKDIR /opt
COPY src/SctpServer.java .
CMD java SctpServer.java $LOCAL_IFACE

Output of docker version:

$ docker version
Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:29:33 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfc
  Built:            Thu Aug 29 05:28:12 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

$ docker info
Client:
 Debug Mode: false

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 12
 Server Version: 19.03.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.0.9-301.fc30.x86_64
 Operating System: Fedora 30 (Workstation Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 3.839GiB
 Name: localhost.localdomain
 ID: YIOB:ZF2L:XEOQ:TONL:6WJG:BKOC:FEBC:TPJJ:NIRJ:ZKFJ:CXL4:BTXF
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  10.1.12.9:5000
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)

I am running this a Fedora 30 Workstation VMWare Guest. Also here's the version of docker-compose. Older versions may not allow sctp in service..ports section.

$ docker-compose --version
docker-compose version 1.25.0-rc2, build 661ac20e

[chrisawad]

Tried a different distro just in case, and it's the same issue...

Here: fedora:30 with java-11-openjdk installed via dnf
Instead of: openjdk:11-jdk which is based on Debian

FROM fedora:30
RUN dnf update -y && dnf install java-11-openjdk lksctp-tools -y && dnf clean all
... copy over files ...

[chrisawad]

Little more info:

If I run the server in a container as normal and run SctpClient locally on the host (towards the server container) via:
$ java SctpClient.java 0.0.0.0:0 <server_container_ip>:3868
The connection works! This indicates that the server container is acting correctly.

If I run the client in a container and run SctpServer locally on the host via:
$ java SctpServer.java 0.0.0.0:3868
The client never connects. Unless there's something else I have to do to get the container to talk to my host, this indicates that there's something going on with the client container.

If I run both SctpClient and SctpServer on the host, everything works fine.

I've also tried using ncat instead of Java to act as an sctp server and sctp client. That showed the same issue.

To use ncat instead of my Java application:
Server: ncat --sctp -l 3868
Client: ncat --sctp <remote_ip> 3868

[chrisawad]

Anyone? I don't think SCTP connections can go out of a docker container.

[chrisawad]

Well that's interesting. If I comment out publishing of the ports from docker-compose.yml, SCTP between containers works fine. I don't think this works the same if it were TCP.

Unexpected behavior?

[dkalashnik]

I can confirm it. With exposed sctp port connection hangs after INIT_ACK, without port it goes as expected, in the attachment you can see pcap dump for both cases with the same tools.

docker_sctp_pcaps.zip

[thaJeztah]

/cc @ishidawataru @AkihiroSuda