[BUG] Network issues

[Penagwin]

Describe the bug
The latest release coupled with the extra port range seems to have fixed my connectivity problems from firefox, however I still can't get it to work from Chrome.

The image is running on a linux box, and I'm testing from a mac on the same network.

I was able to get it to work in chrome by adding --net=host, and this error makes me think that there's a port range that might need to be forwarded or something? I'm not sure.

7:06PM WRN could not get server reflexive address udp6 stun:stun.l.google.com:19302: write udp6 [::]:59037->[2607:f8b0:4001:c1a::7f]:19302: sendto: cannot assign requested address

To Reproduce
Steps to reproduce the behavior:

1. docker pull nurdism/neko:latest
2. docker run -p 8080:8080 -p 59000-59100:59000-59100/udp -e NEKO_PASSWORD='secret' -e NEKO_ADMIN='secret' --shm-size=1gb nurdism/neko:latest
3. Connects with Firefox, not Chrome.

Client Log:

Server Log:

2020-01-29 19:06:21,688 DEBG 'neko' stdout output:
7:06PM DBG request complete (200) module=http req={"agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36","method":"GET","proto":"HTTP/1.1","remote":"10.0.1.46:54824","scheme":"http","uri":"http://10.0.1.101:8080/favicon-32x32.png"} res={"bytes":1003,"elapsed":0.170201,"status":200,"time":"Wed, 29 Jan 2020 19:06:21 UTC"}

2020-01-29 19:06:26,397 DEBG 'neko' stdout output:
7:06PM DBG attempting to upgrade connection module=websocket

2020-01-29 19:06:26,397 DEBG 'neko' stdout output:
7:06PM DBG session created id=HB47-I3893KmvbAgyjrB4La5SUW22_sr module=webrtc
7:06PM DBG session created id=HB47-I3893KmvbAgyjrB4La5SUW22_sr module=websocket
7:06PM DBG new connection created address=10.0.1.46:54852 module=websocket session=HB47-I3893KmvbAgyjrB4La5SUW22_sr

2020-01-29 19:06:26,567 DEBG 'neko' stdout output:
7:06PM DBG recieved message from client module=websocket raw="{\"event\":\"signal/provide\",\"sdp\":\"v=0\\r\\no=- 5685503494432030939 2 IN IP4 127.0.0.1\\r\\ns=-\\r\\nt=0 0\\r\\na=group:BUNDLE 0 1 2\\r\\na=msid-semantic: WMS\\r\\nm=audio 33864 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126\\r\\nc=IN IP4 152.160.163.2\\r\\na=rtcp:9 IN IP4 0.0.0.0\\r\\na=candidate:3408114210 1 udp 2113937151 d3d7beb9-15bc-4a78-8a79-c6a0a57bac0f.local 56317 typ host generation 0 network-cost 999\\r\\na=candidate:842163049 1 udp 1677729535 152.160.163.2 33864 typ srflx raddr 0.0.0.0 rport 0 generation 0 network-cost 999\\r\\na=ice-ufrag:BjL4\\r\\na=ice-pwd:aB9GMlvIcct1Uic+fUviTupm\\r\\na=ice-options:trickle\\r\\na=fingerprint:sha-256 C7:88:3B:49:80:69:5B:F9:AA:1F:03:4D:F0:A0:3F:73:D6:E4:23:85:22:22:BF:E7:42:BC:D4:15:FF:96:17:60\\r\\na=setup:actpass\\r\\na=mid:0\\r\\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\\r\\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\\r\\na=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\\r\\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\\r\\na=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\\r\\na=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\\r\\na=recvonly\\r\\na=rtcp-mux\\r\\na=rtpmap:111 opus/48000/2\\r\\na=rtcp-fb:111 transport-cc\\r\\na=fmtp:111 minptime=10;useinbandfec=1\\r\\na=rtpmap:103 ISAC/16000\\r\\na=rtpmap:104 ISAC/32000\\r\\na=rtpmap:9 G722/8000\\r\\na=rtpmap:0 PCMU/8000\\r\\na=rtpmap:8 PCMA/8000\\r\\na=rtpmap:106 CN/32000\\r\\na=rtpmap:105 CN/16000\\r\\na=rtpmap:13 CN/8000\\r\\na=rtpmap:110 telephone-event/48000\\r\\na=rtpmap:112 telephone-event/32000\\r\\na=rtpmap:113 telephone-event/16000\\r\\na=rtpmap:126 telephone-event/8000\\r\\nm=video 33191 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 122 127 121 125 107 108 109 124 120 123 119 114 115 116\\r\\nc=IN IP4 152.160.163.2\\r\\na=rtcp:9 IN IP4 0.0.0.0\\r\\na=candidate:3408114210 1 udp 2113937151 d3d7beb9-15bc-4a78-8a79-c6a0a57bac0f.local 56983 typ host generation 0 network-cost 999\\r\\na=candidate:842163049 1 udp 1677729535 152.160.163.2 33191 typ srflx raddr 0.0.0.0 rport 0 generation 0 network-cost 999\\r\\na=ice-ufrag:BjL4\\r\\na=ice-pwd:aB9GMlvIcct1Uic+fUviTupm\\r\\na=ice-options:trickle\\r\\na=fingerprint:sha-256 C7:88:3B:49:80:69:5B:F9:AA:1F:03:4D:F0:A0:3F:73:D6:E4:23:85:22:22:BF:E7:42:BC:D4:15:FF:96:17:60\\r\\na=setup:actpass\\r\\na=mid:1\\r\\na=extmap:14 urn:ietf:params:rtp-hdrext:toffset\\r\\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\\r\\na=extmap:13 urn:3gpp:video-orientation\\r\\na=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\\r\\na=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\\r\\na=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\\r\\na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\\r\\na=extmap:8 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07\\r\\na=extmap:9 http://www.webrtc.org/experiments/rtp-hdrext/color-space\\r\\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\\r\\na=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\\r\\na=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\\r\\na=recvonly\\r\\na=rtcp-mux\\r\\na=rtcp-rsize\\r\\na=rtpmap:96 VP8/90000\\r\\na=rtcp-fb:96 goog-remb\\r\\na=rtcp-fb:96 transport-cc\\r\\na=rtcp-fb:96 ccm fir\\r\\na=rtcp-fb:96 nack\\r\\na=rtcp-fb:96 nack pli\\r\\na=rtpmap:97 rtx/90000\\r\\na=fmtp:97 apt=96\\r\\na=rtpmap:98 VP9/90000\\r\\na=rtcp-fb:98 goog-remb\\r\\na=rtcp-fb:98 transport-cc\\r\\na=rtcp-fb:98 ccm fir\\r\\na=rtcp-fb:98 nack\\r\\na=rtcp-fb:98 nack pli\\r\\na=fmtp:98 profile-id=0\\r\\na=rtpmap:99 rtx/90000\\r\\na=fmtp:99 apt=98\\r\\na=rtpmap:100 VP9/90000\\r\\na=rtcp-fb:100 goog-remb\\r\\na=rtcp-fb:100 transport-cc\\r\\na=rtcp-fb:100 ccm fir\\r\\na=rtcp-fb:100 nack\\r\\na=rtcp-fb:100 nack pli\\r\\na=fmtp:100 profile-id=2\\r\\na=rtpmap:101 rtx/90000\\r\\na=fmtp:101 apt=100\\r\\na=rtpmap:102 H264/90000\\r\\na=rtcp-fb:102 goog-remb\\r\\na=rtcp-fb:102 transport-cc\\r\\na=rtcp-fb:102 ccm fir\\r\\na=rtcp-fb:102 nack\\r\\na=rtcp-fb:102 nack pli\\r\\na=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f\\r\\na=rtpmap:122 rtx/90000\\r\\na=fmtp:122 apt=102\\r\\na=rtpmap:127 H264/90000\\r\\na=rtcp-fb:127 goog-remb\\r\\na=rtcp-fb:127 transport-cc\\r\\na=rtcp-fb:127 ccm fir\\r\\na=rtcp-fb:127 nack\\r\\na=rtcp-fb:127 nack pli\\r\\na=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f\\r\\na=rtpmap:121 rtx/90000\\r\\na=fmtp:121 apt=127\\r\\na=rtpmap:125 H264/90000\\r\\na=rtcp-fb:125 goog-remb\\r\\na=rtcp-fb:125 transport-cc\\r\\na=rtcp-fb:125 ccm fir\\r\\na=rtcp-fb:125 nack\\r\\na=rtcp-fb:125 nack pli\\r\\na=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f\\r\\na=rtpmap:107 rtx/90000\\r\\na=fmtp:107 apt=125\\r\\na=rtpmap:108 H264/90000\\r\\na=rtcp-fb:108 goog-remb\\r\\na=rtcp-fb:108 transport-cc\\r\\na=rtcp-fb:108 ccm fir\\r\\na=rtcp-fb:108 nack\\r\\na=rtcp-fb:108 nack pli\\r\\na=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f\\r\\na=rtpmap:109 rtx/90000\\r\\na=fmtp:109 apt=108\\r\\na=rtpmap:124 H264/90000\\r\\na=rtcp-fb:124 goog-remb\\r\\na=rtcp-fb:124 transport-cc\\r\\na=rtcp-fb:124 ccm fir\\r\\na=rtcp-fb:124 nack\\r\\na=rtcp-fb:124 nack pli\\r\\na=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d0032\\r\\na=rtpmap:120 rtx/90000\\r\\na=fmtp:120 apt=124\\r\\na=rtpmap:123 H264/90000\\r\\na=rtcp-fb:123 goog-remb\\r\\na=rtcp-fb:123 transport-cc\\r\\na=rtcp-fb:123 ccm fir\\r\\na=rtcp-fb:123 nack\\r\\na=rtcp-fb:123 nack pli\\r\\na=fmtp:123 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640032\\r\\na=rtpmap:119 rtx/90000\\r\\na=fmtp:119 apt=123\\r\\na=rtpmap:114 red/90000\\r\\na=rtpmap:115 rtx/90000\\r\\na=fmtp:115 apt=114\\r\\na=rtpmap:116 ulpfec/90000\\r\\nm=application 14512 UDP/DTLS/SCTP webrtc-datachannel\\r\\nc=IN IP4 152.160.163.2\\r\\na=candidate:3408114210 1 udp 2113937151 d3d7beb9-15bc-4a78-8a79-c6a0a57bac0f.local 59725 typ host generation 0 network-cost 999\\r\\na=candidate:842163049 1 udp 1677729535 152.160.163.2 14512 typ srflx raddr 0.0.0.0 rport 0 generation 0 network-cost 999\\r\\na=ice-ufrag:BjL4\\r\\na=ice-pwd:aB9GMlvIcct1Uic+fUviTupm\\r\\na=ice-options:trickle\\r\\na=fingerprint:sha-256 C7:88:3B:49:80:69:5B:F9:AA:1F:03:4D:F0:A0:3F:73:D6:E4:23:85:22:22:BF:E7:42:BC:D4:15:FF:96:17:60\\r\\na=setup:actpass\\r\\na=mid:2\\r\\na=sctp-port:5000\\r\\na=max-message-size:262144\\r\\n\"}" session=HB47-I3893KmvbAgyjrB4La5SUW22_sr

2020-01-29 19:06:26,615 DEBG 'neko' stdout output:
7:06PM WRN could not get server reflexive address udp6 stun:stun.l.google.com:19302: write udp6 [::]:59037->[2607:f8b0:4001:c1a::7f]:19302: sendto: cannot assign requested address
 module=webrtc subsystem=ice

2020-01-29 19:06:26,615 DEBG 'neko' stdout output:
7:06PM INF signaling state changed to have-remote-offer module=webrtc subsystem=pc

2020-01-29 19:06:26,616 DEBG 'neko' stdout output:
7:06PM DBG Started agent: isControlling? false, remoteUfrag: "BjL4", remotePwd: "aB9GMlvIcct1Uic+fUviTupm" module=webrtc subsystem=ice

2020-01-29 19:06:26,616 DEBG 'neko' stdout output:
7:06PM INF Setting new connection state: Checking module=webrtc subsystem=ice
7:06PM INF ICE connection state changed: checking module=webrtc subsystem=pc
7:06PM INF signaling state changed to stable module=webrtc subsystem=pc

2020-01-29 19:06:36,616 DEBG 'neko' stdout output:
7:06PM INF Setting new connection state: Failed module=webrtc subsystem=ice
7:06PM INF ICE connection state changed: failed module=webrtc subsystem=pc

2020-01-29 19:06:36,617 DEBG 'neko' stdout output:
7:06PM INF peer connection state changed: failed module=webrtc subsystem=pc
7:06PM INF peer disconnected id=HB47-I3893KmvbAgyjrB4La5SUW22_sr module=webrtc
7:06PM DBG read message error error="read tcp 172.17.0.2:8080->10.0.1.46:54852: use of closed network connection" module=websocket
7:06PM DBG handle socket ending address=10.0.1.46:54852 module=websocket
7:06PM DBG session destroyed id=HB47-I3893KmvbAgyjrB4La5SUW22_sr module=webrtc
7:06PM DBG session destroyed id=HB47-I3893KmvbAgyjrB4La5SUW22_sr module=websocket
7:06PM DBG session ended address=10.0.1.46:54852 module=websocket session=HB47-I3893KmvbAgyjrB4La5SUW22_sr
7:06PM DBG request complete (0) module=http req={"agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36","method":"GET","proto":"HTTP/1.1","remote":"10.0.1.46:54852","scheme":"http","uri":"http://10.0.1.101:8080/ws?password=secret"} res={"bytes":0,"elapsed":10219.653686,"status":0,"time":"Wed, 29 Jan 2020 19:06:36 UTC"}
7:06PM DBG session destroyed id=HB47-I3893KmvbAgyjrB4La5SUW22_sr module=webrtc
7:06PM DBG session destroyed id=HB47-I3893KmvbAgyjrB4La5SUW22_sr module=websocket```

[nurdism]

have you opened the ports on your system? (ufw maybe?) this is a tough one cause I cant reproduce it. How are you accessing the site? is it through the docker ip, localhost or your public ip?

[Penagwin]

The docker instance is running on a separate linux box on my network, and I'm connecting through it's local ip address. It's possible that the network firewall is blocking something (I don't control that here), but I have no custom ufw rules on that box.

May I recommend you keep this issue open, as I have a feeling some others will run into a similar issue.

[nurdism]

if its working on firefox and not chrome, then the ports are correct. I did notice something,

WRN could not get server reflexive address udp6 stun:stun.l.google.com:19302

its warning something about udp6, and I believe this means its trying to use ipv6 to create a peer connection, check and see if you have any ipv6 rules in your firewall blocking the connection.

go here https://test.webrtc.org/ and tell me if ipv6 is enabled for you, try running that in the browser too

[Penagwin]

Might be on to something there. Maybe Chrome is trying for ipv6 only? Is there a way to force it to try over ipv4?

[nurdism]

I'm not sure, I haven't used ipv6 before.

[nurdism]

not sure what is causing this issue, if anyone has any suggestions?

[sad]

Everything was working fine for me previously, but I am now encountering this error as well. Tried with both the Firefox and Chrome images. I can connect on Firefox but not type (possibly due to the clipboard), and on Chrome sometimes I am able to log in but I am unable to see anything.

edit: server logs, and client logs

(Chrome on MacOS, but same behaviour on Windows)

[Penagwin]

@sad To your knowledge, does the network you're using normally support ipv6 (comcast for example)? And does your computer have an ipv6 address regardless of the question above (You can check by using ifconfig, ipconfig, or ip addr depending on your system).

For me my laptop had an ipv6 address, but the network I'm on doesn't really support ipv6, so I can't connect over the internet over ipv6.

I think what's happening is your computer has an ipv6 address so the browsers try to use that and fail. Likely we need to somehow force it to use ipv4?

EDIT: Also what's the output of test.webrtc.org (You can skip the audio/video test)

[sad]

My ISP has never supported ipv6, but I was able to run neko pretty smoothly up until the other day over an ipv4 connection.

[nurdism]

I've found a way to kinda debug this, add -e PION_LOG_TRACE='all' when you run the docker container and view chrome://webrtc-internals, if you can post the logs from this and we can see whats up with the connection

[sad]

With PION_LOG_TRACE='all', here are my server logs. After about 4 or 5 retrys, I was able to connect and join the room, but I got no video. Client logs here.

Here is the data from chrome://webrtc-internals, I'm not entirely sure how I should format this so if you would like I can take a screenshot instead. Hope this proves useful, thanks.

I should also add that I too tried with network_mode: "host" and it made no difference.

edit: okay so I used this site, and got the following:

The server stun:stun.l.google.com:19302 returned an error with code=701:
STUN allocate request timed out.

I double checked and this appears to be the same ICE server that neko uses. There seems to be a few issues open about this server and Chrome, such as this one and this one. Seems like this could be the issue.

@Penagwin do you think you could verify that this is the same on your end?

[nurdism]

@sad good research, thanks for posting the logs :D I'm going to keep digging and see if I can come up with something.

[sad]

@sad good research, thanks for posting the logs :D I'm going to keep digging and see if I can come up with something.

Looking forward to it as I'm a big fan of this project! Thanks for all your hard work. If there's anything further I can do (including testing), just let me know. You can also reach me at reid#0001 on Discord.

[Sean-Der]

@nurdism One thing you could do is run Neko in ICE Lite mode. ICE Lite is a less complicated setup, and basically just has the server do a pong instead of doing all the ICE logic.

By default Pion is a full ICE Agent, great if you have two peers behind NATs. However since Neko is always directly accessible (since you need that for HTTP) traffic, would I try doing this mode! Then you can also drop the STUN server from Neko.

You can enable ICE Lite here then have users set the IP address that neko is accessible at via this

You need to do ^ because the server doesn't have a way to find out its public IP by default. Maybe you could add something on Neko that goes and fetches the Public IP (and then calls SetNAT1To1IPs)

[nurdism]

@Sean-Der thank you so much for looking into this.

[nurdism]

ok so I have changed to ICE Lite and did some testing and it seems to work, I've pushed the containers to docker hub. Test them out, if you're still having issues try clearing your cache, if you're using cloud flair clear the cache on cloud flair's side too

[sad]

ok so I have changed to ICE Lite and did some testing and it seems to work, I've pushed the containers to docker hub. Test them out, if you're still having issues try clearing your cache, if you're using cloud flair clear the cache on cloud flair's side too

All fixed for me, thank you.

[nurdism]

awesomeness @Penagwin @danielpmc @xiya233 can you try the new version and see if this is working for you, I'll keep this open for a few days if no reply.

[Sean-Der]

That is awesome, nice work @nurdism getting that in so fast! Such an amazing project, and you are closing out issues so quickly I wish I was as effective :)

[nurdism]

I'm going to close this issue now, if this still continues to be an issue open a new issue and we'll continue from there.