Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ESP8266WifiServer timeout when requesting through external IP #8853

Open
nbaztec opened this issue Feb 12, 2023 · 8 comments
Open

ESP8266WifiServer timeout when requesting through external IP #8853

nbaztec opened this issue Feb 12, 2023 · 8 comments

Comments

@nbaztec
Copy link

nbaztec commented Feb 12, 2023

I have a basic ESP8266WebServer that sends roughly 1000 characters worth of response via send(). Everything works fine when the traffic is over the local IPv4 address 192.168.xx.xx.

However, I have set up port forwarding over my external IP to be directed to the NodeMCU device, and when the request is made over the external IP 87.xx.xx.xx via a device connected to the local network, the server times out during send() operation. Upon closer inspection, it seems that ClientContext::availableForWrite() / tcp_sndbuf returns 0 in this case. Increasing the timeout to 10s didn't help. Seems the buffer is never available.

As an additional step, I set up a VPN on the request device to make the request device's IP as 105.xx.xx.xx and made the same request. This time ClientContext::availableForWrite() / tcp_sndbuf returned a non-zero value and the request was successful.

Any ideas why the tcp_sndbuf would not work when a request is made by a device connected on the same network but via external IP that is port forwarded to my NodeMCU?

@d-a-v
Copy link
Collaborator

d-a-v commented Feb 12, 2023

You can try and play with the embedded Netdump library which will eventually show the differences between your port redirect and your vpn setup.

@nbaztec
Copy link
Author

nbaztec commented Feb 14, 2023

That sketch crashes intermittently with (decoded exception)

Exception 28: LoadProhibited: A load referenced a page mapped with an attribute that does not permit loads
PC: 0x4021ff24: tcp_output at core/tcp_out.c line 1390
EXCVADDR: 0x0000000c

Decoding stack results
0x402250e3: ip_chksum_pseudo at core/inet_chksum.c line 392
0x4021fe3e: tcp_output at core/tcp_out.c line 1621
0x4020ac81: ClientContext::_write_from_source(char const*, unsigned int) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/libraries/ESP8266WiFi/src/include/ClientContext.h line 474
0x4020d844: Stream::SendGenericPeekBuffer(Print*, int, int, unsigned int) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/Print.h line 63
0x4020d064: Print::println(int, int) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/Print.cpp line 238
0x40207ac4: esp8266webserver::ESP8266WebServerTemplate ::send(int, char const*, Stream*, unsigned int) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/Stream.h line 64
0x4020dacb: String::changeBuffer(unsigned int) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/WString.h line 328
0x40207be7: esp8266webserver::ESP8266WebServerTemplate ::send(int, char const*, String const&) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/WString.h line 115
0x4020db5f: String::reserve(unsigned int) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/WString.cpp line 192
0x4020de01: String::String(unsigned char, unsigned char) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/WString.cpp line 44
0x40207d5a: std::_Function_handler   >::_M_invoke(const std::_Any_data &) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/WString.h line 115
0x40205e95: esp8266webserver::FunctionRequestHandler ::handle(esp8266webserver::ESP8266WebServerTemplate &, HTTPMethod, String const&) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/libraries/ESP8266WebServer/src/detail/RequestHandlersImpl.h line 62
0x40207c34: esp8266webserver::ESP8266WebServerTemplate ::_handleRequest() at /home/nish/.arduino15/packages/esp8266/tools/xtensa-lx106-elf-gcc/3.1.0-gcc10.3-e5f9fec/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h line 498
0x4020b3a0: WiFiServer::accept() at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/libraries/ESP8266WiFi/src/WiFiServer.cpp line 147
0x40208172: esp8266webserver::ESP8266WebServerTemplate ::handleClient() at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/libraries/ESP8266WebServer/src/ESP8266WebServer-impl.h line 282
0x4020ee2c: loop_wrapper() at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/core_esp8266_main.cpp line 258

raw:

11:37:16.130 ->     5780 0 out HTTP 192.168.178.27>192.168.178.76 80:44384 [PA] len: 114 seq: 6511, ack: 3693242410, wnd: 1696 
11:37:16.130 ->            H .P.`...o."h*P...$...
11:37:16.130 ->            D HTTP/1.1 200 OK..Content-Type: text/html..Content-Length: 50..Connection: keep-a
11:37:16.163 ->            D live..Ke    5798 0 in  HTTP 192.168.178.76>192.168.178.27 44384:80 [A] len: 0 seq: 3693242410, ack: 6625, wnd: 64126 
11:37:16.163 ->            H .`.P."h*....P..~....
11:37:16.163 ->     5811 0 out HTTP 192.168.178.27>192.168.178.76 80:44384 [A] len: 0 seq: 6511, ack: 3693242410, wnd: 1696 
11:37:16.163 ->            H .P.`...o."h*P.......
11:37:16.163 ->     5824 0 out HTTP 192.168.178.27>192.168.178.76 80:44384 [PA] len: 114 seq: 6511, ack: 3693242410, wnd: 1696 
11:37:16.196 ->            H .P.`...o."h*P...$...
11:37:16.196 ->            D HTTP/1.1 200 OK..Content-Type: text/html..Content-Length: 50..Connection: keep-a
11:37:16.196 ->            D live..Keep-Alive: timeout=2000....
11:37:16.196 -> ep-Alive: timeout=2000....
11:37:16.196 -> 
11:37:16.196 -> --------------- CUT HERE FOR EXCEPTION DECODER ---------------
11:37:16.196 -> 
11:37:16.196 -> Exception (28):
11:37:16.229 -> epc1=0x4021ff24 epc2=0x00000000 epc3=0x00000000 excvaddr=0x0000000c depc=0x00000000
11:37:16.229 -> 
11:37:16.229 -> >>>stack>>>
11:37:16.229 -> 
11:37:16.229 -> ctx: cont
11:37:16.229 -> sp: 3ffffbc0 end: 3fffffd0 offset: 0190
11:37:16.229 -> 3ffffd50:  3ffeee84 00000000 00000000 402250e3  
11:37:16.229 -> 3ffffd60:  00000001 00000072 3fff01e4 4021fe3e  
11:37:16.229 -> 3ffffd70:  3ffeee84 3fff034c 3fff0164 00000000  
11:37:16.229 -> 3ffffd80:  00000860 3fff0040 0000196f 00000072  
11:37:16.262 -> 3ffffd90:  3ffffd00 00000001 3ffffe54 00000002  
11:37:16.262 -> 3ffffda0:  00000001 00000000 3ffefe54 4020ac81  
11:37:16.262 -> 3ffffdb0:  00000072 00000000 3ffeea1c 00000072  
11:37:16.262 -> 3ffffdc0:  3ffee89c 00000007 3ffeea1c 00000072  
11:37:16.262 -> 3ffffdd0:  3ffee89c 3ffeea1c 3ffffe30 4020d844  
11:37:16.262 -> 3ffffde0:  04c4b400 5611d6d9 3ffeea00 4020d064  
11:37:16.262 -> 3ffffdf0:  00000000 ffffffff 00000000 ffffffff  
11:37:16.262 -> 3ffffe00:  00000000 ffffffff 00000000 3fff01e4  
11:37:16.295 -> 3ffffe10:  3ffe87f0 00000000 3fffff08 000000c8  
11:37:16.295 -> 3ffffe20:  3ffffe90 3ffee868 00000032 40207ac4  
11:37:16.295 -> 3ffffe30:  4021233c 00000000 000003e8 00000043  
11:37:16.295 -> 3ffffe40:  00000000 3fff01e4 00000072 3fff0101  
11:37:16.295 -> 3ffffe50:  00000000 3fff01e4 0072007f 89000020  
11:37:16.295 -> 3ffffe60:  3ffffeb4 00000040 3ffffeec 4020dacb  
11:37:16.295 -> 3ffffe70:  3fff011c 00000009 3ffffeb4 3ffe8884  
11:37:16.328 -> 3ffffe80:  3fff011c 000000c8 3ffee868 40207be7  
11:37:16.328 -> 3ffffe90:  4021233c 00000000 000003e8 4020db5f  
11:37:16.328 -> 3ffffea0:  00000000 3fff011c 00000032 4020de01  
11:37:16.328 -> 3ffffeb0:  00000000 74786574 6d74682f 0900006c  
11:37:16.328 -> 3ffffec0:  3ffe8847 3ffe8857 3ffffeec 3ffeeadc  
11:37:16.328 -> 3ffffed0:  3fffdad0 00000001 3ffefa44 40207d5a  
11:37:16.328 -> 3ffffee0:  00000000 002d002f 00000000 00000000  
11:37:16.328 -> 3ffffef0:  0032003f 00000000 3fff011c 0032003f  
11:37:16.361 -> 3fffff00:  81000000 00000000 00000000 3ffeeadc  
11:37:16.361 -> 3fffff10:  3fffdad0 3ffee8c0 3ffefa44 40205e95  
11:37:16.361 -> 3fffff20:  00000000 3ffee8c0 3ffee868 40207c34  
11:37:16.361 -> 3fffff30:  3fffffa0 00000000 00000000 000e000f  
11:37:16.361 -> 3fffff40:  00000000 3ffeea1c 3ffee87c 4020b3a0  
11:37:16.361 -> 3fffff50:  00000000 0000168c 00000000 00000001  
11:37:16.361 -> 3fffff60:  00000000 00000000 3ffee868 3ffeeadc  
11:37:16.394 -> 3fffff70:  3fffdad0 3ffee89c 3ffee868 40208172  
11:37:16.394 -> 3fffff80:  40212448 00000000 00001388 3ffeeadc  
11:37:16.394 -> 3fffff90:  00000000 00000000 3ffefe54 00000000  
11:37:16.394 -> 3fffffa0:  3fffdad0 00000000 3ffeeab0 3ffeeadc  
11:37:16.394 -> 3fffffb0:  3fffdad0 00000000 3ffeeab0 4020ee2c  
11:37:16.394 -> 3fffffc0:  feefeffe feefeffe 3fffdab0 40100d79  
11:37:16.394 -> <<<stack<<<
11:37:16.394 -> 
11:37:16.394 -> --------------- CUT HERE FOR EXCEPTION DECODER ---------------
11:37:16.427 -> 
11:37:16.427 ->  ets Jan  8 2013,rst cause:2, boot mode:(3,6)
11:37:16.427 -> 
11:37:16.427 -> load 0x4010f000, len 3424, room 16 
11:37:16.427 -> tail 0
11:37:16.427 -> chksum 0x2e
11:37:16.427 -> load 0x3fff20b8, len 40, room 8 
11:37:16.427 -> tail 0
11:37:16.427 -> chksum 0x2b
11:37:16.427 -> csum 0x2b

I tried to mimize the sketch to only contain:

 webServer.on("/req", []() {
    static int rq = 0;
    String a = "<h1>You are connected, Number of requests = " + String(rq++) + "</h1>";
    webServer.send(200, "text/html", a);
  });

  webServer.on("/reset", []() {
    nd.reset();
    tracefile.close();
    tcpServer.close();
    webServer.send(200, "text.html", "<h1>Netdump session reset</h1>");
  });

//  webServer.serveStatic("/", *filesystem, "/");
  webServer.begin();

  startSerial(SerialOption::HTTPChar); 

@nbaztec
Copy link
Author

nbaztec commented Feb 14, 2023

Even through the crashes I was able to analyze the tcpdump on the client side and the nodemcu side:

It seems that over external IP the ACKs for the HTTP body are not reaching the NodeMCU, but over a VPN they do. I'm still confused so as why it is like this.

External IP (hangs on body receive)

13:14:06.077418 IP nodemcu.network.9980 > home-pc.local.48394: Flags [P.], seq 6511:6627, ack 2424900121, win 2060, length 116
13:14:06.077465 IP home-pc.local.48394 > nodemcu.network.9980: Flags [R], seq 2424900121, win 0, length 0
13:14:06.952399 IP home-pc.local.41472 > nodemcu.network.9980: Flags [S], seq 110092386, win 64240, options [mss 1460,sackOK,TS val 3588784746 ecr 0,nop,wscale 7], length 0
13:14:07.099363 IP nodemcu.network.9980 > home-pc.local.41472: Flags [S.], seq 6552, ack 110092387, win 2144, options [mss 536,nop,nop,sackOK], length 0
13:14:07.099388 IP home-pc.local.41472 > nodemcu.network.9980: Flags [.], ack 1, win 64240, length 0
13:14:07.099430 IP home-pc.local.41472 > nodemcu.network.9980: Flags [P.], seq 1:84, ack 1, win 64240, length 83
13:14:07.142672 IP nodemcu.network.9980 > home-pc.local.41472: Flags [.], ack 84, win 2061, length 0
13:14:07.155944 IP nodemcu.network.9980 > home-pc.local.41472: Flags [P.], seq 1:117, ack 84, win 2061, length 116
13:14:07.155954 IP home-pc.local.41472 > nodemcu.network.9980: Flags [.], ack 117, win 64124, length 0
13:14:07.217393 IP nodemcu.network.9980 > home-pc.local.41472: Flags [.], ack 84, win 2061, length 0
13:14:07.217407 IP home-pc.local.41472 > nodemcu.network.9980: Flags [.], ack 117, win 64124, length 0
13:15:07.639824 IP home-pc.local.41472 > nodemcu.network.9980: Flags [.], ack 117, win 64124, length 0
13:15:07.774334 IP nodemcu.network.9980 > home-pc.local.41472: Flags [R.], seq 117, ack 83, win 24584, length 0

External IP over VPN (good)

13:04:27.819146 IP nish-pc.53648 > nodemcu.network.9980: Flags [F.], seq 2573049806, ack 8445, win 64116, length 0
13:04:27.881189 IP nodemcu.network.9980 > nish-pc.53648: Flags [R.], seq 1, ack 1, win 24584, length 0
13:04:32.129136 IP nish-pc.59006 > nodemcu.network.9980: Flags [S], seq 33849241, win 64768, options [mss 1408,sackOK,TS val 2649173485 ecr 0,nop,wscale 7], length 0
13:04:32.283496 IP nodemcu.network.9980 > nish-pc.59006: Flags [S.], seq 6510, ack 33849242, win 2144, options [mss 536,nop,nop,sackOK], length 0
13:04:32.283524 IP nish-pc.59006 > nodemcu.network.9980: Flags [.], ack 1, win 64768, length 0
13:04:32.283565 IP nish-pc.59006 > nodemcu.network.9980: Flags [P.], seq 1:84, ack 1, win 64768, length 83
13:04:32.357765 IP nodemcu.network.9980 > nish-pc.59006: Flags [P.], seq 1:117, ack 84, win 2061, length 116
13:04:32.357795 IP nish-pc.59006 > nodemcu.network.9980: Flags [.], ack 117, win 64652, length 0
13:04:32.392284 IP nodemcu.network.9980 > nish-pc.59006: Flags [.], seq 117:653, ack 84, win 2061, length 536
13:04:32.392315 IP nish-pc.59006 > nodemcu.network.9980: Flags [.], ack 653, win 64116, length 0
13:04:32.483747 IP nodemcu.network.9980 > nish-pc.59006: Flags [.], seq 117:653, ack 84, win 2061, length 536
13:04:32.483764 IP nish-pc.59006 > nodemcu.network.9980: Flags [.], ack 653, win 64116, options [nop,nop,sack 1 {117:653}], length 0
13:04:32.555288 IP nodemcu.network.9980 > nish-pc.59006: Flags [P.], seq 653:1073, ack 84, win 2061, length 420
13:04:32.555308 IP nish-pc.59006 > nodemcu.network.9980: Flags [.], ack 1073, win 64116, length 0
13:05:34.203822 IP nish-pc.59006 > nodemcu.network.9980: Flags [.], ack 1073, win 64116, length 0
13:05:34.258016 IP nodemcu.network.9980 > nish-pc.59006: Flags [R.], seq 1073, ack 83, win 24584, length 0

@nbaztec
Copy link
Author

nbaztec commented Feb 16, 2023

Note that I have a RPi also on the same network behind the same kind of port-forwarding that works just fine with the external IP, so I doubt it's anything network related.

@TD-er
Copy link
Contributor

TD-er commented Feb 27, 2023

Maybe timeout related?
Is the connection via the external IP rather slow, or at least relatively high latency?
Does the ESP have a proper gateway configured?

@JacekKac
Copy link

JacekKac commented Aug 8, 2023

I can confim similar problem. Connecting directly from lan works, but when try to access my esp using port forward / vpn it's not working.
VPN client with browser -> VPN server on raspberry pi -> port redirect from raspberry to esp (raspberry and esp in the same subnet).

Page is loading very slowly, and eating whole ram of esp device

@mcspr
Copy link
Collaborator

mcspr commented Sep 19, 2023

(to the deleted message and the above)

Might be useful to also record what network & firewall configuration used to pass the packets through the network.
Also, how 'high bandwidth' version bares in comparison. iirc old lwip v1 had a very big problem when using mixed mss 536 vs. 1460 values, like in the log above

@JacekKac
Copy link

not sure if this will help you but building the firmware with andruino framework 4.2.1 solved the issue, the issue was present when using 2.6.3.
lwip v1vs2 make no difference.
In my case there was a problem with memory leak - the browser requested the site, and nothing happens, only few packets were send.. and in few minutes my esp devices runns out of memory and crushes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants