Default value for sdkconfig parm CONFIG_HTTPD_MAX_REQ_HDR_LEN=512 too small in Arduino

[craigdetter]

Using a form method="post" from a web page to the ESP32 device (using SoftAP, and web_server built from file_server example).
<form id="setup-form" action="/domain/setup_form_content" method="post"> ... </form>

Causes error condition with message:
"Header fields are too long for server to interpret"

The header length of a Wireshark trace (see below) shows 554 byte long header being sent by Google Chrome.

Hardware:

Board: ESP32 Adafruit Huzzah32
Core Installation version: v5.0-dev-553-g42cce06704-dirty
IDE name: Arduino IDE or VSCode
Flash Frequency: N/A
PSRAM enabled: N/A
Upload Speed: N/A
Computer OS: Mac OSX Catalina latest with Google Chrome latest version

Description:

Describe your problem here

The default in build for the sdkconfig configuration parameter is:
CONFIG_HTTPD_MAX_REQ_HDR_LEN=512 (bytes).
It needs to be set at least 768 bytes, but 1024 would be better.

This is a problem for all development platforms and ESP32 targets, but a real burden for customers using Arduino because the binary isn't easy to recompile.

On Google Chrome latest version running on MacOS on the latest Catalina OS:
using Wireshark trace - shows the "post" header is 554 bytes long (not incl. CR/LF)

Host: 192.168.4.1\r\n
Connection: keep-alive\r\n
Content-Length: 95\r\n
Cache-Control: max-age=0\r\n
Upgrade-Insecure-Requests: 1\r\n
Origin: http://192.168.4.1\r\n
Content-Type: application/x-www-form-urlencoded\r\n
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36\r\n
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9\r\n
Referer: http://192.168.4.1/\r\n
Accept-Encoding: gzip, deflate\r\n
Accept-Language: en-US,en;q=0.9,pl;q=0.8\r\n
\r\n

On Chrome "get" header is at least 478 bytes ....

_Host: 192.168.4.1\r\n
Connection: keep-alive\r\n
Upgrade-Insecure-Requests: 1\r\n
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36\r\n
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9\r\n
Referer: http://192.168.4.1/\r\n
Accept-Encoding: gzip, deflate\r\n
Accept-Language: en-US,en;q=0.9,pl;q=0.8\r\n
\r\n

Wireshark Trace - Packet 218 ("post" from Chrome to ESP32):

Wireshark Trace - Packet 219 (response from ESP32 to Chrome):

_

[atanisoft]

@craigdetter Submit a PR over on https://github.com/espressif/esp32-arduino-lib-builder to modify the value(s) for this and if accepted it will make it's way over here.

[VojtechBartoska]

relates to espressif/esp32-arduino-lib-builder#52

[LenShustek]

This problem has been talked about for years in various forums. Will it ever get fixed?

Until this is fixed, a server using arduino-esp32 that implements any web page buttons will fail when the client browser is Chrome or Edge. It works with Firefox and the (obsolete) Internet Explorer, but a that's not a requirement you can impose on users. Please help, or we need to abandon the Arduino IDE!

[me-no-dev]

nice... this issue is here since December. And is already fixed in the lib-builder. will land here shortly.

[LenShustek]

Much appreciated, thanks!
But just for the record, it had been reported as early as July 2019; see #2983.

[LenShustek]

It appears to be fixed in idf-release/v4.4, but can you give us newbies some guidance about using that on Windows?
The installation instructions say to use the Arduino IDE Boards Manager, but it needs a link to a file like package_esp32_index.json that I can't find in the V4.4 repo.
When I instead try to clone the repo in GitHub desktop I get Authentication failed for github.com/espressif/arduino-esp32/tree, and my username and password doesn't cut it.

[atanisoft]

can you give us newbies some guidance about using that on Windows?

https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html

All you need to do is use the correct boards manager URL in Arduino IDE and install v2.0.x and it should work without issues. You don't need GitHub Desktop or other tools unless you need bleeding edge (not recommended in most cases).

[LenShustek]

Doesn't that give me V2.0.2 from Dec 23, 2021, and not the idf-release/v4.4 from 6 days ago that is 2 commits ahead of master?

[atanisoft]

Ahh, yes that is true. The changes on idf-release/v4.4 are temporary in nature and will be removed once PR #6174 has been merged to master. It would be best to wait for that to be merged before trying to pick up the updated version.

[LenShustek]

Ok, thanks; I'll keep waiting. I tried all sort of bizarre ways to get the Arduino IDE to be happy with the idf-release/v4.4 that I could download, but without success.

[LenShustek]

Thanks. It seems complicated, but I might give it a go. I do need the updated sdkconfig file to get the benefit of the new value for CONFIG_HTTPD_MAX_REQ_HDR_LEN.

But although I'm a good programmer with 50+ years of experience, my skill level with modern toolchains is weak. I may well fail and have to wait for the changes to propagate to the master.

[atanisoft]

@LenShustek it is available now on master, if you are using Arduino IDE you would need to follow the instructions for git clone as documented here: https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html#windows (scroll down for Linux or MacOS). You will not be able to use the boards manager approach until the next release.

[LenShustek]

I'm confused. https://github.com/espressif/arduino-esp32 shows the "latest release" as from Dec 23, 2021. And if I look at blob/master/tools/sdk/esp32/sdkconfig in that repository it still shows the old value for CONFIG_HTTPD_MAX_REQ_HDR_LEN.

[atanisoft]

Sorry you are right. master is still using 512, you can pull the updated values from branch idf-release/v4.4.

A minor update to the previously linked instructions:
Replace git clone https://github.com/espressif/arduino-esp32.git esp32 with
git clone --branch idf-release/v4.4 https://github.com/espressif/arduino-esp32.git esp32

If you are using a graphical Git client you should be able to specify the branch to clone as idf-release/v4.4