Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node v18.15.0 and higher ignore --max-http-header-size #47246

Closed
havran opened this issue Mar 24, 2023 · 12 comments · Fixed by nodejs/undici#2234
Closed

Node v18.15.0 and higher ignore --max-http-header-size #47246

havran opened this issue Mar 24, 2023 · 12 comments · Fixed by nodejs/undici#2234
Labels
http Issues or PRs related to the http subsystem.

Comments

@havran
Copy link

havran commented Mar 24, 2023

Version

v18.15.0 and v19.8.1

Platform

Linux FESK-LTP0007 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux (Ubuntu 22.04)

Subsystem

No response

What steps will reproduce the bug?

I use urql in my NextJS application, to communicate with backend CMS. For some query CMS send extra large headers with response (around 22kb).

I use node 18 as runner for project where I use URQL library to GraphQL request to backend CMS, but there I get console error:

[newro-15509] Network Error! CombinedError: [Network] fetch failed
    at makeErrorResult (file:///data/***/node_modules/@urql/core/dist/urql-core-chunk.mjs:284:10)
    at file:///data/***/node_modules/@urql/core/dist/urql-core-chunk.mjs:462:15 {
  graphQLErrors: [],
  networkError: TypeError: fetch failed
      at Object.fetch (node:internal/deps/undici/undici:11457:11) {
    cause: HeadersOverflowError: Headers Overflow Error
        at Parser.trackHeader (node:internal/deps/undici/undici:9453:37)
        at Parser.onHeaderValue (node:internal/deps/undici/undici:9448:14)
        at wasm_on_header_value (node:internal/deps/undici/undici:9257:34)
        at wasm://wasm/00036ac6:wasm-function[49]:0xf91
        at wasm://wasm/00036ac6:wasm-function[68]:0x7316
        at wasm://wasm/00036ac6:wasm-function[67]:0x154d
        at wasm://wasm/00036ac6:wasm-function[21]:0x552
        at Parser.execute (node:internal/deps/undici/undici:9373:26)
        at Parser.readMore (node:internal/deps/undici/undici:9352:16)
        at Socket.onSocketReadable (node:internal/deps/undici/undici:9660:14) {
      code: 'UND_ERR_HEADERS_OVERFLOW'
    }
  },
  response: undefined
}

I investigate and I found solution to set http header size by using environment: NODE_OPTIONS=--max-http-header-size=32768

This seems to be ignored for Node version 18 and higher (I also try node 19). On Node 16, this environment variable solve error.

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior? Why is that the expected behavior?

No response

What do you see instead?

I see error message:

[newro-15509] Network Error! CombinedError: [Network] fetch failed
    at makeErrorResult (file:///data/***/node_modules/@urql/core/dist/urql-core-chunk.mjs:284:10)
    at file:///data/***/node_modules/@urql/core/dist/urql-core-chunk.mjs:462:15 {
  graphQLErrors: [],
  networkError: TypeError: fetch failed
      at Object.fetch (node:internal/deps/undici/undici:11457:11) {
    cause: HeadersOverflowError: Headers Overflow Error
        at Parser.trackHeader (node:internal/deps/undici/undici:9453:37)
        at Parser.onHeaderValue (node:internal/deps/undici/undici:9448:14)
        at wasm_on_header_value (node:internal/deps/undici/undici:9257:34)
        at wasm://wasm/00036ac6:wasm-function[49]:0xf91
        at wasm://wasm/00036ac6:wasm-function[68]:0x7316
        at wasm://wasm/00036ac6:wasm-function[67]:0x154d
        at wasm://wasm/00036ac6:wasm-function[21]:0x552
        at Parser.execute (node:internal/deps/undici/undici:9373:26)
        at Parser.readMore (node:internal/deps/undici/undici:9352:16)
        at Socket.onSocketReadable (node:internal/deps/undici/undici:9660:14) {
      code: 'UND_ERR_HEADERS_OVERFLOW'
    }
  },
  response: undefined
}

Additional information

No response

@deokjinkim deokjinkim changed the title Node v18.15.0 and gigher ignore --max-http-header-size Node v18.15.0 and higher ignore --max-http-header-size Mar 24, 2023
@deokjinkim deokjinkim added the http Issues or PRs related to the http subsystem. label Mar 24, 2023
@bnoordhuis
Copy link
Member

Can you include a test case? No third-party modules, just built-in modules.

I'm reasonably sure the option gets passed to undici (the new http library) because we have tests that exercise the flag.

@havran
Copy link
Author

havran commented Mar 25, 2023

Ok, I just create wery simple HTTP server:

# server.js

const http = require("http");

const host = 'localhost';
const port = 8000;

const requestListener = function (req, res) {
    res.setHeader("Content-Type", "application/json");
    res.setHeader("X-FakeSize", new Array(20000).join('a'));
    res.writeHead(200);
    res.end("My first server!");
};

const server = http.createServer(requestListener);
server.listen(port, host, () => {
    console.log(`Server is running on http://${host}:${port}`);
});

I run server node server.js and in another window I run NODE_OPTIONS=--max-http-header-size=32768 node and on prompt I use simple fetch request:

$ NODE_OPTIONS=--max-http-header-size=32768 node
Welcome to Node.js v18.15.0.
Type ".help" for more information.
> fetch('http://localhost:8000').then(res => console.log(res));
Promise {
  <pending>,
  [Symbol(async_id_symbol)]: 60,
  [Symbol(trigger_async_id_symbol)]: 27
}
> Uncaught TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11413:11) {
  cause: HeadersOverflowError: Headers Overflow Error
      at Parser.trackHeader (node:internal/deps/undici/undici:9411:37)
      at Parser.onHeaderValue (node:internal/deps/undici/undici:9406:14)
      at wasm_on_header_value (node:internal/deps/undici/undici:9215:34)
      at wasm://wasm/00036ac6:wasm-function[49]:0xf91
      at wasm://wasm/00036ac6:wasm-function[68]:0x7316
      at wasm://wasm/00036ac6:wasm-function[67]:0x154d
      at wasm://wasm/00036ac6:wasm-function[21]:0x552
      at Parser.execute (node:internal/deps/undici/undici:9331:26)
      at Parser.readMore (node:internal/deps/undici/undici:9310:16)
      at Socket.onSocketReadable (node:internal/deps/undici/undici:9618:14) {
    code: 'UND_ERR_HEADERS_OVERFLOW'
  }
}
>

Node version used:

$ node --version
v18.15.0

os

 neofetch
            .-/+oossssoo+/-.               havran@SGC
        `:+ssssssssssssssssss+:`           ----------
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 20.04.5 LTS on Windows 10 x86_64
    .ossssssssssssssssssdMMMNysssso.       Kernel: 5.15.90.1-microsoft-standard-WSL2
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Uptime: 23 mins
  +ssssssssshmydMMMMMMMNddddyssssssss+     Packages: 1234 (dpkg), 19 (brew)
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Shell: bash 5.0.17
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Theme: Adwaita [GTK3]
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Icons: Adwaita [GTK3]
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   Terminal: Relay(2291)
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: AMD Ryzen 7 1700 (8) @ 2.994GHz
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   GPU: 43ca:00:00.0 Microsoft Corporation Device 008e
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Memory: 883MiB / 7945MiB
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/
  +sssssssssdmydMMMMMMMMddddyssssssss+
   /ssssssssssshdmNNNNmyNMMMMhssssss/
    .ossssssssssssssssssdMMMNysssso.
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.

@havran
Copy link
Author

havran commented Mar 25, 2023

I also attach image where I test node --max-http-header-size=32768, seems it is works, http.maxHeaderSize is set correct, but fetch request still show UND_ERR_HEADERS_OVERFLOW:

image

@climba03003
Copy link
Contributor

climba03003 commented Mar 25, 2023

undici does not use http.maxHeaderSize as the default.
It has it's own default value and you need to change it yourself.

https://github.com/nodejs/undici/blob/e6fc80f809d1217814c044f52ed40ef13f21e43c/lib/client.js#L225

You can change the behavior by

const { Agent, setGlobalDispatcher } = require('undici')

setGlobalDispatcher(new Agent({
  maxHeaderSize: http.maxHeaderSize
}))

Full code that would work for your usage.

const http = require("http");
const { fetch, Agent, setGlobalDispatcher } = require('undici')

setGlobalDispatcher(new Agent({
  maxHeaderSize: http.maxHeaderSize
}))

const host = 'localhost';
const port = 8000;

const requestListener = function (req, res) {
    res.setHeader("Content-Type", "application/json");
    res.setHeader("X-FakeSize", new Array(20000).join('a'));
    res.writeHead(200);
    res.end("My first server!");
};

const server = http.createServer(requestListener);
server.listen(port, host, () => {
    console.log(`Server is running on http://${host}:${port}`);

    fetch(`http://${host}:${port}`)
      .then(function() {
        console.log('normal')
        process.exit(0)
      })
      .catch(function(err) {
        console.log('error')
        process.exit(1)
      })
});

@bnoordhuis
Copy link
Member

undici does not use http.maxHeaderSize as the default.

@nodejs/http if ☝️is true, isn't that a bug?

@mcollina
Copy link
Member

Given it's a read-only value, I think undici can use that value as the default when creating the default Agent.

Shall we move this issue to undici?

@havran
Copy link
Author

havran commented Mar 26, 2023

IMHO this behaviour should be same than in older node versions. Also when user use frameworks like NextJS, then is unexpected when there is no simple way change this value.

@adoprog
Copy link

adoprog commented Aug 23, 2023

Just got this error in NextJS after upgrading to its recent version. Is there any way to apply the workaround to it?

@balazsorban44
Copy link

balazsorban44 commented Aug 31, 2023

My two cents here, @mcollina's #47246 (comment) makes a lot of sense to me. In newer versions of Node.js, I consider undici as an implementation detail of globalThis.fetch, and any Node.js flag should configure fetch accordingly.

I don't mind if it's a different flag, even something like --undici-max-http-header-size, but it should be configurable without importing undici.

@Ethan-Arrowood
Copy link
Contributor

For this particular problem, users can use setGlobalDispatcher to modify this value and it should work. But I agree that since that method is not documented well; undici should start respecting more of Node's http flags/options.

It seems @mcollina is also in agreement; so lets ship it 🚢

@climba03003
Copy link
Contributor

I believe it just the matter of will someone willing to open PR to update undici default value to http.maxHeaderSize.

@balazsorban44
Copy link

Hi everyone, this has shipped in Node.js, you can upgrade, and will be able to set this flag, I believe the issue can be closed @havran.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
http Issues or PRs related to the http subsystem.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants