Fix "Parse Error" when performing HTTP requests

[flotwig]

• Closes Cypress is unable to load site in v3.5.0 due to "Parse Error" #5602

User facing changelog

• Fixed a regression in 3.5.0 where certain HTTP requests could fail with "Parse Error: Invalid header value char" or "Parse Error: Header overflow".

Additional details

• was originally caused by Node upgrade, a workaround became available with Electron 8.1.0 Electron 8.1.1 #6555
• this PR...
  • re-adds this reverted PR: Always pass NODE_OPTIONS with max-http-header-size #5452
  • reverts this PR: Use websockets to stub large XHR response bodies instead of headers #5525
  • and finishes up this PR: [WIP] Detect if NODE_OPTIONS are present in binary; if not, respawn #5492
  • the end result is that Electron is always launched with NODE_OPTIONS=--max-http-header-size=${1000 ** 2} --http-parser=legacy, which fixes the sources of the "Parse Error"s
  • some additional code is added to the response-middleware to suppress further "Parse Error" errors that still occur for invalid header values

---

Notes:

• "header overflow" will be fixed by There is no way to set --max-http-header-size in Electron 5.x electron/electron#20831 and adding a bigger max http header size
• able to reproduce by adding test to visit header with null char:

  cypress/packages/server/test/e2e/6_visit_spec.coffee

  Lines 72 to 85 in 390ad4a

  	## https://github.com/cypress-io/cypress/issues/5602
  	app.get "/invalid-header-char", (req, res) ->
  	## express/node may interfere if we just use res.setHeader
  	res.connection.write(
  	"""
  	HTTP/1.1 200 OK
  	Content-Type: text/html
  	Set-Cookie: foo=bar-#{String.fromCharCode(1)}-baz
  	foo
  	"""
  	)
  	res.connection.end()

• passing --http-parser=legacy in NODE_OPTIONS gets rid of the error
  • opened issue to enable this option in Electron: There is no way to set the --http-parser NODE_OPTION in a packaged app electron/electron#21693
• even with --http-parser=legacy, still fails, but in a different place:

  TypeError [ERR_INVALID_CHAR] [ERR_INVALID_CHAR]: Invalid character in header content ["x-foo"]     
    at ServerResponse.setHeader (_http_outgoing.js:467:3)
  

  • fixed this by catching ERR_INVALID_CHAR errors and manually doing the setHeader steps in that case:

    cypress/packages/proxy/lib/http/response-middleware.ts

    Lines 162 to 185 in 390ad4a

    	// run the original function - if an "invalid header char" error is raised,
    	// set the header manually. this way we can retain Node's original error behavior
    	try {
    	return originalSetHeader.call(this, k, v)
    	} catch (err) {
    	if (err.code !== 'ERR_INVALID_CHAR') {
    	throw err
    	}
    	debug('setHeader error ignored %o', { code: err.code, err })
    	if (!kOutHeaders) {
    	kOutHeaders = getKOutHeadersSymbol()
    	}
    	// https://github.com/nodejs/node/blob/42cce5a9d0fd905bf4ad7a2528c36572dfb8b5ad/lib/_http_outgoing.js#L483-L495
    	let headers = this[kOutHeaders]
    	if (!headers) {
    	this[kOutHeaders] = headers = Object.create(null)
    	}
    	headers[k.toLowerCase()] = [k, v]
    	}

  • gnarly code to find the kOutHeaders symbol, can't find a better way

How has the user experience changed?

PR Tasks

• Have tests been added/updated?
• Has the original issue been tagged with a release in ZenHub?
• [na] Has a PR for user-facing changes been opened in cypress-documentation?
• [na] Have API changes been updated in the type definitions?
• [na] Have new configuration options been added to the cypress.schema.json?

[cypress-bot]

Thanks for the contribution! Below are some guidelines Cypress uses when doing PR reviews.

• Please write [WIP] in the title of your Pull Request if your PR is not ready for review - someone will review your PR as soon as the [WIP] is removed.
• Please familiarize yourself with the PR Review Checklist and feel free to make updates on your PR based on these guidelines.

PR Review Checklist

If any of the following requirements can't be met, leave a comment in the review selecting 'Request changes', otherwise 'Approve'.

User Experience

• The feature/bugfix is self-documenting from within the product.
• The change provides the end user with a way to fix their problem (no dead ends).

Functionality

• The code works and performs its intended function with the correct logic.
• Performance has been factored in (for example, the code cleans up after itself to not cause memory leaks).
• The code guards against edge cases and invalid input and has tests to cover it.

Maintainability

• The code is readable (too many nested 'if's are a bad sign).
• Names used for variables, methods, etc, clearly describe their function.
• The code is easy to understood and there are relevant comments explaining.
• New algorithms are documented in the code with link(s) to external docs (flowcharts, w3c, chrome, firefox).
• There are comments containing link(s) to the addressed issue (in tests and code).

Quality

• The change does not reimplement code.
• There's not a module from the ecosystem that should be used instead.
• There is no redundant or duplicate code.
• There are no irrelevant comments left in the code.
• Tests are testing the code’s intended functionality in the best way possible.

Internal

• The original issue has been tagged with a release in ZenHub.

[cypress]

---

Test summary

3270 • 0 • 49 • 0

---

Run details

Project	cypress
Status	Passed
Commit	4060beb
Started	Mar 17, 2020 8:29 PM
Ended	Mar 17, 2020 8:36 PM
Duration	06:21 💡
OS	Linux Debian - 10.1
Browser	Firefox 74

View run in Cypress Dashboard ➡️

---

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

[CypressJoseph]

lgtm! some pretty arcane things here, but looks really solid 👍

just out of curiosity -- why do we use the legacy electron http parser? (is the idea just compliance with previous cypress behavior?)

[flotwig]

why do we use the legacy electron http parser? (is the idea just compliance with previous cypress behavior?)

@CypressJoseph yeah, it fixes the regression in Cypress, but imo it's not just about backwards-compatibility - Cypress should be a 100% transparent HTTP proxy, even if the HTTP traffic it's proxying is a little out-of-spec. Cypress should load pages the same as a normal browser does, and browsers are very tolerant. So when Node introduced a new, stricter HTTP parser that's on by default, it made the proxy start rejecting some previously-acceptable traffic. See this comment for technical info on the changes that made this necessary.