Chrome remote debugging expects host header

[grantstephens]

This is a slightly weird one and I don't think it is strictly speaking a puppeteer bug but thought I'd raise it here anyway.
Since version 1.2, when trying to connect to chrome via the remote debugging interface a host header must be specified.
This is not a problem in puppeteer per se, but we use python to get the ws address and so the python requests line now has to include headers={'Host':''} otherwise chrome returns an error: Host header is specified and is not an IP address or localhost.
I assume that puppeteer is setting this header in the background somewhere or chrome doesn't care anymore once the request comes in via the websocket. I also could not find anything documented about this change, but I'm assuming that is has something to do with the .dev domain changes to HTTPS.

[gios]

I was testing this bug with previous puppeteer version and had the same error. Maybe this bug is related to chrome because I used v67 chrome. With v66 chrome everything is OK.

[ermyril]

Got the same error with a Chromium 67.0.3383.0 while trying to get a docker-container to work with my laravel application, curl {host with chrome-headless} gives me that

Will try to get version 66 by now

[aslushnikov]

@grantstephens thanks for reaching to us.

This was recently introduced in https://chromium-review.googlesource.com/c/chromium/src/+/952522. The error is issued if the host header is set to non-localhost and non-IP address.

What's your python library? It looks like it sets a non-trivial host header, so you have to override it with empty value.

cc @pavelfeldman

[grantstephens]

I'm using requests, which AFAIK does not send a host header by default, but I may be wrong. Either way- couldn't find any changelog for it suddenly happening (From the chrome side) so thought I'd just put it here in case anybody else was having a similar issue.
Maybe adding something in the documentation about it?

[hbakhtiyor]

and so what's the workaround of the issue?

[bean5]

I started getting this when I upgraded from version 65 to 66.

[bean5]

"The error is issued if the host header is set to non-localhost and non-IP address." Does this imply that the host header should use an IP address? Which header? Is this an error in a websites configuration or an error in puppeteer?

[cyrus-and]

@aslushnikov @pavelfeldman the problem of setting the host header to the empty value is that it will break the webSocketDebuggerUrl field for the /json/list and /json/new endpoints:

$ curl -H host: localhost:9222/json/list                                                       
[ {
   "description": "",
   "devtoolsFrontendUrl": "/devtools/inspector.html?ws=/devtools/page/A4566872556EA7F94C344A8ABEA30A73",
   "id": "A4566872556EA7F94C344A8ABEA30A73",
   "title": "about:blank",
   "type": "page",
   "url": "about:blank",
   "webSocketDebuggerUrl": "ws:///devtools/page/A4566872556EA7F94C344A8ABEA30A73"
} ]

[itaylor]

I've been working around this issue by doing a dns lookup in the code that sets up puppeteer (running inside Docker for Mac) to call back to my host machine.

const endPoint = await changeUrlHostToUseIp(process.env.CHROME_WS_ENDPOINT);
browser = await puppeteer.connect({
  browserWSEndpoint: endPoint,
  // You may need the following if you use this browser to connect to a machine with self-signed certs
  // ignoreHTTPSErrors: true,
});

const lookupAsync = require('util').promisify(require('dns').lookup);

// Fix for https://github.com/GoogleChrome/puppeteer/issues/2242
async function changeUrlHostToUseIp(urlString) {
  const urlParsed = url.parse(urlString);
  const { address: hostIp } = await lookupAsync(urlParsed.hostname);
  delete urlParsed.host;
  urlParsed.hostname = hostIp;
  return url.format(urlParsed);
}

For reference, my CHROME_WS_ENDPOINT env var looks like: ws://host.docker.internal:9223/devtools/page/FF3312E555CDEADBC3759BF0DE46B72B

[bean5]

@itaylor thank you for your work-around. Before I go trying it, I figured I'd ask you: Does that work-around require set ignoreHTTPSErrors: true or is that just what you happened to be using?

[itaylor]

@bean5, it shouldn't require ignoreHTTPSErrors, that's just the configuration I'm using as I'm using this to test an app I work on, and it uses self-signed certs. I've edited the above to make that clear.

[pauldraper]

Wait, what is the fix?

[bean5]

I read wondering the same thing. Please re-open unless this is intended functionality.

[driq]

@aslushnikov Can you clarify why this ticket has been closed? Is the behaviour in described in this ticket intentional?

Please re-open or clarify

[pauldraper]

While we're waint to hear back (3 months and counting), @itaylor 's workaround is good.

[drzraf]

Issue is on the Chrome side, at https://bugs.chromium.org/p/chromium/issues/detail?id=813540
On the puppeteer side, it could be fixed by adding a clear-hostname flag to pass to the instance:
https://github.com/GoogleChrome/puppeteer/blob/f79d0a3be86281b481609f2f9ecf6208df32c1a7/lib/Launcher.js#L385
requestOptions := {"host": "localhost"}

[imShara]

@drzraf can you explain your solution?

[dvdotsenko]

Since this is on top of results when I google "Host header is specified and is not an IP address or localhost" + "puppeteer" let's leave crumbs for others:

= WHY

A security bug was filed against Chromium Dev Tools:
https://bugs.chromium.org/p/chromium/issues/detail?id=813540

Around v66 of Chromium this patch was introduced:
https://chromium-review.googlesource.com/c/chromium/src/+/952522

That patch inspects all HTTP (not WS) requests hitting Dev Tools endpoint and before doing anything else, validates it against:

bool RequestIsSafeToServe(const net::HttpServerRequestInfo& info) {
  // For browser-originating requests, serve only those that are coming from
  // pages loaded off localhost or fixed IPs.
  ...
  return url.HostIsIPAddress() || net::IsLocalHostname(url.host(), nullptr);
}

where

bool IsLocalHostname(const std::string& host) {
  return host == "localhost" || 
    host == "localhost.localdomain" || 
    base::EndsWith(host, ".localhost", base::CompareCase::SENSITIVE);
}

This means that any non-IP-address-looking hostname that is NOT localhost, *.localhost or localhost.localdomain will result in error "Host header is specified and is not an IP address or localhost" error.

= WHAT TO DO ABOUT IT

== Most reliable way to fix it

DNS resolve the hostname and feed that IP address to puppeteer instead of hostname.

(what @itaylor mentioned above, but in simpler, promise-ified DNS)

const puppeteer = require('puppeteer-core')
const dns = require('dns').promises

const port = '9222'
const hostname_dirty = 'host.docker.internal'
// This pipes the hostname through DNS lookup and resolves it to an IP address
const { address: hostname } = await dns.lookup(hostname_dirty)

const browser = await puppeteer.connect({
    browserURL: `http://${hostname}:${port}`
})

== Accidental Fixes / Shortcuts

Standard address for "host machine" on Docker is host.docker.internal, but on Docker for Mac and similar there is another, older alias - docker.for.mac.localhost. host.docker.internal will NOT work, while docker.for.mac.localhost will natively work.

Obviously aliasing headless chrome in your /etc/hosts under some *.localhost hostname will also work.