Referer header not working

[Rub3nCT]

add_header method is not working fine with Referer parameter, also tested with User-Agent and seems to be working fine.

$mech->add_header( Referer => $ref );

reload method is also crashing with ignoreCache option. Is working fine without ignoreCache option.

$mech->reload( ignoreCache => 1 );

Throws:

Uncaught exception from user code:
        Invalid parameters
        ignoreCache: boolean value expected
        -32602 at ... Chrome/DevToolsProtocol/Target.pm line 460

Test code attached comparing WWW::Mechanize and WWW::Mechanize::Chrome results.
test.zip

[Corion]

Thank you very much for reporting the issues!

I think Chrome since v63+ does not send the Referer header anymore.

I'll look at the ignoreCache method and add a proper test for that.

[Rub3nCT]

Maybe this will help with Referer header:

Starting from Chrome 79, request header modifications affect Cross-Origin Resource Sharing (CORS) checks. If modified headers for cross-origin requests do not meet the criteria, it will result in sending a CORS preflight to ask the server if such headers can be accepted. If you really need to modify headers in a way to violate the CORS protocol, you need to specify 'extraHeaders' in opt_extraInfoSpec. On the other hand, response header modifications do not work to deceive CORS checks. If you need to deceive the CORS protocol, you also need to specify 'extraHeaders' for the response modifications.

Starting from Chrome 79, the webRequest API does not intercept CORS preflight requests and responses by default. A CORS preflight for a request URL is visible to an extension if there is a listener with 'extraHeaders' specified in opt_extraInfoSpec for the request URL. onBeforeRequest can also take 'extraHeaders' from Chrome 79.

Starting from Chrome 79, the following request header is not provided and cannot be modified or removed without specifying 'extraHeaders' in opt_extraInfoSpec:

• Origin

Starting from Chrome 72, if you need to modify responses before Cross Origin Read Blocking (CORB) can block the response, you need to specify 'extraHeaders' in opt_extraInfpSpec.

Starting from Chrome 72, the following request headers are not provided and cannot be modified or removed without specifying 'extraHeaders' in opt_extraInfoSpec:

• Accept-Language
• Accept-Encoding
• Referer
• Cookie

Starting from Chrome 72, the Set-Cookie response header is not provided and cannot be modified or removed without specifying 'extraHeaders' in opt_extraInfoSpec.

Source: https://developer.chrome.com/extensions/webRequest

[Corion]

Thank you for the link, but I don't see how it applies to the situation. The Referer header is not settable/addable since at least Chrome version 63, so the linked documentation does not seem to apply to the situation.

[Corion]

See https://bugs.chromium.org/p/chromium/issues/detail?id=849972 for the corresponding Chromium bug