fix: merge cookies provided by user with session cookies

[B4nan]

Session cookies are no longer saved on the Request object, but passing via requestAsBrowserOptions instead. The cookie value gets calculated based on the session cookie and Request object - cookies are merged. preNavigationHooks are now fired after this merging happens, so with requestAsBrowserOptions filled with the cookie header.

If two cookies with similar name (where only casing differs), we print a warning, but merge them as two distict cookies.

The request.headers.cookies is still possible to override via pre-nav hooks, its value difference will be merged back to the requestAsBrowserOptions after pre-nav hooks are executed. This allows to work with both request.headers and requestAsBrowserOptions.headers inside the pre-nav hooks.

Closes #1197

[corford]

@B4nan One quick comment (not sure how much it matters): _getRequestOptions() currently will overwrite any changes to request.headers.Cookie made in a preNavHook, with the contents of requestAsBrowserOptions.headers.Cookie.

I guess altering crawlingContext.request.headers in a preNavHook (which is what we currently do) should be considered an anti-pattern? i.e. header manipulation should always be done on requestAsBrowserOptions.headers ?

So far we've been doing it on crawlingContext.request.headers because it's the same access pattern for PuppeteerCrawler preNavHooks

[B4nan]

Thanks for the input, this would indeed be quite a breaking change, I will try to implement support for both. Should be enough to diff the cookie from context.request.headers after hooks are executed, and merge it back to requestAsBrowserOptions.

[szmarczak]

lgtm except that cookie names are case sensitive