Request headers #155

Closed
soulgalore opened this Issue Jun 17, 2016 · 9 comments

Projects

None yet

3 participants

@soulgalore
Member

In older versions you could add your own request headers, maybe we should support that?

@soulgalore soulgalore added the 1.0 label Jun 17, 2016
@soulgalore soulgalore added this to the 1.0 milestone Jun 17, 2016
@soulgalore soulgalore referenced this issue in sitespeedio/sitespeed.io Sep 21, 2016
Closed

--requestHeaders no longer present in v4 #1214

@eripa
eripa commented Dec 6, 2016

Casting a vote for this. We currently protect our Dev and QA sites using a custom cookie which would work if I can pass headers. I'm open to a workaround if there is one. 😄 Using curl I do this -H "Cookie: mycookie=somevalueyepyep"

@tobli
Member
tobli commented Dec 6, 2016

Unfortunately I can't think of a way, short of writing browser plugins (complex) or adding a proxy (messes with http2 and more) to add custom headers to the requests. However for cookies there's a workaround that I think/hope should work. Normally the browser is restarted (cookies and caches cleared) between each run. Pre-scripts however are run in the same browser session as the main url. So if you use a pre-script to load a page (preferably a page on the same domain that doesn't return a 4xx error) and use the selenium api to set the cookie ( driver.options().addCookie({name: 'foo', value: 'bar'}); ) it should work. I haven't confirmed how the browsers handle this, but potentially you could access a page at any domain and set the domain for the new cookie explicitly by adding the 'domain' key to the cookie explicitly. See https://github.com/SeleniumHQ/selenium/blob/master/javascript/node/selenium-webdriver/lib/webdriver.js#L1223 for more details. We'd love to hear if this works for you.

@eripa
eripa commented Dec 7, 2016

thanks for the quick reply @tobli!

I'm not familiar with Selenium. I tried these two approaches as preScript, without success. Any pointers?

Error is TypeError: driver.options is not a function

module.exports = {
    run(context) {
        return context.runWithDriver((driver) => {
        return driver.get('https://example.com/unprotected.php')
            .then(() => {
            driver.options().addCookie({name: 'foo', value: 'bar'});
            });
        })
    }
    };

and

module.exports = {
  run(context) {
    return context.runWithDriver((driver) => {
      return driver.get('https://example.com/unprotected.php')
        .then(() => driver.options().addCookie({name: 'foo', value: 'bar'}));
    })
  }
};```

@soulgalore
Member

@eripa could be that the Selenium docs is outdated? I think this will work:

module.exports = {
   run(context) {
       return context.runWithDriver((driver) => {
               return driver.get('https://example.com/unprotected.php')
               .then(() => {
                       driver.manage().addCookie({name: 'foo', value: 'bar'});
                   });
           })
   }
};
@eripa
eripa commented Dec 8, 2016

thanks @soulgalore! This approach almost works, using this I can scan the landing page. It does however seem like the --crawler.depth doesn't have any effect when I'm using the preScript. It simply stops crawling after the first level.

@tobli
Member
tobli commented Dec 8, 2016

You're right, the crawler is separate from the browser and doesn't use the prescript. Right now there isn't a way to combine the two. I'd recommend doing a separate crawl first, and then feeding those urls to sitespeed.io.

While the crawler library has a feature to add custom headers, it's not something we use today. Another option we've investigated is to crawl via the browser, using javascript to extract links from the page. Both has their pros and cons, and adds complexity so we haven't added it yet. Let us know if the pre-crawl (or fixed URL list) is a viable option. If not I propose we discuss your use case/constraints in a separate feature request issue.

@eripa
eripa commented Dec 8, 2016

Given my usecase, and the fact that this is for dev/qa sites I think feeding a fixed list of URLs is good enough for my use. I just wanted to let you know, but you already did! 😄

Thanks again for your quick response times! I don't think it's necessary for me to have Request Headers given this solution.

@tobli
Member
tobli commented Dec 8, 2016

Great! Happy it's working for you, and thanks for the feedback.

@tobli
Member
tobli commented Dec 13, 2016

Given the reasons stated above, I'd say no to request header support for now. We could reconsider if new opportunities arise in the future.

@tobli tobli closed this Dec 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment