HTTP request header support #48

wants to merge 1 commit into


None yet
6 participants


I needed to add some additional HTTP request headers and didn't see any support for that currently. Happy to change anything / add more specs to cover the change.


jgnagy commented Apr 25, 2012

I agree that this should be in there... please accept this so everyone can benefit! :-)

didlix commented Oct 4, 2013



brutuscat commented Dec 14, 2014

@ashmckenzie would you please submit a new PR in the new fork called Medusa?
Instead of using Net::HTTP uses OpenURI, which means that the headers should be passed in the options argument as seen in the http.rb. Just make sure that the keys are Strings. See OpenURI option docs.

options must be a hash.
Each option with a string key specifies an extra header field for HTTP. I.e., it is ignored for FTP without HTTP proxy.
The hash may include other options, where keys are symbols

Thank you!
I was allowed used in crawling require Basic authentication page .


require 'anemone'
require 'base64'

url = ""
auth_base64 = Base64.encode64('USER:PASSWORD').gsub(/\n/, "")
headers = {"Authorization" => "Basic #{auth_base64}"}

Anemone.crawl(url, {:http_request_headers => headers}) do |anemone|
  anemone.on_every_page do |page|
    puts page.url
    puts page.body.toutf8

@chriskite Can we merge this?

The need is not . It is just information .


brutuscat commented Sep 18, 2015

@atgs-ghayakawa @paresharma could u help adapt this PR to a new PR in the new fork called Medusa?

Hi, I forked your fork so that I could do a PR with these changes. But, I see that you have already added BAA support:

I can just use Medusa in place of Anemone with BAA. 👍

Medusa.crawl(url, { http_basic_authentication: [username, password] }) do |medusa|
  medusa.on_every_page do |page|
    puts page.code

Now, that I am at it I'll do a general clean up and make it compatible with Ruby 2.2 (which would mean dropping support for Kyoto and Tokyo, I guess). Will do a PR when it's done. 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment