Add User-Agent for open call

[krisdigital]

Got 403 when trying to identify https://www.ultimate-guitar.com/modules/rss/all_updates.xml.php

Maybe solves #19?

[geoffw8]

I mean, this repo is nothing to do with me - but your fix works so I've approved it. Would be good to have a spec for it ideally but I haven't looked at the rest of the coverage. I can't merge anyway, so its up to @damog

[damog]

I disagree that this is how the user agent should be overridden. Ideally, a set of headers is passed over to override the request, and not force all subsequent ones to Mozilla or whatever. Happy to look into it but happier to receive a patch that would do just that.

[krisdigital]

@damog Don't know if it is overkill to make the accept-header customizable and how many feeds have this problem. But it is a strange behaviour considering

curl https://www.ultimate-guitar.com/modules/rss/all_updates.xml.php -I -H 'User-Agent:'
HTTP/2 200 
date: Sun, 06 Jan 2019 21:33:10 GMT
content-type: text/xml; charset=ISO-8859-1
server: nginx
...

open("https://www.ultimate-guitar.com/modules/rss/all_updates.xml.php")
OpenURI::HTTPError: 403 Forbidden

 open("https://www.ultimate-guitar.com/modules/rss/all_updates.xml.php", "User-Agent" => "feedbag")
=> #<Tempfile:/var/folders/61/d5yqnx1x20v2gn095pmqcpfc0000gn/T/open-uri20190106-20239-18y8z4h>

So curl works without user-agent, open does not.. Does not matter which user-agent is used unless it is not empty and it works..

[damog]

@krisdigital: I tried debugging this for a bit today without much luck. Is there a simple way to see the full request headers from OpenURI?

[krisdigital]

@damog it is mysterious.. Here is what I found out

Open-Uri default headers

Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: */*
User-Agent: Ruby
Host: localhost:8000

curl default headers

Host: localhost:8000
User-Agent: curl/7.54.0
Accept: */*

I used a little python script: https://gist.github.com/phrawzty/62540f146ee5e74ea1ab

So in my case, the server I was trying to speak to seems to have a problem with the user agent Ruby.. 💎

[damog]

@krisdigital -- interesting. I guess the user agent could be signed as Feedbag but I'd hate to break the simplicity of using the default open-uri behavior though. Regardless, if that server in particular doesn't like Ruby, well, I guess it's definitively up to the admin not to allow for automated access.

[krisdigital]

@damog Understandable! I think we can close the ticket then.. Thank's for looking into it!