Support for custom user agent

[hdclark]

What

A few simple lines were added to provide a "--user-agent" option to specify a custom user agent.

How it changes the code

A new variable (opt.useragent) is introduced. No functionality breaks (I think!) As soon as possible, we update std_header['User-Agent'] with the custom user agent string.

How it should be used

youtube-dl --user-agent "My Custom User Agent"  "http://www.website.com/"

How it changes the HTML

Without a custom user agent, we can expect

prompt$  youtube-dl 'http://127.0.0.1:8080/somesite'

to yield the GET header

Host: 127.0.0.1:8080
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:5.0.1) Gecko/20100101 Firefox/5.0.1
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: close

When we specify a custom agent

prompt$  youtube-dl --user-agent 'My Custom User Agent' 'http://127.0.0.1:8080/somesite'

we get the modified request

Host: 127.0.0.1:8080
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: My Custom User Agent
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: close

Notes

My Python experience is limited. My tests all worked, but please ensure I've not done something silly.

[hdclark]

Note that this will help alleviate issues arising from 'dated' user agents, like issue #390.

Also note that using youtube-dl without a custom user agent can be a PITA when, for example, performing

mplayer -user-agent 'FOO' "`youtube-dl --get-url 'http://...'`"

If the website is picky about ensuring the resolver and player have the same user agent. YouTube and Vimeo will occasionally complain about this.

[rbrito]

Oh, just to complement the previous message where I talked about the rebase, you will quite probably have to do a git push --force to send your changes to your repo after you change history, but it is worth it to make the tree cleaner.

Another solution (probably cleaner) would be to create a branch from HEAD of the master repo (say, fix-useragent) put your changes there and ask github to send a pull request from that branch of yours to the master branch of rg3/youtube-dl.

[hdclark]

Hey rbrito,

As far I can tell I've re-based my repo to remove the snafu. Github tells me that the original request has been changed to reflect the change to my master branch. Is this suitable?

[rbrito]

Hi, @hdclark.

On Sep 25 2012, hdclark wrote:

As far I can tell I've re-based my repo to remove the snafu. Github tells
me that the request has been changed to reflect the change to my master
branch. Is this suitable?

This looks much cleaner. I think that @ FiloSottile may want to consider
this version instead of any previous version in his tree.

Also, while this is a more general solution than mine in issue #390, it
would be nice to have that applied as we get a newer User-Agent by
default.

Thanks,

Rogério Brito : rbrito@{ime.usp.br,gmail.com} : GPG key 4096R/BCFCAAAA
http://rb.doesntexist.org/blog : Projects : https://github.com/rbrito/
DebianQA: http://qa.debian.org/developer.php?login=rbrito%40ime.usp.br

[FiloSottile]

LGTM, it should work and not conflict with other user_agent stuff, like blip.tv fix.

[phihag]

Merged. Thanks!

[Liontooth]

This isn't working for me --

$ youtube-dl --user-agent "Mozilla/5.0" -e $URL
Usage: youtube-dl [options] url [url...]

youtube-dl: error: no such option: --user-agent

$ youtube-dl --version
2012.09.27

This is the macports version.

[hdclark]

Same for me (Arch Linux, 9/27.)

I think it was packaged prior to merge. If you look at the options, no user agent flag exists.

[Liontooth]

I compiled the latest tarball and still get the same result.

[hdclark]

I looked at the source and the user agent code is included in init.py. I'm not sure why it isn't working in the 9/27 binary and haven't a clue about the tarball. You could try cloning the repo - I would think that should work.

I'm quite far away from a reasonable machine for a few days so I will have to do more digging after the weekend (holiday in Canada.)

[hdclark]

Upon expection, performing
git clone https://github.com/rg3/youtube-dl.git

followed by
make clean && make

followed by
./youtube-dl --user-agent 'Some Alternative Agent' --get-url 'http://...'

works fine. It appears the packaged binary/tarball simply did not include this pull.

Maybe @phihag could comment?

[phihag]

Should be fixed with the new 2012.10.09 release

wget https://github.com/rg3/youtube-dl/raw/2012.10.09/youtube-dl -O youtube-dl
./youtube-dl --user-agent 'foo'

works fine (i.e. doesn't say no such option: --user-agent) on my system.