Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upGitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
Inconsistent user-agent #3528
Inconsistent user-agent #3528
Comments
|
This apparently not just affects the user-agent but also the proxy settings, which might leak information you do not want to share! |
|
Hi binfalse, I think I found out where the HEAD request is not getting the user-agent added. generic.py is the extractor that gets loaded for non-explicitly-supported websites and it did not have an addheader() method in there to utilize either the default header or one specified. By affecting the proxy settings, do you mean that the user-agent is inconsistent when the client sends http/https requests to the proxy? Best regards, |
|
Thank you for the report. I believe this problem to be fixed in youtube-dl 2014.08.24.5 and newer. Type |
When calling youtube-dl with a generic (not-explicitly-supported) website it first sends a
HEADrequest to get the http response code:https://github.com/rg3/youtube-dl/blob/c990bb36338c422d4da9f092384da1dfbb528d87/youtube_dl/extractor/generic.py#L351
Unfortunately, that call does not set the user-agent. Thus, the web server will see a
HEADrequest from aPython-urllib/*.*client immediately followed by aGETrequest from the same client supplying a different (the desired) user-agent.I am not sure if that also affects other extractors...