-
Notifications
You must be signed in to change notification settings - Fork 9.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Auto-set a Chinese extractor proxy. #1063
base: develop
Are you sure you want to change the base?
Conversation
I used BeautifulSoup to parse a proxy listing website http://www.proxynova.com/proxy-server-list/country-cn/, and then pick from them. Tested on my local machine and it worked.
Hello @xuehuichao, 祝您健康! |
Hi there, Thanks for your PR. It is always a pain to bypass the geographic restrictions outside China. That's a very good observation. The only point I am not 100% sure is whether it is stable enough to build in a 3rd party service to achieve some core functions. @soimort What's your opinion? Yours, |
The code itself is ok, but I'm skeptical about the reliability of using 3rd party services (especially, when they are random selected). What if the auto-selected proxy is malicious, or just behaves strangely without forwarding data correctly? We have no guarantee about this, and I don't want to mislead users to think that this is a |
Thanks for your comments! I guess we can split our discussions into two layers: (1) fundamentally, do we want this feature in you-get? (2) technically, how can we improve the current implementation, if we decide to have this feature.
I guess the technical issues can all be resolved, but the most fundamental question is whether you folks would want this feature included in you-get. I personally think this feature would be worthwhile, as a lot of users want to download from Chinese video websites, and it's a lot of pain finding proxies manually. With that said, I would totally understand if you think the proxy picker should be implemented in a separate tool. Regarding the technical details:
Please let me know what you think. |
Sorry for the late response.
True. I don't know anything about ProxyNova, how it operates and provides these free proxy services, but I can imagine that they are not as reliable as you may think. It claims to enable anonymity, but you could achieve the same thing via Tor as well. Instead of that we pick an arbitrary proxy for users, they must be aware of the transparency and the risk (there's very little risk if you're only going to download videos though) of using it -- and if they really do, they would just pick their own. Thus I'm reluctant to let As for now, "picking a freely available proxy" was never my intent; I want to use just my own proxy, so it's not really a priority for me.
How? IMHO we should not handle that level of complexity, for this simple command-line tool that uses regexp to scrape the web. We are not going to make it a fault-tolerant, server middleware anyway. As a follow-up on the technical side: BeautifulSoup is cool, but we are not going to introduce any external dependency to this simple tool at this stage. |
It could be risky to pass the login credentials via a random service provided by god-knows-who. I cannot see any good scenario that a well-awarded user would need this function; also I am concerning about the mortality burden of providing risky service to end-users. BTW BS4 is good: but from my point of view I cannot see the necessarily of adding this as our dependency. |
I do think this is very important funciton. It maybe good to provide the option to the users and let the users to decide whether to take the risk. Also, for some services, credentials are not nessesary. It dose not matter whether the proxeis are malicious. I couldn't care less whether they are trying to peak the video I am watching. |
This pull request enables "-C" and "--china", to auto-set a proxy for Chinese video websites.
A common usecase of you-get is to download website content from Chinese websites. One inconvenience is: we always need a proxy, most likely by looking up on the web. This patch automated this process. It parses a proxy listing website, and randomly choose a proxy from that list.
This is my first pull request to this repo. Please don't hesitate to share your comments on this pull request. Thanks!
This change is![Reviewable](https://camo.githubusercontent.com/23b05f5fb48215c989e92cc44cf6512512d083132bd3daf689867c8d9d386888/68747470733a2f2f72657669657761626c652e696f2f7265766965775f627574746f6e2e737667)