Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spider 中怎么使用http代理 #96

Closed
landonzeng opened this issue Dec 13, 2018 · 5 comments
Closed

spider 中怎么使用http代理 #96

landonzeng opened this issue Dec 13, 2018 · 5 comments
Labels

Comments

@landonzeng
Copy link

看了源码,还是不太清楚如何使用
在自己代码里这样写的,但是跑不起来,其中123.115.120.182:9000是代理ip地址和端口
`

public static void Run()
{
WebProxy webProxy = new WebProxy();
webProxy.Address = new Uri("http://123.115.120.182:9000");
ProxyInfo proxyInfo = new ProxyInfo(webProxy);
ProxySupplier p = new ProxySupplier(proxyInfo);
HttpProxyPool.Instance = new HttpProxyPool(p);
List buildInfoList = new List();
List untiInfoList = new List();
int sleep = Convert.ToInt32(JsonConfigurationHelper.GetAppSettings("ThreadNum", "SleepTime"));
using (var conn = new SqlConnection(Env.DataConnectionString))
{
buildInfoList = conn.Query("SELECT Building_Id FROM T_UnitInfo WITH(NOLOCK) WHERE ISNULL(Building_Id,'')!='' GROUP BY Building_Id").ToList();
untiInfoList = conn.Query("SELECT Unit_Id FROM T_UnitInfo WITH(NOLOCK) WHERE ISNULL(Unit_Id,'')!='' GROUP BY Unit_Id").ToList();
}
BuildInfoSpider spider = new BuildInfoSpider(buildInfoList, untiInfoList, sleep);
var threadNum = JsonConfigurationHelper.GetAppSettings("ThreadNum", "Number");
spider.ThreadNum = Convert.ToInt32(threadNum);
spider.CycleRetryTimes = 5;
spider.Run();
}

`

@gonemad4u
Copy link

我也是,普通的request我就是这么加的proxy,不知道用dotnetspider怎么加

HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("https://www.baidu.com");
WebProxy myproxy = new WebProxy("ssvpxy20", 8080);
myproxy.BypassProxyOnLocal = false;
Request.Proxy = myproxy;
Request.Credentials = CredentialCache.DefaultCredentials;
Request.Proxy.Credentials = CredentialCache.DefaultCredentials;

@sanjusss
Copy link
Contributor

sanjusss commented Apr 4, 2019

仅针对我发帖时的3.0.8版(3.x版本略有不同)
要使用代理,需要将下载器HttpClientDownloader的UseProxy属性设置成true、HttpProxyPool属性设置为代理池的实例。
由于设置的固定代理,你需要重新实现IHttpProxyPool接口,使其GetProxy无限制可以返回固定值。

@landonzeng
Copy link
Author

@sanjusss 能提供个demo吗?

@Interopnl
Copy link

Could someone please provide some code examples on howto implement a custom IProxySupplier?

Thanks in advanced!

@zlzforever
Copy link
Collaborator

Sorry for the delay, if you want to use proxy, i suggest you to checout refactory-5.0 branch. There are lot s of update for use proxy.

A lots of proxy provider support WEB API to get proxies, and it return data like:
x.x.x.x:8080
y.y.y.y:8181

I impl a KuaidailiProxySupplier in the refactory-5.0 branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants