Skip to content

A Very Simple Spider With DHT Crawler, Written by C#.

Notifications You must be signed in to change notification settings

csecong/DHTSpider

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

39 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DHTSpider

A Very Simple Spider With DHT Crawler, Written by C#.

  • The bittorrent library bases on DhtWalker,MonoTorrent.

介绍

DHTSpider 是一个由C#编写的 DHT 爬虫, 从全球DHT 网络里"嗅探"正在下载的资源, 并把资源metadata(种子信息)进行抓取下载.

相关

使用 MongoDB 进行数据存储.

使用 ElasticSearch 进行全文检索(待定).

环境

  1. .net 版本 >=4.5.2

  2. 运行的机器需要独立IP , 能够访问外网 ( 翻墙 )

参数

Setting.InitSetting();//初始化相关参数
    
    LocalPort = 6881;

    IsSaveTorrent = true;

    TorrentSavePath = @"torrent\";

    IsUsingMongoDb = false;

    InitResolverThreadCount = 10;

    MaxResolverThreadCount = 50;

    WaitSeedsCount = 100;


    //DhtSpiderSetting

    DhtSpiderSetting.MaxSendQueue = 50;
    DhtSpiderSetting.MaxFindSendPer = 100;
    DhtSpiderSetting.MaxWaitCount = 1000;
    DhtSpiderSetting.MaxCacheCount = 5000;
    

感谢

项目开发中,bittorrent相关类库参考了 DhtWalkerMonoTorrent, 在此感谢作者 →_→

About

A Very Simple Spider With DHT Crawler, Written by C#.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • C# 100.0%