Skip to content
Web crawler for dotnet core
C# PowerShell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src
.gitattributes
.gitignore
Build.ps1
README.md
Recluse.sln
appveyor.yml
global.json

README.md

Recluse

Build status MyGet CI NuGet

Recluse is a simple dotnet core web crawler.

Usage from Console app:

IServiceCollection services = new ServiceCollection();

services.AddSingleton<ICrawlHandler, LogCrawlHandler>();
services.AddRecluseCrawler();
            
var serviceProvider = services.BuildServiceProvider();
var crawler = serviceProvider.GetService<RecluseCrawler>();

var task = crawler.CrawlAsync(new CrawlTask(new Uri("http://news.ycombinator.com")));

crawler.Start();
task.Wait();
var obj = task.Result;
Console.WriteLine($"{obj.Uri} -  {obj.StatusCode} - {obj.Headers}");
foreach (var item in obj.Links)
{
    Console.WriteLine($"{item.Uri} -  {item.LinkText} - {item.LinkType}");
}
You can’t perform that action at this time.