#

Crawler

A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).

Here are 200 public repositories matching this topic...

dotnetcore / DotnetSpider

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

crawler csharp cross-platform dotnetcore distributed

Updated Jul 9, 2024
C#

hardkoded / puppeteer-sharp

Headless Chrome .NET API

crawler chrome automation csharp crawling chromium e2e webautomation e2e-testing puppeteer

Updated Sep 6, 2024
C#

sjdirect / abot

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Updated Jul 17, 2024
C#

johanneszab / TumblThree

A Tumblr Blog Backup Application

windows crawler downloader backup internationalization csharp tumblr wpf mvvm mef tumblr-blog tumblr-likes hidden-tumblr password-protected-tumblr tumblr-search tumblr-tags safe-mode-tumblr

Updated Mar 7, 2021
C#

ohhsodead / FileMasta

A search application to explore, discover and share online files

Updated May 11, 2022
C#

TumblThreeApp / TumblThree

A Tumblr and Twitter Blog Backup Application

Updated Sep 7, 2024
C#

stanzhai / Html2Article

Html网页正文提取

html content crawler spider article topic

Updated May 9, 2022
C#

microfisher / Strong-Web-Crawler

基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。

crawler phantomjs web-crawler sellenium

Updated Oct 25, 2019
C#

zhupingqi / RuiJi.Net

crawler framework, distributed crawler extractor

crawler scraper extractor netcore scrapy owin headless-chrome

Updated Jun 20, 2024
C#

InfinityCrawler

TurnerSoftware / InfinityCrawler

A simple but powerful web crawler library for .NET

crawler spider web-crawler robots-txt web-crawling

Updated Dec 15, 2023
C#

zhaotianff / CSharpCrawler

C#爬虫示例程序，想学习爬虫入门知识的可以看过来。后续会慢慢加入更多爬虫相关的知识。

crawler csharp wpf

Updated Dec 8, 2022
C#

mehmetozkaya / DotnetCrawler

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-w…

crawler csharp dotnetcore scraping crawling webscraper scrapy entity-framework-core webcrawler webscraping scrapy-crawler ddd-architecture htmlagilitypack webcrawling webcrawler-htmlagilitypack

Updated Dec 20, 2022
C#

stulzq / HttpCode.Core

简单、易用、高效一个有态度的开源.Net Http请求框架!可以用制作爬虫，api请求等等。

crawler httprequest net-standard net-core httpmock httpcode

Updated Dec 13, 2018
C#

Liu233w / acm-statistics

An online tool (crawler) to analyze users performance in online judges (coding competition websites). Supported OJ: POJ, HDU, HYSBZ, CodeForces, UVA, ICPC Live Archive, FZU, SPOJ, Timus (URAL), LeetCode_CN, CSU, LibreOJ, 洛谷, 牛客OJ, Lutece (UESTC), AtCoder, AIZU, CodeChef, El Judge, BNUOJ, Codewars, UOJ, NBUT, 51Nod, DMOJ, VJudge

nodejs javascript docker crawler vue csharp acm-icpc codeforces-api codechef-api spoj-api

Updated Aug 26, 2024
C#

pavlovtech / WebReaper

Web scraper, crawler and parser in C#. Designed as simple, declarative and scalable web scraping solution.

parser crawler scraper parsing scraping webcrawler webscraping scraping-websites datamining scraping-api scraping-tool scraping-web scraping-data

Updated Nov 21, 2023
C#

johanneszab / TumblTwo

TumblTwo, an Improved Fork of TumblOne, a Tumblr Downloader.

photos crawler downloader tumblr ripper videos tumblr-blog tumblr-downloader

Updated Sep 9, 2017
C#

Misterhex / WebCrawler

Just a simple web crawler which return crawled links as IObservable using reactive extension and async await.

c-sharp crawler dotnet web-crawler reactive-extension

Updated Jul 2, 2019
C#

rollrat / custom-crawler

🌌 High productivity semi-automatic crawler generator 🛠️🧰

crawler crawling crawling-framework wpf-application custom-crawler

Updated Oct 5, 2023
C#

thomas11 / AzureSearchCrawler

A simple web crawler, using Abot, that indexes page contents into Azure Search.

crawler azure azure-search abot

Updated Sep 2, 2024
C#

kant2002 / ncrawler

Web Crawler written in C#

crawler scrapper

Updated Oct 13, 2019
C#

Followers: 404 followers
Wikipedia: Wikipedia