Skip to content
Pre-release
Pre-release

@Turnerj Turnerj released this Feb 20, 2021

Changes

  • Increased the amount and the detail of debug logging for crawling

🐛 Bug Fixes

🧰 Maintenance

  • Bump HtmlAgilityPack from 1.8.11 to 1.11.30
  • Bump Microsoft.Extensions.Logging.Abstractions from 2.2.0 to 3.1.12
  • Bump TurnerSoftware.RobotsExclusionTools from 0.3.0 to 0.7.0
  • Bump TurnerSoftware.SitemapTools from 0.3.0 to 0.6.0
Assets 4
Pre-release
Pre-release

@Turnerj Turnerj released this Aug 17, 2019

Improved support for in-page Robots configuration (#7)
Added fragment support for crawl data (#7)
Added support for request timeouts (#7)
Added support for max pages to crawl (#7)
Obey no-follow on anchors (#7)
Various other minor tweaks and fixes (#7)

Assets 4
Pre-release
Pre-release

@Turnerj Turnerj released this Mar 15, 2019

Replaced crawl contents stream with a string, allowing easier access to contents.

Assets 2
Pre-release
Pre-release

@Turnerj Turnerj released this Jan 1, 2019

Initial release with core functionality

Assets 2