Release Notes Heritrix 3.4.0 20210527
Summary of changes since Release Notes - Heritrix 3.4.0-20210527 - see the full changelog for more details.
- Support for extracting URLs in sitemaps #262 (kris-sigur)
- Allow tuning of BDB-JE evictor and cleaner threads. #384 (anjackson)
- Upgrade to Spring 5.3.3 #366 (ato)
- ait youtube-dl options #359 (galgeek)
- Strip quotes from URL value. #352 (BitBaron)
- Fixes leaky file handles #348 (adam-miller)
- youtube-dl --no-playlist #341 (galgeek)
- Fixes extractor multiple regex matcher recycle #335 (adam-miller)
- Warc convention for storing ftp responses has been to use a WARC reso… #334 (adam-miller)
- Remove deprecated sudo setting. #333 (dengliming)
- don't youtubedl receivedFromAMQP #330 (galgeek)
- youtube-dl no cache dir #329 (galgeek)
- best medium-ish size #327 (galgeek)
- Recycle the regex Matcher after use. #317 (adam-miller)
- SFTP results now stored as
resourcerecords #\331*. - Usable variables, e.g. for warcWriter template #363
- Heritrix 3.3 out-of-the-box archives pages with meta noindex #351
- Error Binding hostname or ip to Web UI #339
none
Structured Guides:
User Guide
- Introduction
- New Features in 3.0 and 3.1
- Your First Crawl
- Checkpointing
- Main Console Page
- Profiles
- Heritrix Output
- Common Heritrix Use Cases
- Jobs
- Configuring Jobs and Profiles
- Processing Chains
- Credentials
- Creating Jobs and Profiles
- Outside the User Interface
- A Quick Guide to Creating a Profile
- Job Page
- Frontier
- Spring Framework
- Multiple Machine Crawling
- Heritrix3 on Mac OS X
- Heritrix3 on Windows
- Responsible Crawling
- Adding URIs mid-crawl
- Politeness parameters
- BeanShell Script For Downloading Video
- crawl manifest
- JVM Options
- Frontier queue budgets
- BeanShell User Notes
- Facebook and Twitter Scroll-down
- Deduping (Duplication Reduction)
- Force speculative embed URIs into single queue.
- Heritrix3 Useful Scripts
- How-To Feed URLs in bulk to a crawler
- MatchesListRegexDecideRule vs NotMatchesListRegexDecideRule
- WARC (Web ARChive)
- When taking a snapshot Heritrix renames crawl.log
- YouTube
- H3 Dev Notes for Crawl Operators
- Development Notes
- Spring Crawl Configuration
- Build Box
- Potential Cleanup-Refactorings
- Future Directions Brainstorming
- Documentation Wishlist
- Web Spam Detection for Heritrix
- Style Guide
- HOWTO Ship a Heritrix Release
- Heritrix in Eclipse