-
Notifications
You must be signed in to change notification settings - Fork 756
Release Notes 1.14.3
These are the project wiki Release Notes for the 1.14.3 release.
Release 1.14.3 is a 'micro' release with a number of small bugfixes and new requested features.
The 1.14.3 release is now available at the archive-crawler Sourceforge project.
Option to read persisted URI history at crawl start (HER-1586)
The PersistLoadProcessor has a new setting, 'preload-source', which may be a local path to a persisted URI history log (as written by PersistLogProcessor on a previous crawl), a URI to a similar log, or a local path to a prior crawl state directory with URI history info (as stored by PersistStoreProcessor on a previous crawl). At crawl start, this history info will be preloaded for consultation by the duplication-reduction features, if activated. See Feature Notes - 1.12.0 for more details about duplication-reduction options.
BDB-JE environments (crawl 'state' directories) now use 'shared cache' by default (HER-1594)
Previously, each crawl's BerkeleyDB-JE environment used its own in-memory cache. Unless changed to something smaller than the default 60% of all Java heap memory, this would cause problems for running simultaneous crawls in the same Java VM. (And if changed to a smaller number, cache memory might not be used efficiently between crawls.) A recent feature of BDB-JE allows all open environments to share the same cache allotment. We now activate this option by default.
If you launch multiple simultaneous crawls in the same Java VM – which is only common where Heritrix is embedded in a larger crawl scheduling system – you should now allow each crawl to specify a 'bdb-cache-percentage' equal to the total you'd like all crawls to share. (When opening multiple environments with a shared-cache, the cache adopts the size specified by the most-recently-opened environment.)
In addition to the usual suspects, this release includes contributed fixes or functionality from:
- Siznax
- Christian Dehning
The following 14 tracked issues are recorded as addressed in this 1.14.3 release:
http://webteam.archive.org/jira/secure/ReleaseNote.jspa?projectId=10021&styleName=Html&version=10090
type
key
summary
status
Unable to locate JIRA server for this macro. It may be due to Application Link configuration.
Structured Guides:
User Guide
- Introduction
- New Features in 3.0 and 3.1
- Your First Crawl
- Checkpointing
- Main Console Page
- Profiles
- Heritrix Output
- Common Heritrix Use Cases
- Jobs
- Configuring Jobs and Profiles
- Processing Chains
- Credentials
- Creating Jobs and Profiles
- Outside the User Interface
- A Quick Guide to Creating a Profile
- Job Page
- Frontier
- Spring Framework
- Multiple Machine Crawling
- Heritrix3 on Mac OS X
- Heritrix3 on Windows
- Responsible Crawling
- Adding URIs mid-crawl
- Politeness parameters
- BeanShell Script For Downloading Video
- crawl manifest
- JVM Options
- Frontier queue budgets
- BeanShell User Notes
- Facebook and Twitter Scroll-down
- Deduping (Duplication Reduction)
- Force speculative embed URIs into single queue.
- Heritrix3 Useful Scripts
- How-To Feed URLs in bulk to a crawler
- MatchesListRegexDecideRule vs NotMatchesListRegexDecideRule
- WARC (Web ARChive)
- When taking a snapshot Heritrix renames crawl.log
- YouTube
- H3 Dev Notes for Crawl Operators
- Development Notes
- Spring Crawl Configuration
- Build Box
- Potential Cleanup-Refactorings
- Future Directions Brainstorming
- Documentation Wishlist
- Web Spam Detection for Heritrix
- Style Guide
- HOWTO Ship a Heritrix Release
- Heritrix in Eclipse