-
Notifications
You must be signed in to change notification settings - Fork 756
Continuous Recrawling Phase B Design Notes
Alex Osborne edited this page Jul 4, 2018
·
2 revisions
General idea: expand core capabilities of default Frontier so that it can do revisits simply via passed-in guidance from outside optional components.
Plan: expand options for alreadyIncluded structure
- move outside frontier if possible
- implement Mercator-style merging-lists version
- add option to store history (or at least queue/queueKey) in merging-list; allows efficient random-access to enqueued URIs
Plan: Enhance queueing features:
- add a disk/BDB-backed DelayedQueue-like WorkQueue that can be mixed with classic queues; only releases URIs at configured future times
- resolution of DelayWorkQueue is intentionally crude (~day) to avoid random inserts everywhere
- each classic queue becomes the 'ASAP' queue for the same host/etc; an optional 'future' queue can be added
- implement a mutual-exclusion process for related queues (same domain, same IP) to prevent simultaneous issuance of URIs; acquire named permits, wait for permit to become available (FIFO)
- reconcile budget-rotation with precedence
- have session-budgets cause precedence promotion/demotion rather than 'deactivation'; or
- evaluate eliminating budgets in favor of more sophisticated, large-range precedence-policies
Plan: introduce queue progress heuristics/predictions
- collect progress stats on rates of discovery, completion
- create credible estimates of queue-completion times
Plan: implement naive toy revisit policy
- fixed-revisit interval as test of queue/frontier capabilities
Stage B will likely generate an early-alpha-quality test release of eventual 2.4.
Structured Guides:
User Guide
- Introduction
- New Features in 3.0 and 3.1
- Your First Crawl
- Checkpointing
- Main Console Page
- Profiles
- Heritrix Output
- Common Heritrix Use Cases
- Jobs
- Configuring Jobs and Profiles
- Processing Chains
- Credentials
- Creating Jobs and Profiles
- Outside the User Interface
- A Quick Guide to Creating a Profile
- Job Page
- Frontier
- Spring Framework
- Multiple Machine Crawling
- Heritrix3 on Mac OS X
- Heritrix3 on Windows
- Responsible Crawling
- Adding URIs mid-crawl
- Politeness parameters
- BeanShell Script For Downloading Video
- crawl manifest
- JVM Options
- Frontier queue budgets
- BeanShell User Notes
- Facebook and Twitter Scroll-down
- Deduping (Duplication Reduction)
- Force speculative embed URIs into single queue.
- Heritrix3 Useful Scripts
- How-To Feed URLs in bulk to a crawler
- MatchesListRegexDecideRule vs NotMatchesListRegexDecideRule
- WARC (Web ARChive)
- When taking a snapshot Heritrix renames crawl.log
- YouTube
- H3 Dev Notes for Crawl Operators
- Development Notes
- Spring Crawl Configuration
- Build Box
- Potential Cleanup-Refactorings
- Future Directions Brainstorming
- Documentation Wishlist
- Web Spam Detection for Heritrix
- Style Guide
- HOWTO Ship a Heritrix Release
- Heritrix in Eclipse