New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SourceSeedDecideRule, SeedLimitsEnforcer #137

Merged
merged 8 commits into from Jan 13, 2016

Conversation

Projects
None yet
2 participants
@nlevitt
Member

nlevitt commented Dec 4, 2015

No description provided.

nlevitt added some commits Dec 4, 2015

add support to StatisticsTracker to keep a CrawledBytesHistotable per…
… source tag when trackSources is enabled; integration test for this functionality
SeedLimitsEnforcer (contrib) DecideRule that rejects CrawlURI if sour…
…ce seed byte or document limit has been reached
change class originally known as SeedLimitsEnforcer to
SourceQuotaEnforcer; make it a Processor instead of a DecideRule
(because checking quota at link scoping time doesn't work, since many
urls which would go over quota can be added to the frontier); support
quotas on any of the fields tracked by CrawledBytesHistotable

adam-miller added a commit that referenced this pull request Jan 13, 2016

Merge pull request #137 from nlevitt/seed-limits
SourceSeedDecideRule, SeedLimitsEnforcer

@adam-miller adam-miller merged commit 3dbf754 into internetarchive:master Jan 13, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment