Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Configurable Robots caches #557

Closed
jnioche opened this issue Apr 4, 2018 · 0 comments
Closed

Configurable Robots caches #557

jnioche opened this issue Apr 4, 2018 · 0 comments
Assignees
@jnioche
Labels
core enhancement
Milestone
1.9

Comments

@jnioche
Copy link
Contributor

jnioche commented Apr 4, 2018

This is currently hard-coded

https://github.com/DigitalPebble/storm-crawler/blob/3fc7d23ad0fc50033ecfc1c035cddd1543c21b6c/core/src/main/java/com/digitalpebble/stormcrawler/protocol/RobotRulesParser.java#L51

We should allow this to be done via config, like we do for the status updater bolt cache with status.updater.cache.spec
@jnioche jnioche added this to the 1.9 milestone Apr 4, 2018
@jnioche jnioche self-assigned this Apr 4, 2018
@jnioche jnioche changed the title Configurable Robots cache Configurable Robots caches Apr 5, 2018
jnioche added a commit that referenced this issue Apr 5, 2018
@jnioche
Configurable Robots caches #557
19b3f50
@jnioche jnioche closed this as completed Apr 5, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jnioche jnioche

Labels
core enhancement
Projects
None yet
Milestone
1.9
Development

No branches or pull requests
1 participant
@jnioche