Skip to content

Issues: ScaleUnlimited/flink-crawler

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

27 Open 84 Closed
27 Open 84 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Include UrlDBFunction's domain scores in saved state bug in progress actively being worked on
#163 opened Sep 5, 2018 by Schmed
@kkrugler
Sync up the total active urls value with state in UrlDbFunction enhancement
#162 opened Aug 16, 2018 by vmagotra
1
Switch to real source for DomainScore tuples cleanup
#160 opened Jun 30, 2018 by kkrugler
@kkrugler
CrawlTopologyTest.testAsync now fails for me in progress actively being worked on task
#151 opened May 9, 2018 by Schmed
@vmagotra @Schmed
5
Verify metrics using reporter approach
#138 opened Apr 25, 2018 by kkrugler
@Schmed
2
Try using stream harness support for unit testing in progress actively being worked on
#133 opened Apr 11, 2018 by Schmed
@Schmed
7
Make CC fetcher responsive to interrupts cleanup
#123 opened Mar 23, 2018 by kkrugler
Add support for rocksdb CLI option enhancement
#122 opened Mar 23, 2018 by kkrugler
Restore pre-fetching state during restore of UrlDBFunction state bug
#121 opened Mar 23, 2018 by kkrugler
@kkrugler
Support SiteMapIndex in SimpleSiteMapParser enhancement
#120 opened Mar 23, 2018 by kkrugler
Support checkpointing of SeedUrlSource in progress actively being worked on
#108 opened Mar 16, 2018 by kkrugler
@kkrugler
1
Figure out better back-pressure monitoring for in flight URLs enhancement
#104 opened Mar 15, 2018 by kkrugler Future
Fix up jars put into our uber-jar cleanup
#97 opened Mar 14, 2018 by kkrugler
@vmagotra
Add tests for checkpointing
#93 opened Feb 21, 2018 by kkrugler Flink Forward 2018
@kkrugler
2
Auto-retry https URLs that fail with IOException using http protocol enhancement
#83 opened Dec 21, 2017 by kkrugler
Fix Tika configuration cleanup
#79 opened Nov 15, 2017 by kkrugler
@vmagotra
1
CommonCrawlFetcher should limit bytes requested to max content size cleanup
#77 opened Nov 10, 2017 by kkrugler
Fix parsing errors during processing of Common Crawl data cleanup
#73 opened Nov 9, 2017 by kkrugler
@vmagotra
3
Reload robots.txt after some appropriate interval enhancement
#54 opened Oct 15, 2017 by kkrugler
Set refetch time for blocked URLs based on robots.txt reload/retry time enhancement
#53 opened Oct 15, 2017 by kkrugler
Use CommonCrawl robots.txt data if we're in common crawl mode enhancement
#44 opened Sep 20, 2017 by kkrugler
Support multi-threading during tests enhancement
#41 opened Sep 20, 2017 by kkrugler
1
Out of Memory in CommonCrawlFetcherTest.testMultiThreading bug
#38 opened Jun 21, 2017 by Schmed
@kkrugler
Use one FetchFunction and one ParseFunction for all types of URLs cleanup
#34 opened Apr 19, 2017 by kkrugler
1
Calculate a DomainRank as part of the crawling process enhancement
#25 opened Feb 2, 2017 by kkrugler
@vmagotra @Schmed
5
ProTip! Add no:assignee to see everything that’s not assigned.