Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WARC files not properly closed if topology killed or died #10

Closed
sebastian-nagel opened this issue Oct 5, 2016 · 4 comments
Closed

WARC files not properly closed if topology killed or died #10

sebastian-nagel opened this issue Oct 5, 2016 · 4 comments

Comments

@sebastian-nagel
Copy link
Collaborator

If the crawl topology died or was killed the WARC file is not properly closed. This causes an error when decompressing the WARC file: gzip: CC-NEWS-20160926233041-00001.warc.gz: unexpected end of file.
@sebastian-nagel
Copy link
Collaborator Author

Looks like the cleanup() method in WARCHdfsBolt/GzipHdfsBolt is not called when a topology is killed. The last record in the WARC file is truncated, compared with the worker.log more than 200 records are missing from the WARC file.

@sebastian-nagel
Copy link
Collaborator Author

... because of "there is no guarentee that cleanup will be called" (BaseRichBolt.cleanup())?

@jnioche
Copy link
Contributor

jnioche commented Oct 17, 2016

Exactly. There could be a way of improving things though by using the sync policy (i.e. when we mark/fail the tuples) better? A lower value would mean that we flush more often.

@sebastian-nagel
Copy link
Collaborator Author

This is fixed by apache/incubator-stormcrawler#596 which is now included after the upgrade to StormCrawler 1.12.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jnioche @sebastian-nagel