Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FileFeedStorage creates empty file when no items are scraped #872

Closed
gbirke opened this issue Sep 2, 2014 · 2 comments
Closed

FileFeedStorage creates empty file when no items are scraped #872

gbirke opened this issue Sep 2, 2014 · 2 comments

Comments

@gbirke
Copy link

@gbirke gbirke commented Sep 2, 2014

When no items are scraped, the corresponding file is created none the less, because it is created in by the storage.open call in FeedExporter.open_spider. This behavior ignores the the setting of FEED_STORE_EMPTY when using file export.

My proposal for this would be to add a cleanup method to the IFeedStorage interface. Then FeedExporter.close_spider can call that method before returning in case slot.itemcount is zero and self.store_empty is False. cleanup could also be called internally from the store methods of the IFeedStorage implementations.

@michael-yin
Copy link

@michael-yin michael-yin commented Sep 20, 2014

@gbirke I have met the same problem, there are so many empty files when you save data to files, and your proposal sounds like a good idea to me. IFeedStorage need to be added a new method called in close_spider of FeedExporter

@dfockler
Copy link

@dfockler dfockler commented Dec 23, 2014

I'm having a similar issue, where the JSONItemExporter will write out the opening [ when calling it's start_exporting method but if there are no items scraped, it's finish_exporting method never gets called, and the closing ] is never added, so any resources consuming that JSON file will throw errors because of the invalid JSON format.

gbirke added a commit to gbirke/scrapy that referenced this issue Jan 10, 2015
FileFeedStorage left empty files when no items were scraped. This patch
adds a cleanup method to the IFeedStorage interface that will be called
by FeedExporter when no items were scraped.

Fixes scrapy#872
redapple added a commit to redapple/scrapy that referenced this issue Sep 16, 2016
@kmike kmike closed this in #2258 Sep 19, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants
You can’t perform that action at this time.