-
Notifications
You must be signed in to change notification settings - Fork 10.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Might be nice to always call exporter.finish_exporting
#5537
Comments
Without looking at it for enough time, it looks like we should call |
Can I solve this issue. Looks like just need to add the 'slot.finish_exporting()' before the if condition in the 'def _close_slot(self, slot, spider)' |
@HatimZ probably, but you need to make sure this code isn't called if |
@wRAR .Okay. Are there any examples/resources for writing test cases? |
If you mean a test case for making sure it's ohly called when start_exporting() was called it would just be a test spider with some exporter enabled and some different edge cases like zero items, maybe an exception happening at some early point etc. |
I checked the code to make sure that
|
Also if test will be required can you point me to an already written test case, which I can use as an example. I am a beginner hence need some help. Thanks. |
@wRAR . If I can solve this and you can merge it until Monday, it will be counted towards hacktoberfest. |
Yes, this sounds correct.
I don't have one handy, but it looks like tests/test_exporters.py is a place with a lot of tests of the related code. |
Description
When an exception happens inside Exporter (for whatever reason) - itemcount never gets increased (that's OK). But then during closing, the code doesn't get chance to act
This is the place in the code:
https://github.com/scrapy/scrapy/blob/master/scrapy/extensions/feedexport.py#L357
Perhaps the lines 357 and 353 should have been swapped?
Additional context
I'm using
slot.storage
to open a database; and an exporter to start a session -- store() gets called (to close connection to the database) but session didn't have chance to commit -- yes, there is nothing to commit, but still - given the semantics (also visible from the code -- whenfile
is supposed to be a file descriptor it might be worth consideration to always callfinish_exporting
to give the file descriptor chance to react before it is closed by the parent object)necessary caveat: I might be mis-using the API and it is a minor detail, so please forgive if this looks silly (and feature not a bug)
The text was updated successfully, but these errors were encountered: