Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storage.store is called only for the first feed when empty #4621

Closed
StasDeep opened this issue Jun 10, 2020 · 2 comments · Fixed by #4626
Closed

Storage.store is called only for the first feed when empty #4621

StasDeep opened this issue Jun 10, 2020 · 2 comments · Fixed by #4626
Labels

Comments

@StasDeep
Copy link
Contributor

StasDeep commented Jun 10, 2020

Description

When having FEED_STORE_EMPTY=False and multiple feeds, and the output is empty, storage.store is called until first empty feed slot is encountered.

Steps to Reproduce

  1. Create a test.py file with this spider
  2. scrapy runspider test.py -L INFO

Expected behavior:

2020-06-10 10:50:27 [test] INFO: Storing in thread: gs://bucket/output.json
2020-06-10 10:50:27 [test] INFO: Storing in thread: gs://bucket/output.csv

Actual behavior:

2020-06-10 10:50:27 [test] INFO: Storing in thread: gs://bucket/output.json

Reproduces how often: always

Versions

Scrapy       : 2.1.0
lxml         : 4.5.1.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 20.3.0
Python       : 3.7.4 (default, Sep  4 2019, 15:20:53) - [Clang 10.0.0 (clang-1000.10.44.4)]
pyOpenSSL    : 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020)
cryptography : 2.9.2
Platform     : Darwin-19.4.0-x86_64-i386-64bit

Reason

In FeedExporter.close_spider, it returns a deferred from inside a loop instead of adding it to deferred_list

        deferred_list = []
        for slot in self.slots:
            if not slot.itemcount and not slot.store_empty:
                # We need to call slot.storage.store nonetheless to get the file
                # properly closed.
                return defer.maybeDeferred(slot.storage.store, slot.file)
@StasDeep
Copy link
Contributor Author

StasDeep commented Jun 10, 2020

Will provide a fix for this.

@Gallaecio
Copy link
Member

Gallaecio commented Jun 14, 2020

Great catch!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants