Skip to content

Duplicated feed logs when having multiple feeds #4619

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
StasDeep opened this issue Jun 9, 2020 · 1 comment · Fixed by #4629
Closed

Duplicated feed logs when having multiple feeds #4619

StasDeep opened this issue Jun 9, 2020 · 1 comment · Fixed by #4629
Labels

Comments

@StasDeep
Copy link
Contributor

StasDeep commented Jun 9, 2020

Description

When I have multiple feeds and a blocking feed storage, I get duplicated feed logs.

Steps to Reproduce

  1. Create a test.py file with this spider and settings
  2. scrapy runspider test.py -L INFO

Expected behavior:

2020-06-09 19:23:03 [scrapy.extensions.feedexport] INFO: Stored json feed (10 items) in: gs://bucket/output.json
2020-06-09 19:23:03 [scrapy.extensions.feedexport] INFO: Stored csv feed (10 items) in: gs://bucket/output.csv

Actual behavior:

2020-06-09 19:23:03 [scrapy.extensions.feedexport] INFO: Stored csv feed (10 items) in: gs://bucket/output.csv
2020-06-09 19:23:03 [scrapy.extensions.feedexport] INFO: Stored csv feed (10 items) in: gs://bucket/output.csv

Versions

Scrapy       : 2.1.0
lxml         : 4.5.1.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 20.3.0
Python       : 3.7.4 (default, Sep  4 2019, 15:20:53) - [Clang 10.0.0 (clang-1000.10.44.4)]
pyOpenSSL    : 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020)
cryptography : 2.9.2
Platform     : Darwin-19.4.0-x86_64-i386-64bit

Additional context

We use custom GoogleCloudFeedStorage and it takes some time to store the data. While it's doing uploading, next iteration of a for loop inside FeedExporter.close_spider comes and creates new log_args object, although the last one is used in closure.

@StasDeep
Copy link
Contributor Author

StasDeep commented Jun 9, 2020

I think I can create a PR fixing that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants