Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOWNLOAD_FAIL_ON_DATALOSS logged for all requests #4217

Open
Gallaecio opened this issue Dec 5, 2019 · 2 comments
Open

DOWNLOAD_FAIL_ON_DATALOSS logged for all requests #4217

Gallaecio opened this issue Dec 5, 2019 · 2 comments
Labels

Comments

@Gallaecio
Copy link
Member

Gallaecio commented Dec 5, 2019

[scrapy.core.downloader.handlers.http11] Got data loss in . If you want to process broken responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False -- This message won't be shown in further requests

This message is logged every time you get data loss on a request.

I’ve had a quick look at the code, and it figures, since the flag to avoid repeating the message is stored on a _ResponseReader instance, and for each response a new _ResponseReader instance is created.

Relates to #2590 (original implementation).

@Gallaecio Gallaecio added the bug label Dec 5, 2019
@Gallaecio Gallaecio changed the title DOWNLOAD_FAIL_ON_DATALOSS warning is logged for all requests The DOWNLOAD_FAIL_ON_DATALOSS warning is logged for all requests Dec 5, 2019
@Gallaecio Gallaecio changed the title The DOWNLOAD_FAIL_ON_DATALOSS warning is logged for all requests DOWNLOAD_FAIL_ON_DATALOSS logged for all requests Dec 5, 2019
@luchiago
Copy link

luchiago commented Dec 9, 2019

Hi @Gallaecio, I'm a newcomer and I would like to give a try on this.

@Gallaecio
Copy link
Member Author

Go ahead. Although I’m note sure it is best suited for a newcomer; don’t feel bad it seems overwhelming, the affected parts of code are rather complex, it may take some code reading.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants