Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG] Make Filesystem storage backend default again. #541

Merged
merged 1 commit into from Jan 17, 2014

Conversation

@kmike
Copy link
Member

@kmike kmike commented Jan 16, 2014

This is a temporary solution for #500.

@dangra
Copy link
Member

@dangra dangra commented Jan 17, 2014

To be or not to be part of 0.22? /cc @pablohoffman @shane42 @redapple

@kmike
Copy link
Member Author

@kmike kmike commented Jan 17, 2014

My vote is to make filesystem storage default for 0.22 and properly fix GH-500 in 0.24, but I'm biased :)

@pablohoffman
Copy link
Member

@pablohoffman pablohoffman commented Jan 17, 2014

I feel less qualified to make a decision about this, since I haven't used httpcache myself in more than a year, so whatever you guys decide is fine for me.

@rmax
Copy link
Contributor

@rmax rmax commented Jan 17, 2014

I have conflicted feelings about this. It seems this only affects hardcore
spider developers who iterate fast (thus stopping the crawls uncleanly).

But so far this haven't been an issue that affects a wide user base,
otherwise this would popup in scrapy-users years ago. Therefore the current
default is good enough, and hardcore developers can always change the cache
backend anytime.

My vote is to keep the current default and work in a solution to this issue
for the next release.

On Fri, Jan 17, 2014 at 11:52 AM, Pablo Hoffman notifications@github.comwrote:

I feel less qualified to make a decision about this, since I haven't used
httpcache myself in more than a year, so whatever you guys decide is fine
for me.


Reply to this email directly or view it on GitHubhttps://github.com//pull/541#issuecomment-32617090
.

@dangra
Copy link
Member

@dangra dangra commented Jan 17, 2014

It's a backward incompatible change, but it must happen at some point.

Ideally Scrapy should warn the user if the project is using the old backend and HTTPCACHE_STORAGE is not set.

anyway, I vote to switch the backend and make it clear in the release notes, no warning in code.

@kmike
Copy link
Member Author

@kmike kmike commented Jan 17, 2014

@darkrho - based on last year's chat log, at least 6 SH developers were bitten by this issue. This is a big number, considering that not all developers use http cache locally, and that not all developers talk about that in chat. I don't know why isn't it a visible issue in scrapy-users group. Maybe people don't use http cache during development much, or they feel shame that they kill the spider and think it's an error on their side.

dangra added a commit that referenced this pull request Jan 17, 2014
[MRG] Make Filesystem storage backend default again.
@dangra dangra merged commit 3c1e226 into scrapy:master Jan 17, 2014
1 check passed
1 check passed
default The Travis CI build passed
Details
@kmike kmike deleted the kmike:fs-as-default-cache branch Jun 24, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants