[MRG+1] [settings/default_settings.py] dont retry 400 #1289
Some servers don't follow this suggestion - e.g. according to Amazon DynamoDB docs client should retry some of requests if their server returned HTTP 400 error.
For me changing it to match HTTP specs makes sense, +1 to change it.
But I wonder why was 400 in the list in the first place. It was in Scrapy since forever - HTTP 400 is in list in the initially imported code.
interesting, but still only some of errors related to 400 should be retried so even for DynamoDB retrying all 400 by default is not recommended, only 4 out of 12 types of 400 responses should be retried according to their docs.
I thought about raising this because it seems dangerous. If you have broken spider generating 100k requests all of them returning 400, with current default Scrapy settings your spider will most likely generate 300k bad requests...