DataLoss error when content-length doesn't match actual body size

[rmax]

The example website below returns a Content-Length header larger than the actual response size. The webserver itself is broken (or misconfigured) because it returns the Content-Length from the uncompressed body while it sends the compressed body.

$ scrapy shell "http://www.ebk-gruppe.com/"  --set RETRY_ENABLED=0
2017-02-23 00:26:18 [scrapy.utils.log] INFO: Scrapy 1.3.2 started (bot: scrapybot)
2017-02-23 00:26:18 [scrapy.utils.log] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0, 'RETRY_ENABLED': '0', 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter'}
2017-02-23 00:26:18 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.memusage.MemoryUsage']
2017-02-23 00:26:19 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-02-23 00:26:19 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-02-23 00:26:19 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-02-23 00:26:19 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6025
2017-02-23 00:26:19 [scrapy.core.engine] INFO: Spider opened
Traceback (most recent call last):
  File "<snip>/bin/scrapy", line 11, in <module>
    load_entry_point('Scrapy', 'console_scripts', 'scrapy')()
  File "<snip>/scrapy/scrapy/cmdline.py", line 142, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "<snip>/scrapy/scrapy/cmdline.py", line 88, in _run_print_help
    func(*a, **kw)
  File "<snip>/scrapy/scrapy/cmdline.py", line 149, in _run_command
    cmd.run(args, opts)
  File "<snip>/scrapy/scrapy/commands/shell.py", line 73, in run
    shell.start(url=url, redirect=not opts.no_redirect)
  File "<snip>/scrapy/scrapy/shell.py", line 48, in start
    self.fetch(url, spider, redirect=redirect)
  File "<snip>/scrapy/scrapy/shell.py", line 115, in fetch
    reactor, self._schedule, request, spider)
  File "<snip>/twisted/src/twisted/internet/threads.py", line 122, in blockingCallFromThread
    result.raiseException()
  File "<snip>/twisted/src/twisted/python/failure.py", line 372, in raiseException
    raise self.value.with_traceback(self.tb)
twisted.web._newclient.ResponseFailed: [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>, <twisted.python.failure.Failure twisted.web.http._DataLoss: >]

[rmax]

The PR #2585 fixes this issue by adding a new setting DOWNLOADER_ALLOW_DATA_LOSS and customizing the transfer decoders to allow transfer data loss, that is, not raising _DataLoss nor PotentialDataLoss exceptions.

$ scrapy shell "http://www.ebk-gruppe.com/"  --set RETRY_ENABLED=0 --loglevel WARN --set DOWNLOADER_ALLOW_DATA_LOSS=1 -c "len(response.body)"
2017-02-23 00:32:50 [scrapy.core.downloader.handlers.http11] WARNING: Data loss, not all data received
8493

However, the fix requires to apply two patches to twisted ( twisted/twisted#711 and twisted/twisted#712 ) to allow to a clean solution.

[redapple]

I had a look at a similar issue with HTTP 3xx in #2493 (comment)
There, I suggested a patch catching _DataLoss and flagging it only, as for PotentialDataLoss.
(So maybe twisted/twisted#711 is not strictly needed.)

twisted/twisted#711 and twisted/twisted#712 look useful nonetheless.

[rmax]

@redapple nice approach without requiring twisted patches. I made a PR #2590 based on your suggestion and it works well, although the tests were a bit tricky.

[rmax]

Fixed by #2590

[lzyADA]

Hello, scrapy [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.><twisted.python.failure.Failure twisted.web.http._DataLoss: Chunked decoder in 'CHUNK_LENGTH' state , still expecting more data to get to 'FINISHED' state.>
Pull the request