Error: Response body is unavailable for redirect responses

[simonw]

Got this from:

shot-scraper lite.datasette.io --log-requests -

The error does not show for:

shot-scraper https://lite.datasette.io/ --log-requests -

Full output:

Exception in callback SyncBase._sync.<locals>.callback(<Task finishe...t responses')>) at /Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_sync_base.py:104
handle: <Handle SyncBase._sync.<locals>.callback(<Task finishe...t responses')>) at /Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_sync_base.py:104>
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 105, in callback
    g_self.switch()
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_browser_context.py", line 122, in <lambda>
    lambda params: self._on_response(
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_browser_context.py", line 397, in _on_response
    page.emit(Page.Events.Response, response)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/pyee/_base.py", line 113, in emit
    handled = self._call_handlers(event, args, kwargs)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/pyee/_base.py", line 96, in _call_handlers
    self._emit_run(f, args, kwargs)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/pyee/_asyncio.py", line 42, in _emit_run
    self.emit('error', exc)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/pyee/_base.py", line 116, in emit
    self._emit_handle_potential_error(event, args[0] if args else None)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/pyee/_base.py", line 86, in _emit_handle_potential_error
    raise error
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/pyee/_asyncio.py", line 40, in _emit_run
    coro = f(*args, **kwargs)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_impl_to_api_mapping.py", line 88, in wrapper_func
    return handler(
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/shot_scraper/cli.py", line 730, in on_response
    body = response.body()
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 574, in body
    self._sync("response.body", self._impl_obj.body())
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 111, in _sync
    return task.result()
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_network.py", line 375, in body
    binary = await self._channel.send("body")
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 39, in send
    return await self.inner_send(method, params, False)
  File "/Users/simon/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 63, in inner_send
    result = next(iter(done)).result()
playwright._impl._api_types.Error: Response body is unavailable for redirect responses
{"method": "GET", "url": "https://lite.datasette.io/", "size": 10516, "timing": {"startTime": 1663211415586.7231, "domainLookupStart": 0.219, "domainLookupEnd": 0.229, "connectStart": 0.229, "secureConnectionStart": 17.698, "connectEnd": 55.296, "requestStart": 55.535, "responseStart": 74.426, "responseEnd": 75.545}}
{"method": "GET", "url": "https://plausible.io/js/script.manual.js", "size": 1005, "timing": {"startTime": 1663211415673.0908, "domainLookupStart": 250.503, "domainLookupEnd": 250.529, "connectStart": 250.529, "secureConnectionStart": 360.11, "connectEnd": 393.09, "requestStart": 393.473, "responseStart": 425.886, "responseEnd": 427.409}}
{"method": "GET", "url": "https://latest.datasette.io/-/static/app.css?cead5a", "size": 16230, "timing": {"startTime": 1663211415672.284, "domainLookupStart": 0.293, "domainLookupEnd": 362.088, "connectStart": 362.088, "secureConnectionStart": 381.133, "connectEnd": 432.42, "requestStart": 432.625, "responseStart": 521.151, "responseEnd": 539.768}}
Screenshot of 'http://lite.datasette.io' written to 'lite-datasette-io.4.png'

[simonw]

Relevant code:

shot-scraper/shot_scraper/cli.py

Lines 732 to 736 in c674e9e

	except Error as ex:
	if "Network.getResponseBody" in ex.message:
	size = None
	else:
	raise

[simonw]

I'm going to catch ALL playwright errors here and turn them into size = None.

[simonw]

I'm going to add a status field to the response output too.

[simonw]

With the fix:

% shot-scraper lite.datasette.io --log-requests -
{"method": "GET", "url": "http://lite.datasette.io/", "status": 301, "size": null, "timing": {"startTime": 1663211613523.8718, "domainLookupStart": 2.464, "domainLookupEnd": 3.331, "connectStart": 3.331, "secureConnectionStart": -1, "connectEnd": 18.979, "requestStart": 19.894, "responseStart": 44.126, "responseEnd": 49.278999998932704}}
{"method": "GET", "url": "https://lite.datasette.io/", "status": 200, "size": 10516, "timing": {"startTime": 1663211613570.946, "domainLookupStart": 1.024, "domainLookupEnd": 1.036, "connectStart": 1.036, "secureConnectionStart": 26.774, "connectEnd": 69.628, "requestStart": 70.825, "responseStart": 98.848, "responseEnd": 100.328}}
{"method": "GET", "url": "https://plausible.io/js/script.manual.js", "status": 200, "size": 1005, "timing": {"startTime": 1663211613690.144, "domainLookupStart": 0.104, "domainLookupEnd": 43.69, "connectStart": 43.69, "secureConnectionStart": 68.988, "connectEnd": 105.833, "requestStart": 106.014, "responseStart": 137.988, "responseEnd": 140.421}}
{"method": "GET", "url": "https://latest.datasette.io/-/static/app.css?cead5a", "status": 200, "size": 16230, "timing": {"startTime": 1663211613689.5251, "domainLookupStart": 0.168, "domainLookupEnd": 0.874, "connectStart": 0.874, "secureConnectionStart": 41.048, "connectEnd": 84.11, "requestStart": 85.061, "responseStart": 169.099, "responseEnd": 185.029}}
Screenshot of 'http://lite.datasette.io' written to 'lite-datasette-io.4.png'