Backend crawl log streaming API: Stream logs from WACZ files after crawl concludes

[tw4l]

Sub-task for #796

Once a crawl is finished, the API endpoint should stream logs from all of the WACZ files created by the crawl.

[tw4l]

First pass is implemented in #682.

We'll want to move to properly streaming logs, currently blocked by aio-libs/aiobotocore#991

[ikreymer]

Until the aiobotocore is resolved, we may be able to use the sync download option, since we've already implemented this to support collection downloads via https://github.com/webrecorder/browsertrix-cloud/blob/main/backend/btrixcloud/storages.py#L358

[tw4l]

Implemented as a sync stream in #1168 . Closing for now, though we may eventually want to make this async.

[tw4l]

Still seems to be a memory issue, looking into it.

Could just fetch from presigned URLs

[tw4l]

@Chickensoupwithrice In your court now if you want to try to figure this out :)

[Chickensoupwithrice]

Alright, after much experimenting I've managed to nail down exactly where we're no longer doing generators and instead load up all the logs. It's in the way we're calling stream_log_bytes_as_line_dicts (which by itself does return a generator) but instead we're extending an array by the output of the generator leading to loading the entire log file into memory and then run out of memory.

Switching extend to append does mean we're generators all the way down, but then when I try to execute on this generator, I get read timeouts on the DO space?

Still investigating.