Unable to load a large content

[derevnjuk]

Environment

Component	Version
Node.js	v16.19.0
Client (Chrome/Chromium/...)	Chrome/Chromium
OS running Node.js	MacOS
OS running the client	MacOS
chrome-remote-interface	v0.32.0

Is the client running in a container? NO

Description

Unable to retrieve a response body that is greater than 100MB. The maximum size of a message that can be sent through Chrome DevTools is 256 MB: https://source.chromium.org/chromium/chromium/src/+/main:content/browser/devtools/devtools_http_handler.cc;l=90?q=kSendBufferSizeForDevTools&ss=chromium

Steps to reproduce

1. Server an index.html file that loads a JSON greater than 100MB in size. Example:

index.js

const express = require('express');
const app = express();

app.get('/', (_, res) => res.sendFile('index.html', { root: __dirname }));

app.get('/api/keys', (_, res) =>
 res.json({
   keys: Array(1000)
     .fill(0)
     .map((_noop, idx) => ({
       key: randomBytes(100000).toString('hex'),
       id: idx
     }))
 })
);

app.listen(3000, () => {
 console.log("Server started on port 3000");
});

index.html

<p id="result"></p>
<script>
 const result = document.querySelector("#result");

 fetch("/api/keys")
   .then((response) => response.json())
   .then((data) => {
     result.innerHTML = `Found ${data.keys.length} keys`
   });
</script>

2. Open the root page and initiate the listening process for incoming requests.

const maxSize = 256 * 1024 * 1204;
const client = await CDP();

client.Network.enable({
    maxResourceBufferSize: maxSize,
    maxTotalBufferSize: maxSize,
});

let targetRequestId;

client.Network.on('requestWillBeSent', async ({request, requestId}) => {
    if (request.url.includes('/api/keys')) {
        targetRequestId = requestId;
    }
});
client.Network.on('loadingFinished', async ({requestId}) => {
    try {
        if (!targetRequestId || targetRequestId !== requestId) {
            return;
        }
        console.log(`Loading the content for ${requestId}`);
        const res = await client.Network.getResponseBody({requestId});
        console.log(`Content loaded: ${Buffer.from(res.body).byteLength} bytes`);
    } catch (e) {
        console.error('Unable to load the content due to the error:');
        console.error(e);
    }
});
client.once('ready', () => client.Page.navigate({url: 'http://localhost:3000'}));
client.once('disconnect', () => console.log('The socket disconnected.'));

Actual behavior

The connection has been unexpectedly terminated, resulting in the inability to retrieve the content.

For details please refer to puppeteer/puppeteer#4543 and NeuraLegion/cypress-har-generator#177