PDE-2826 fix(core): backpressure issue with node-fetch patch

[eliangcs]

Using node-fetch 2.6.6 (Node.js 14), this program hangs on my laptop:

const fs = require('fs');
const fetch = require('node-fetch');

async function main() {
  // 36864 bytes = 16 KB
  // If it's 36863, the bug won't happen
  const downloadUrl = 'https://httpbin.org/stream-bytes/36864';
  const downloadResponse = await fetch(downloadUrl);

  const uploadUrl = 'https://httpbin.org/put';
  const uploadOpts = {
    method: 'PUT',
    body: downloadResponse.body,
  };

  const uploadRequest = new fetch.Request(uploadUrl, uploadOpts);
  const uploadPromise = fetch(uploadRequest);
  const uploadResponse = await uploadPromise;
  console.log(uploadResponse.status);
}

main();

When you pass a fetch.Request object to fetch(), fetch() copies the Request object. And if the Request has a streamable body, the Request constructor clones the body by teeing the stream into two PassThrough streams. Then we read one of the PassThrough stream (A) and not the other (B). This causes B's internal buffer to fill up, so Node.js has to pause reading from the source until someone consumes B's buffer.

To fix, we add that someone to consume B. Right after the fetch(uploadRequest) call, we pipe uploadRequest.body to /dev/null:

  const uploadPromise = fetch(uploadRequest);

  uploadRequest.body.pipe(fs.createWriteStream('/dev/null'));  // <---

  const uploadResponse = await uploadPromise;

And that solves the hanging problem.

This PR does something similar with our node-fetch patch.

[xavdid]

wowza! Nice one