Premature close

[weijarz]

import fetch from 'node-fetch'

const response = await fetch('https://fex.baidu.com/feed.xml')
const body = await response.text()

console.log(body)

Throws

file:///home/wjzhou/tmp/tmp/fetchbug/node_modules/node-fetch/src/body.js:210
		const error_ = error instanceof FetchBaseError ? error : new FetchError(`Invalid response body while trying to fetch ${data.url}: ${error.message}`, 'system', error);
		                                                         ^

FetchError: Invalid response body while trying to fetch https://fex.baidu.com/feed.xml: Premature close
    at consumeBody (file:///home/wjzhou/tmp/tmp/fetchbug/node_modules/node-fetch/src/body.js:210:60)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Response.text (file:///home/wjzhou/tmp/tmp/fetchbug/node_modules/node-fetch/src/body.js:129:18)
    at async file:///home/wjzhou/tmp/tmp/fetchbug/b.js:4:14 {
  type: 'system',
  errno: 'ERR_STREAM_PREMATURE_CLOSE',
  code: 'ERR_STREAM_PREMATURE_CLOSE',
  erroredSysCall: undefined
}

software	version
node-fetch	3.0.0-beta.10
node	v14.16.0
npm	7.17.0
Operating System	archlinux

[tekwiz]

@weijarz I'm not getting this error on 3.0.0-beta.10. This likely means that the connection is, in fact, closing prematurely.

[jimmywarting]

using pipeline results in weird behavior...

I'm trying just basic stuff... And comparing what got have done... sindresorhus/got@7b2ccb0

I can't get this fetch('https://fex.baidu.com/feed.xml') to work in beta.10 with node 16.5 locally. Seems like GOT got it right?

import https from 'https'
import Stream, {PassThrough, pipeline} from 'stream';

https.request('https://fex.baidu.com/feed.xml', async (res) => {
  // res.on('data', (chunk) => {
  //   console.log(chunk.byteLength)
  // })
  // res.on('end', () => {
  //   console.log('end')
  // })
  // res.on('error', (err) => {
  //   console.log(err)
  // })
  // res.on('close', () => {
  //   console.log('close')
  // })

  console.log('Starts to read response')

  await pipeline(
    res,
    new Stream.Writable({
      write(chunk, encoding, callback) {
        console.log('got data evt')
        callback()
      }
    }),
    (err) => {
      if (err) {
        console.error('Pipeline failed.', err);
      } else {
        console.log('Pipeline succeeded.');
      }
    }
  )
  console.log('Weird this is called before Pipeline succeeded')

}).end()

Result:

Starts to read response
got data evt
Weird this is called before Pipeline succeeded
got data evt
--||--
got data evt
Pipeline succeeded.

Seems like got ignores the premuture close and continues as usual?
I also wonder where 0\r\n comes from...

node-fetch/src/index.js

Line 290 in b50fbc1

const LAST_CHUNK = Buffer.from('0\r\n');

All responses dose not ends with 0\r\n as in plain text? like binary files such as zip, pdf, movies etc.
I'm starting to wonder if we tackle this all wrong...
should we perhaps use stream.finished instead?

[weijarz]

@tekwiz
Today I tested it on a VPS (debian 10, node v14) and also had this problem.
There are a large number of addresses that will encounter this issue,like

https://linux.cn/rss.xml
https://www.ifanr.com/feed
https://www.zhihu.com/rss

[tekwiz]

It sounds like Node.js 16 has fixed some of the chunked encoding problems. I'll do some more testing today.

[jimmywarting]

Just tried out this 4 on node 16.5 with d19fdac from #1222

...And it seems to work fine now

• https://linux.cn/rss.xml
• https://www.ifanr.com/feed
• https://www.zhihu.com/rss
• https://fex.baidu.com/feed.xml

It sounds like Node.js 16 has fixed some of the chunked encoding problems

That's grate news

It sounds like Node.js 16 has fixed some of the chunked encoding problems. I'll do some more testing today.

@tekwiz I'm marking your PR as draft until you are done testing

[tekwiz]

@weijarz Can you check if this issue is fixed with node-fetch@3.1.0?

[weijarz]

@tekwiz It has been fixed.

[i-am-the-slime]

I'm on 3.2.6 and I still see this. How do I get around it?

[AndreiSoroka]

3.2.8 same problem

[AndreiSoroka]

Ok, I solved that problem. Just migrated to node 18.6

https://nodejs.org/tr/blog/release/v17.5.0/

after 17.5.0 node js have native fetch

[Treverix]

There seems to be a regression. With node 16.15.1, node-fetch 3.1.0 works while more recent versions don't. I downgraded to 3.1.0 for as long as I have to keep using Node 16.x

[kunal15595]

getting this in node 20.5.1, library version 2.6.12

[steveluscher]

If you folks are using an http.Agent with your connections in keepAlive mode with a fixed number of maxSockets, #1767 is the problem.

[lanistor]

Seems fixed in node-fetch@3.3, testing with node v16.19.1