RangeError: Array buffer allocation failed

[tahv0]

RangeError: Array buffer allocation failed
    at new ArrayBuffer (<anonymous>)
    at new Uint8Array (<anonymous>)
    at new FastBuffer (buffer.js:79:1)
    at Function.alloc (buffer.js:283:10)
    at new Buffer (buffer.js:180:19)
    at module.exports.Reader.addChunk (/home/node/app/node_modules/packet-reader/index.js:35:21)
    at Socket.<anonymous> (/home/node/app/node_modules/pg/lib/connection.js:117:18)
    at Socket.emit (events.js:182:13)
    at addChunk (_stream_readable.js:283:12)
    at readableAddChunk (_stream_readable.js:264:11)
    at Socket.Readable.push (_stream_readable.js:219:10)
    at Pipe.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

Hi. There is possibly a memory allocation bug. Reader from packet-reader becomes full and it is not cleared after an event is emitted?

I use pg-listen package which uses this library and after like 10k trigger events I got this error.

[williamoliveira]

Having the same problem

03-03-2019 19:56 +00:00: RangeError [ERR_INVALID_OPT_VALUE]: The value "2147483648" is invalid for option "size"
0|api    |     at Function.alloc (buffer.js:279:3)
0|api    |     at module.exports.Reader.addChunk (/root/api/node_modules/packet-reader/index.js:35:28)
0|api    |     at Socket.<anonymous> (/root/api/node_modules/pg/lib/connection.js:117:18)
0|api    |     at Socket.emit (events.js:182:13)
0|api    |     at Socket.EventEmitter.emit (domain.js:460:23)
0|api    |     at addChunk (_stream_readable.js:283:12)
0|api    |     at readableAddChunk (_stream_readable.js:264:11)
0|api    |     at Socket.Readable.push (_stream_readable.js:219:10)
0|api    |     at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

[williamoliveira]

Found my problem, my query was too big, it had over 30k uuids in a where in

[mhtsharma948]

I'm also facing the same issue and I confirmed, this is because of the large set of data, also, this issue is only faced on windows OS for the size of data I'm providing.

[tahv0]

I had very little set of data and relatively small query. OS: Used GAE Flex environment.

[mhtsharma948]

Is there any known temporary solution available for the issue, by the time the issue get resolved?

[kokokenada]

+1 GAE Flex as well. Works fine on local OSX

[tswaters]

Hi, I got this exact error. I'm using pg-copy-streams, for migrating data between environments... this was encountered in a docker container, v10.16.3, debian-stretch. The code works correctly when not run through docker.

The query itself is fairly straight forward -- no large list of entries in an in clause for me. In my case, I have a COPY TO STDOUT and a COPY FROM STDIN piped together:

const sourceStream = sourceClient.query(to("COPY (SELECT * FROM my_table WHERE id IN (SELECT id FROM interesting_things) TO STDOUT"))
const targetStream = targetClient.query(from("COPY my_table FROM STDIN"))
await new Promise((resolve, reject) => {
  targetStream.on('error', reject)
  sourceStream.on('error', reject)
  tartgetStream.on('end', resolve)
  sourceStream.pipe(targetStream)
})

Something like that, anyway. I also have a stream transformer between the two where I make mutations to the data coming in -- and I also have a progress bar that ticks for every new line sent through the source stream.

It's a really interesting error -- the value indicated by the error makes it sound like it's trying to create a 2GB buffer and that fails -- but the entire size of my database up to this point is only 1,424MB

The error returned:

buffer.js:269
    throw err;
    ^

RangeError [ERR_INVALID_OPT_VALUE]: The value "2147483648" is invalid for option "size"
    at Function.alloc (buffer.js:278:3)
    at module.exports.Reader.addChunk (/app/node_modules/packet-reader/index.js:35:28)
    at Socket.<anonymous> (/app/node_modules/pg/lib/connection.js:118:18)
    at Socket.emit (events.js:203:15)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at Socket.Readable.push (_stream_readable.js:224:10)
    at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

Actually, looking around a bit I found a semi-related issue, maybe? This one relates to the same error being raised when a slow target stream is encountered: nodejs/node#25057 --

Based upon that, it may be that my target stream does not have a data handler. I think in my particular case all these things are related.... the target stream is continually buffering data... I have a feeling I'm seeing it only under docker because of the progress bar / logging, maybe? I dunno, I'm just clutching at straws at this point.

I'm going to try and add a dummy data event to the target stream and see how it goes.

[felixfbecker]

It's somewhat unfortunate that pg doesn't seem capable of handling very large queries (even though Postgres can)

[brianc]

It's somewhat unfortunate that pg doesn't seem capable of handling very large queries (even though Postgres can)

yeah every process is going to have some memory constraints. Node doesn't let you make very very large array buffers..its somewhat lower than other runtimes. There are probably some workarounds that could be put in place...one thing is coming up in pg after the 8.0 release I'm going to be refactoring the parser which should reduce the amount of memory it consumes...somewhat significantly in some cases. Can you try running your code with

PG_FAST_CONNECTION=true 

environment variable set? That will opt in to the experimental new parser code. I wouldn't recommend using it in production quite yet, but it'd help me narrow down where the issue might be, as this parser is lighter on memory consumption.

[cxn-pkovacs]

Shouldn't this issue be closed? Do you need some feedback on the "production-ready" implementation in 8.0?

We had the same issue which went away after we upgraded all of our application's dependencies. We haven't ascertained yet which dependency's upgrade fixed it, but PG_FAST_CONNECTION=true helped with the 7.0 pg version (even though it brought with it several protocol-level errors), so...

I mean: reading this thread through and seeing the issue still being open may even make you think that version 8.0 of pg hasn't been released yet and all you have at your disposal to fix/mitigate the issue is a non-production-ready implementation.