AWS Lambda Stream yields empty buffer for images of larger size.

[merla]

I'm using nodejs 'gm' module with imagemagick library to create thumbnails of images and upload on S3. It works well with small images. But with large size images it's not working at all. Sometimes I get Stream yields empty buffer and sometimes it ends the request with "Memory Size: 3008 MB Max Memory Used: 3008 MB"
2018-10-11T16:18:49.713Z dc34f79c-cd70-11e8-8e07-ef336d005721 Error: Stream yields empty buffer at Socket.<anonymous> (/var/task/node_modules/gm/lib/command.js:57:17) at emitNone (events.js:111:20) at Socket.emit (events.js:208:7) at endReadableNT (_stream_readable.js:1064:12) at _combinedTickCallback (internal/process/next_tick.js:138:11) at process._tickDomainCallback (internal/process/next_tick.js:218:9)
below is my code to resize
`function process(response, next) {
console.log("process image");

            gm(response).command('convert')
            .resize(_sizesArray[key].width,_sizesArray[key].width)
            .gravity('Center')
            .extent(_sizesArray[key].width,_sizesArray[key].width)
            .toBuffer(
                    'JPG', function(err,
                        buffer) {
                        if (err) {
                            
                            next(err);
                        } else {
                            console.timeEnd(
                                "processImage"
                            );
                            next(null,
                                buffer,
                                key);
                            
                        }
                    });
          
        }`

How can I fix this issue?

[mudassarsaleem]

@merla did you find a solution for this? I have the same issue. for smaller images, it works fine. but for large images, it gives this error.

[mudassarsaleem]

@merla
please check the following issue. I am able to solve the issue with chunking:
#572

[tl24]

You can try using the limit command to limit memory used. I believe imagemagick should spill over to disk if it goes about the limit:

gm(buffer).limit("memory", "1024MB")

[jhondge]

@merla
you can try my solution #572

[VictorioBerra]

This might be unrelated but I didn't start having this buffer error until I changed my node version in lambda from like 6 to 10. I think this might actually be an issue with bumping the node version.

EDIT: CONFIRMED. My problems went away entirely by downgrading to nodejs8.10

[lblo]

nodejs8.10 will be deprecated soon cf. the bellow Amazon email :

For Node.js 8.x, there will be 2 stages to the runtime deprecation process:

1. Disable Function Create – Beginning January 6, 2020, customers will no longer be able to create functions using Node.js 8.10
2. Disable Function Update – Beginning February 3, 2020, customers will no longer be able to update functions using Node.js 8.10

Working solution with nodejs10.x

1. Install the custom layer https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:145266761615:applications~image-magick-lambda-layer
2. Link any lambda function using ImageMagick witht that custom layer
3. Tada!

Source : https://forums.aws.amazon.com/thread.jspa?messageID=906619&tstart=0

Explanation : "ImageMagick is no longer bundled with the Node.js 10.x runtime"

[j0k3r]

@lblo you got the solution. Thanks!

It doesn't work out of the box for us, we needed to increase the memory of the lambda but also, limit the memory consumption of gm to avoid failure, like:

gm(buffer)
    // our lambda has a memory size of 1536, so I put a lower value in the limit
    .limit('memory', 1472)
    .resize(...)
    // etc...

[boycce]

@lblo you got the solution. Thanks!

It doesn't work out of the box for us, we needed to increase the memory of the lambda but also, limit the memory consumption of gm to avoid failure, like:

gm(buffer)
    // our lambda has a memory size of 1536, so I put a lower value in the limit
    .limit('memory', 1472)
    .resize(...)
    // etc...

Using .limit() doesn't seem to work for me, however increasing the lambda memory does:

Error: stream yields empty buffer