New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Getting EMFILE error on @aws-sdk/client-s3 #5273
Comments
Hi @sumangalam17 thanks for opening this issue, seems like you are using v2 of the SDK and not using ``@aws-sdk/client-s3. I would recommend just using try/catch with await. Can you try to change your code to something like below for V2 const AWS = require('aws-sdk');
const s3 = new AWS.S3({ maxAttempts: 3 });
const saveToS3 = async(bucketName, data) => {
const uploadParams = {
Bucket: bucketName,
Body: JSON.stringify(data),
Key: "file.json"
};
try {
const response = await s3.putObject(uploadParams).promise();
console.log(`Successfully uploaded the file: `, response);
return response;
} catch (err) {
console.error(`Error: uploading`, err);
throw err;
}
} V3 would look like: const { S3Client, PutObjectCommand } = require("@aws-sdk/client-s3");
const s3 = new S3Client({ region: "YOUR_REGION" });
const saveToS3 = async (bucketName, data) => {
const uploadParams = {
Bucket: bucketName,
Body: JSON.stringify(data),
Key: "file.json"
};
try {
const response = await s3.send(new PutObjectCommand(uploadParams));
console.log(`Successfully uploaded the file: `, response);
return response;
} catch (err) {
console.error(`Error: uploading`, err);
throw err;
}
} I tried to call this function several times and wasnt able to reproduce the error. (async() => {
try {
const bucketName = "bucket-name";
for (let i = 0; i < 1000; i++) {
const data = { key: `xyz-${i}` };
await saveToS3(bucketName, data);
}
} catch (error) {
console.error("Error uploading:", error);
}
})(); Please let us know if that doesnt help. thanks |
its not due to
actually its not due to calling that function many time its due to we are triggering sendMessage and that triggering this function and calling client and function both. I tried using V3 and try and catch also but still its not working. |
Definitely seen this when we recently upgraded our v3 AWS dependencies recently. Had to roll back that upgrade. We seemed to see this consistently on many services. Haven't had the time to properly investigate it. But it looked like potentially TCP connections exceeding whatever a given Lambda can cope with. Only happened on Lambda functions that were being called a bunch. |
I am on the same boat here, when using this Javascript SDKv3 s3 client. running in Lambda {
"errorType": "Error",
"errorMessage": "bind EMFILE 0.0.0.0",
"code": "EMFILE",
"errno": -24,
"syscall": "bind",
"address": "0.0.0.0",
"stack": [
"Error: bind EMFILE 0.0.0.0",
" at __node_internal_captureLargerStackTrace (node:internal/errors:496:5)",
" at __node_internal_exceptionWithHostPort (node:internal/errors:671:12)",
" at node:dgram:364:20",
" at process.processTicksAndRejections (node:internal/process/task_queues:83:21)"
]
} And going through my code, always using await with the |
We have been facing this issue too, but with DynamoDb. We had a lambda being called multiple times concurrently (~503) and this lambda calls DynamoDB for a query. Out of those 503 lambdas, about 93 failed with the error:
Any known solution for this? Some forums mentioned it might be a problem with how connections are handled in node. We just upgraded to V3 and as per docs |
I'm also having this issues since passing to v3 for a number of lambda when initializing a new instance of DynamoDb class. Could we have an update on this please ? |
I get lot of emfile messages when try to call any api gw endpoint from lambda. Still stuck till now. Most of the articles I have tried, but no luck |
Describe the bug
SDK version number Which JavaScript Runtime is this issue in? Details of the browser/Node.js/ReactNative version Reproduction Steps Dynamodb client 1:
Dynamodb client 2:
s3 clients:
Large amounts of data processed in this way leads to repetetive errors with emfile |
Hi everyone on the thread, Sorry for the late response. A lot of folks have commented here and shared info, but the part I'm suspicious about is the actual Lambda handler function itself. In order to limit the amount of open file descriptors, you can take advantage of lambda's re-use of the execution environment. By instantiating the SDK client outside of the handler's codepath, you will ensure you have one sdk client and that connections will be manged more efficiently. I added a code example to our docs https://github.com/aws/aws-sdk-js-v3?tab=readme-ov-file#best-practices Let me know if this helps. |
@RanVaknin hey, just a quick note, any idea on how to organize cross-account communication? Or maybe suitable sample, whenever we have in equation at least 2 aws accounts that share services? |
Hi @randomhash , Like I mentioned in my comment above, it's hard to advise on a specific scenario without seeing your actual handler code. If you can share a code example of what your application does we might be able to help you re-arrange your code in a way that avoids recreation of the SDK clients. Thanks, |
This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing. |
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread. |
Checkboxes for prior research
Describe the bug
trying to call this function from sendMessage and getting error like below
error :
tried every possibilities mention in this issue #4345
like calling client globally
using destroy method
setting connection 1 and all
SDK version number
@aws-sdk/package-name@version, 3.418.0
Which JavaScript Runtime is this issue in?
Node.js
Details of the browser/Node.js/ReactNative version
types/node -v 20.2.1
Reproduction Steps
trying to call this function many time from send message
Observed Behavior
n/A
Expected Behavior
n/A
Possible Solution
No response
Additional Information/Context
No response
The text was updated successfully, but these errors were encountered: