Clients read config from file resulting in EMFILE (too many open files) errors #3019
Comments
moltar
added
bug
|
Added a const slurpFile = (path) => new Promise((resolve, reject) => {
console.log(path, new Error().stack)And tested with the following two-liner file: import { SQS } from '@aws-sdk/client-sqs'
const sqs = new SQS({})ResultsAs you can see, it is trying to read the same files twice, and makes at least 4 IO operations per instance of a client. |
|
Now, making this a bit more realistic and instantiating SQS client in a loop. This would be the case when it is instantiated inside a lambda handler. Well, it wouldn't be a loop, but it would be instantiated multiple times when lambda is being executed warm, and very frequently. import { SQS } from '@aws-sdk/client-sqs'
for (const iter of [1, 2]) {
const sqs = new SQS({})
}Then: ResultsIn 8 executions! So the results are never cached globally, and are simply cached per client instance. |
|
Here's a way to remove unnecessary calls: import { SQS } from '@aws-sdk/client-sqs'
const sqs = new SQS({
// removes first 2 calls
credentials: {
accessKeyId: 'x',
secretAccessKey: '',
},
// removes second 2 calls
defaultUserAgentProvider: async () => [],
}) |
|
But wait ... there is more! If you actually try to use the methods of the queue, this will result in even more calls! import { SQS } from '@aws-sdk/client-sqs'
const sqs = new SQS({
credentials: {
accessKeyId: process.env.AWS_ACCESS_KEY_ID || '',
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY || '',
sessionToken: process.env.AWS_SESSION_TOKEN || '',
},
region: 'us-east-1',
defaultUserAgentProvider: async () => [],
})
async function main() {
await sqs.listQueues({})
}
main()ResultsSo we get 8 IO calls for each method call (I tried This is ridiculous and needs to be addressed immediately, as it is causing major issues! |
|
Interestingly, calling |
|
If anyone stumbles on this, the only way I found to shut the file reading, is to do the following hack. import * as sharedIniFileLoader from '@aws-sdk/shared-ini-file-loader'
Object.assign(sharedIniFileLoader, {
loadSharedConfigFiles: async (): Promise<sharedIniFileLoader.SharedConfigFiles> => ({
configFile: {},
credentialsFile: {},
}),
}) |
|
I can confirm that @moltar 's workaround solves this issue in AWS Lambda functions. |
|
The aws-sdk should detect if running in lambda and omit the loadSharedConfigFiles call completely, or at least provide a "supported" way to disable the loadSharedConfigFiles call... |
|
Without this workaround suggested by @moltar it seems v3.49.0 is even getting worse, and opening more file descriptors: probably because of this #3192 ? |
|
Ok, it seems with v3.49.0 it seems these type of EMFILE errors, can still occur (also with that workaround): Error: connect EMFILE 52.94.5.100:443 - Local (undefined:undefined)
Error: getaddrinfo EMFILE dynamodb.eu-west-1.amazonaws.com
Error: A system error occurred: uv_os_homedir returned EMFILE (too many open files)PS. so having a lambda setup, like this is incrementing the probability for such EMFILE errors: https://github.com/fastify/aws-lambda-fastify#lower-cold-start-latency |
import { DynamoDBClient } from '@aws-sdk/client-dynamodb'
import fs from 'fs'
import { lsof } from 'list-open-files'
let readFileCount = 0
const originalReadFile = fs.readFile
fs.readFile = function (f) {
// console.log(`reading file: ${f}`)
readFileCount++
return originalReadFile.apply(fs, arguments)
}
const clientCreations = 1000
for (let index = 0; index < clientCreations; index++) {
new DynamoDBClient()
}
console.log(`made ${clientCreations} client instances`)
console.log(`called readFile ${readFileCount} times`)
lsof().then((ret) => {
const openFiles = ret[0].files.length
console.log(`${openFiles} open fds`)
lsof().then((ret) => {
const openFiles = ret[0].files.length
console.log(`${openFiles} open fds a bit later`)
})
})Beside opening too many file descriptors by reading non-existing config and credential files in lambda... As it seems there is more stuff generating open file descriptors in (v3.47.0 - v3.49.0) then just the readFile file descriptors, I currently stay on v3.46.0 with the above workaround! |
|
@AllanZhengYP @ajredniwja Could you please check into this when you get a chance? This is a serious issue that impacted our production system recently. |
|
@vudh1 you assigned this issue to yourself, can you say something about it? |
|
fyi: found an interesting package: https://github.com/samswen/lambda-emfiles made some tests: Some insights (same workload for all tests) 1) v3.46.0 without any workarounds: Details
=> some leaks (around 230 emfiles) 2) v3.49.0 without any workarounds: Details
=> some leaks and more emfiles (up to more than 600 emfiles) 3) v3.49.0 with loadSharedConfigFiles workaround: Details
=> much less leaks, probably also less emfiles (up to more than 400 emfiles) 4) v3.46.0 with loadSharedConfigFiles workaround: Details
=> much less leaks, much less emfiles (up to 180 emfiles) So seems "4) v3.46.0 with loadSharedConfigFiles workaround" is the best! |
|
An additional curiosity: Defining a custom requestHandler, with a very low socketTimeout reduces drastically the emfiles count: requestHandler: new NodeHttpHandler({
socketTimeout: 10000 // <- this decreases the emfiles count, the Node.js default is 120000
})Details
|
|
Yes, can confirm: still some leaks but, loadSharedConfigFiles workaround + lower socketTimeout = 🚀 👍 (a lot less emfiles) |
Edit: I see the commend: |
|
Wondering if this timeout perhaps need to equal Lambda timeout? Maybe this can be auto-set based on the https://docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html |
|
The only downside with this approach would be that the client has to be instantiated inside the handler. And I usually like to have them outside of the handler to preserve the instances across invocations. |
|
As this seems to be the timeout for an idle socket, I think in lambda this could be automatically defaulted to a very low value, at least magnitute of 10... so 12 seconds instead of 2 minutes... but even lower so 1 second for example, should work. Would be nice if some AWS internal engineer would help to argument on this topic. proposal: // https://github.com/aws/aws-sdk-js-v3/blob/main/packages/node-http-handler/src/node-http-handler.ts#L62
socketTimeout: socketTimeout || 10000and probably also here? // https://github.com/aws/aws-sdk-js-v3/blob/main/packages/node-http-handler/src/node-http2-handler.ts#L44
this.requestTimeout = requestTimeout || 10000; |
|
This issue has been open for nearly three months and no one from AWS has chimed in. I'll reach out to my account manager and see if they can help move things along here. |
https://twitter.com/trivikram/status/1489588283726204928?s=21 |
|
Thanks for that update! |
|
AWS SDK for JavaScript team discussed this issue in our scrum today. |
here we go ;-) #3279 |
|
Pardon any confusion on my part, but it seems like the file descriptor issue is happening for every single API call, not just upon instantiation. I'm only instantiating |
That's why I created #3279 😉 |
|
@trivikr Thanks, Trivikram, much appreciated! |
Thank you for looking at the issue. But why do we need to have any calls in the lambda environment? We need some way to disable config reading completely. Thank you. |
Since I spent too much time on this issue, I refactored shared-ini-file-loader into multiple components and posted an update in #3285 The fix is a merge of both #3281 and #3282:
|
@moltar Can you create a feature request for this question/proposal? |
|
I feel this is part of the same issue, no? The client should not check the disk for credentials if there are env variables set, which is always the case in the Lambda env. I can see it still trying to load I think one approach would be to continue make use the provider pattern, as it is already applied. Just make sure the SDK clients respect any explicit providers given. E.g. could be If you think this is a viable approach, I will open a separate issue. |
A request to create a separate issue is for better tracking. This issue has already got 40 comments. The discussions in this request are currently for reducing readFile calls. In the other issue, we can limit the discussions about skipping readFiles altogether in Lambda environments as you suggested, and how to approach that in AWS SDK for JavaScript and other SDKs.
Yup, this is one of the approaches. Please create a feature request with that suggestion, and other alternatives if you have any. |
|
I'm still seeing this issue in 3.52.0. |
It's probably caused by the sdk requests: #3279 |
The workaround in #3019 (comment) will remove the remaining two readFile calls per client, but the #3279 will still remain. Please use low socket timeouts as suggested by @adrai |
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread. |
|
Reopening this issue per https://twitter.com/thdxr/status/1696615147295379586 |
|
From the tweet
It's very clear that the issue discussed in this specific report was resolved in #3285 I also verified that another issue created as a follow-up was also resolved in #3279 (comment), where we resolve config provider only once. |
|
If you're coming across EMFILE or any other issues in your setup, please create a new feature request or upvote existing ones if they're already reported. |
Describe the bug
This applies to every auto-generated client, but I will use
client-sqsas an example.This is related to: #2271, #2027, #2993
Note that in the above snippet,
loadNodeConfigis called multiple times to get the defaults for the client configuration:aws-sdk-js-v3/clients/client-sqs/src/runtimeConfig.ts
Lines 31 to 53 in bda3b40
Which is the exported fn
loadConfigfrom@aws-sdk/node-config-providerpackage:aws-sdk-js-v3/packages/node-config-provider/src/configLoader.ts
Lines 27 to 37 in 7392341
Which then uses
fromSharedConfigFilesto load config values from the disk.This is a very undesirable "feature", especially in the serverless environments.
Because under heavy load this results in the following errors:
Yes the call is memoized, but if your Lambda is getting executed heavily, and installation happens within the handler, then this call happens multiple times.
Your environment
SDK version number
@aws-sdk/client-sqs@3.40.0Is the issue in the browser/Node.js/ReactNative?
Node.js
Details of the browser/Node.js/ReactNative version
node -vv14.17.5Steps to reproduce
Observed behavior
Expected behavior
Screenshots
N/A
Additional context
N/A
The text was updated successfully, but these errors were encountered: