-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Data Model Deployment Fails With Exit Code 7 (Error: EMFILE: too many open files) #414
Comments
This error seems to be related to relationships (and more than likely the complexity of your project). If I add a relationship to a model, deploy that change, and it deploys successfully, deploying another relationship addition right after that fails. With 100% consistency in my case right now. If, after that relationship deployment failure, I add a regular field to a model and deploy that change, that deployment works. And, more interestingly, after that successful regular field deployment, if I add a relationship to a model and deploy that change, it then also works. With 100% consistency in my case right now. So, doing a regular field addition (a regular field removal also does it unless I'm mistaken), seems to somehow "reset" something. |
@kkh2 Sorry for the inconvenience. We are investigating this. Will provide an update once we have a fix. |
Just wanted to echo I'm having the same issue here |
@prajwalbv99 As the project has grown in complexity, the above workaround I mentioned no longer works. As soon as I add a relationship (one to many), deployment fails with the above EMFILE error. From what I have investigated myself, this is related to Node.js opening too many files (or, file descriptors) asynchronously and then being unable to handle it. What people typically do to get around this is to utilize a package named graceful-fs to limit/defer/queue the amount of open connections until they can be handled. I see that graceful-fs is installed when listing the packages in CloudShell but I don't know if it's being utilized in the AWS code. Is there anything I can do to get around this EMFILE issue myself? |
@kkh2 Apologies for the frustrating experience here. We were able to identify that the EMFILE issue is due to a dependency that we are not able to quickly and/or cleanly remove at the moment. In order to accommodate larger projects like yours, we have increased the default ulimit on the Job container. This change is rolling through the pipeline now and we'll update here when its available in us-east-1. |
@srquinn21 Roger that. Appreciate the help! |
Just wanted to report back and confirm that everything has been working flawlessly since the ulimit increase. |
@kkh2 Thats great to hear! Thanks for the report and hanging in there while we got it fixed. Cheers. |
@srquinn21 Having this issue again. Fails to deploy even the current models with no changes made. Almost like it was working as I was building the project, I then went past a certain threshold, and am now stuck with what I have. Was the default ulimit decreased back to what it used to be? |
@kkh2 Same AppId/region? |
I am also having this issue. appid - d3mrkvd23hhmdy I am trying to update the auth rules on the model. it's not working viea my schema.graphql file so I'm trying to update it from Amplify console. It already has API Key authentication but I want to add Cognito auth. If I add it and then save and deploy, the first attemp always fails. The second attemt passes but it doesn't update the model (if I navigate back to the auth options cognito is not showing) |
We have a schema.graphql with 54 tables and 31 enums. There are 40 many:many relations across the tables. This is by no means a massive schema, and a good size for a moderately complex SaaS app of this type. I would expect that Amplify could handle this size schema. We are still getting the EMFILE error and now ConnectionStack errors and "Limit on the number of resources in a single stack operation exceeded" errors - both seems to be Amplify not creating CloudFormation stacks inside CloudFormation resource limits. I believe this is all related to this issue: aws-amplify/amplify-cli#9762 (comment) (@ Straubulous closed that issue on 4 May 2022 but we are still getting both errors with the latest version of the CLI). @srquinn21 - any progress on resolving this? It's killing our ability to keep our model up to date. We need to create a new app based off this schema but there's no way I can trust Amplify to support any further increase in schema complexity at this point. |
Why is this closed- this is an issue is ap-south-1 region too, we are seeing the same |
Before opening, please confirm:
App Id
d3olzpxyvy8jqo
Region
us-east-1
Environment name
staging
Amplify CLI Version
No response
If applicable, what version of Node.js are you using?
No response
What operating system are you using?
Mac
Browser type?
Safari
Describe the bug
Command failed with exit code 7: amplify push --yes --allow-destructive-graphql-schema-updates - Fetching updates to backend environment: staging from the cloud. - Building resource [...] Uploading files... internal/process/per_thread.js:152 _memoryUsage(memValues); ^ Error: EMFILE: too many open files, uv_resident_set_memory at process.memoryUsage (internal/process/per_thread.js:152:5) at Object.exception.getProcessInfo (/opt/amazon/lib/node_modules/@aws-amplify/cli/node_modules/cfn-lint/node_modules/winston/lib/winston/exception.js:33:26) at Object.exception.getAllInfo (/opt/amazon/lib/node_modules/@aws-amplify/cli/node_modules/cfn-lint/node_modules/winston/lib/winston/exception.js:17:24) at exports.Logger.Logger._uncaughtException (/opt/amazon/lib/node_modules/@aws-amplify/cli/node_modules/cfn-lint/node_modules/winston/lib/winston/logger.js:636:24) at process.emit (events.js:326:22) at process.EventEmitter.emit (domain.js:483:12) at process.emit (/opt/amazon/lib/node_modules/@aws-amplify/cli/node_modules/source-map-support/source-map-support.js:516:21) at process._fatalException (internal/process/execution.js:165:25) { errno: -24, code: 'EMFILE', syscall: 'uv_resident_set_memory' }⚠️ Amplify command hooks are disabled in the current execution environment. ⚠️ See https://docs.amplify.aws/cli/usage/command-hooks/ for more information. �[32mGraphQL schema compiled successfully.�[39m �[32mEdit your schema at [...]
Expected behavior
Expecting to deploy normally. Not an overly complex project but seems to be hitting some kind of a limit. Assuming, if relevant, ulimit -n would yield 1024. Not sure if that is being exceeded and/or if that could be upped for Amplify somewhere. Are there configuration settings somewhere to combat errors like this?
Reproduction steps
Additional information
No response
The text was updated successfully, but these errors were encountered: