Data Model Deployment Fails With Exit Code 7 (Error: EMFILE: too many open files)

[kkh2]

Before opening, please confirm:

• I have searched for duplicate or closed issues.
• I have read the guide for submitting bug reports.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

App Id

d3olzpxyvy8jqo

Region

us-east-1

Environment name

staging

Amplify CLI Version

No response

If applicable, what version of Node.js are you using?

No response

What operating system are you using?

Mac

Browser type?

Safari

Describe the bug

Command failed with exit code 7: amplify push --yes --allow-destructive-graphql-schema-updates - Fetching updates to backend environment: staging from the cloud. - Building resource [...] Uploading files... internal/process/per_thread.js:152 _memoryUsage(memValues); ^ Error: EMFILE: too many open files, uv_resident_set_memory at process.memoryUsage (internal/process/per_thread.js:152:5) at Object.exception.getProcessInfo (/opt/amazon/lib/node_modules/@aws-amplify/cli/node_modules/cfn-lint/node_modules/winston/lib/winston/exception.js:33:26) at Object.exception.getAllInfo (/opt/amazon/lib/node_modules/@aws-amplify/cli/node_modules/cfn-lint/node_modules/winston/lib/winston/exception.js:17:24) at exports.Logger.Logger._uncaughtException (/opt/amazon/lib/node_modules/@aws-amplify/cli/node_modules/cfn-lint/node_modules/winston/lib/winston/logger.js:636:24) at process.emit (events.js:326:22) at process.EventEmitter.emit (domain.js:483:12) at process.emit (/opt/amazon/lib/node_modules/@aws-amplify/cli/node_modules/source-map-support/source-map-support.js:516:21) at process._fatalException (internal/process/execution.js:165:25) { errno: -24, code: 'EMFILE', syscall: 'uv_resident_set_memory' } ⚠️ Amplify command hooks are disabled in the current execution environment. ⚠️ See https://docs.amplify.aws/cli/usage/command-hooks/ for more information. �[32mGraphQL schema compiled successfully.�[39m �[32mEdit your schema at [...]

Expected behavior

Expecting to deploy normally. Not an overly complex project but seems to be hitting some kind of a limit. Assuming, if relevant, ulimit -n would yield 1024. Not sure if that is being exceeded and/or if that could be upped for Amplify somewhere. Are there configuration settings somewhere to combat errors like this?

Reproduction steps

1. Deploy data model model.

Additional information

No response

[kkh2]

This error seems to be related to relationships (and more than likely the complexity of your project).

If I add a relationship to a model, deploy that change, and it deploys successfully, deploying another relationship addition right after that fails. With 100% consistency in my case right now.

If, after that relationship deployment failure, I add a regular field to a model and deploy that change, that deployment works. And, more interestingly, after that successful regular field deployment, if I add a relationship to a model and deploy that change, it then also works. With 100% consistency in my case right now.

So, doing a regular field addition (a regular field removal also does it unless I'm mistaken), seems to somehow "reset" something.

[prajwalbv99]

@kkh2 Sorry for the inconvenience. We are investigating this. Will provide an update once we have a fix.

[johncarmack1984]

Just wanted to echo I'm having the same issue here

[kkh2]

@prajwalbv99 As the project has grown in complexity, the above workaround I mentioned no longer works. As soon as I add a relationship (one to many), deployment fails with the above EMFILE error. From what I have investigated myself, this is related to Node.js opening too many files (or, file descriptors) asynchronously and then being unable to handle it. What people typically do to get around this is to utilize a package named graceful-fs to limit/defer/queue the amount of open connections until they can be handled. I see that graceful-fs is installed when listing the packages in CloudShell but I don't know if it's being utilized in the AWS code. Is there anything I can do to get around this EMFILE issue myself?

[srquinn21]

@kkh2 Apologies for the frustrating experience here. We were able to identify that the EMFILE issue is due to a dependency that we are not able to quickly and/or cleanly remove at the moment. In order to accommodate larger projects like yours, we have increased the default ulimit on the Job container. This change is rolling through the pipeline now and we'll update here when its available in us-east-1.

[kkh2]

@srquinn21 Roger that. Appreciate the help!

[kkh2]

Just wanted to report back and confirm that everything has been working flawlessly since the ulimit increase.

[srquinn21]

@kkh2 Thats great to hear! Thanks for the report and hanging in there while we got it fixed. Cheers.

[kkh2]

@srquinn21 Having this issue again. Fails to deploy even the current models with no changes made. Almost like it was working as I was building the project, I then went past a certain threshold, and am now stuck with what I have. Was the default ulimit decreased back to what it used to be?

[srquinn21]

@kkh2 Same AppId/region?

[gothriveandy]

I am also having this issue.

appid - d3mrkvd23hhmdy
region - ap-southeast-2

I am trying to update the auth rules on the model. it's not working viea my schema.graphql file so I'm trying to update it from Amplify console. It already has API Key authentication but I want to add Cognito auth. If I add it and then save and deploy, the first attemp always fails. The second attemt passes but it doesn't update the model (if I navigate back to the auth options cognito is not showing)

[DougalW]

We have a schema.graphql with 54 tables and 31 enums. There are 40 many:many relations across the tables.

This is by no means a massive schema, and a good size for a moderately complex SaaS app of this type. I would expect that Amplify could handle this size schema.

We are still getting the EMFILE error and now ConnectionStack errors and "Limit on the number of resources in a single stack operation exceeded" errors - both seems to be Amplify not creating CloudFormation stacks inside CloudFormation resource limits.

I believe this is all related to this issue: aws-amplify/amplify-cli#9762 (comment) (@ Straubulous closed that issue on 4 May 2022 but we are still getting both errors with the latest version of the CLI).

@srquinn21 - any progress on resolving this? It's killing our ability to keep our model up to date. We need to create a new app based off this schema but there's no way I can trust Amplify to support any further increase in schema complexity at this point.

[BBopanna]

Why is this closed- this is an issue is ap-south-1 region too, we are seeing the same