Intermittent "no such file or directory" failure during deploy

[VirtuaBoza]

There seems to be some race condition in the deployment process because occasionally I get an error like so when running sst deploy:

|  Updating    MarcRecordQueue sst:aws:Queue → MarcRecordQueueSubscriberRole aws:iam:Role
|  Error       
|  Error: Failed to build function: Error: ENOENT: no such file or directory, rename '/Users/aboza/coracorp/stacks-ils-ion/.sst/artifacts/BookEmbeddingQueueSubscriber-src/packages/functions/src/consumeBookEmbeddingQueue/handler.mjs.map' -> '/Users/aboza/coracorp/stacks-ils-ion/.sst/artifacts/BookEmbeddingQueueSubscriber-map/handler.mjs.map'
|      at file:///Users/aboza/coracorp/stacks-ils-ion/.sst/platform/src/components/aws/function.ts:1103:21 {
|    promise: Promise { <rejected> [Circular *1] }
|  }

_ION-217

[thdxr]

by any chance do you also have sst dev running? or any background processes related to sst that are orphaned?

[VirtuaBoza]

by any chance do you also have sst dev running? or any background processes related to sst that are orphaned?

Not to my knowledge. Is there a way to verify that if I see this error again?

[thdxr]

depending on your os something like ps aux | grep sst

[VirtuaBoza]

depending on your os something like ps aux | grep sst

Thanks. the next time it happens, i'll run that and post the output.

[VirtuaBoza]

@thdxr It happened again. Running ps aux | grep sst prints:

aboza            74190   0.0  0.1 409687776  27296   ??  S    Sat08PM   0:00.55 node /Users/aboza/coracorp/stacks-ils-ion/.sst/platform/node_modules/@pulumi/pulumi/cmd/dynamic-provider/index.js 127.0.0.1:52524
aboza            22094   0.0  0.0 408102576   1200 s010  S+    3:26PM   0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox sst

[VirtuaBoza]

PS. I can't be 100% sure, but it seems like I have better luck getting a successful deploy if I run it with --verbose.

[VirtuaBoza]

PS. I can't be 100% sure, but it seems like I have better luck getting a successful deploy if I run it with --verbose.

Ok debunked that theory lol.

[VirtuaBoza]

@thdxr It happened again. Running ps aux | grep sst prints:

aboza            74190   0.0  0.1 409687776  27296   ??  S    Sat08PM   0:00.55 node /Users/aboza/coracorp/stacks-ils-ion/.sst/platform/node_modules/@pulumi/pulumi/cmd/dynamic-provider/index.js 127.0.0.1:52524
aboza            22094   0.0  0.0 408102576   1200 s010  S+    3:26PM   0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox sst

I just tried killing 74190 and running deploy again. Still failing. That process did not show back up afterward.

[VirtuaBoza]

I'm on v0.0.229 now and still having similar errors. Here's the latest:

|  Error       
|  Error: failed to register new resource TopRevalidationSeederCode [aws:s3/bucketObjectv2:BucketObjectv2]: 2 UNKNOWN: failed to compute archive hash for "source": open /Users/aboza/coracorp/stacks-ils-ion/.sst/artifacts/TopRevalidationSeeder/code.zip: no such file or directory
|      at Object.registerResource (/Users/aboza/coracorp/stacks-ils-ion/.sst/platform/node_modules/@pulumi/runtime/resource.ts:444:27)
|      at new Resource (/Users/aboza/coracorp/stacks-ils-ion/.sst/platform/node_modules/@pulumi/resource.ts:507:13)
|      at new CustomResource (/Users/aboza/coracorp/stacks-ils-ion/.sst/platform/node_modules/@pulumi/resource.ts:962:9)
|      at new BucketObjectv2 (/Users/aboza/coracorp/stacks-ils-ion/.sst/platform/node_modules/@pulumi/s3/bucketObjectv2.ts:439:9)
|      at createBucketObject (file:///Users/aboza/coracorp/stacks-ils-ion/.sst/platform/src/components/aws/function.ts:1353:14)
|      at new _Function (file:///Users/aboza/coracorp/stacks-ils-ion/.sst/platform/src/components/aws/function.ts:911:18)
|      at file:///Users/aboza/coracorp/stacks-ils-ion/.sst/platform/src/components/aws/nextjs.ts:1003:26
|      at /Users/aboza/coracorp/stacks-ils-ion/.sst/platform/node_modules/@pulumi/output.ts:404:31
|      at Generator.next (<anonymous>)
|      at /Users/aboza/coracorp/stacks-ils-ion/.sst/platform/node_modules/@pulumi/pulumi/output.js:21:71 {
|    promise: Promise { <rejected> [Circular *1] }
|  }

[VirtuaBoza]

And again, slightly different output:

×  Failed
   [Error: ENOENT: no such file or directory, open '/Users/aboza/coracorp/stacks-ils-ion/.sst/artifacts/TopRevalidationQueueSubscriber/code.zip'] {
     errno: -2,
     code: 'ENOENT',
     syscall: 'open',
     path: '/Users/aboza/coracorp/stacks-ils-ion/.sst/artifacts/TopRevalidationQueueSubscriber/code.zip',
     promise: Promise { <rejected> [Circular *1] }
   }

I only started getting these errors related to my Next component(s) when I added a second Next component, and it might be true that I started seeing the errors related to the Queue component(s) when I added a second Queue. Just a hunch that having more than one of some type of component may be related to the issue.

[VirtuaBoza]

I only started getting these errors related to my Next component(s) when I added a second Next component, and it might be true that I started seeing the errors related to the Queue component(s) when I added a second Queue. Just a hunch that having more than one of some type of component may be related to the issue.

I just removed my second Next app and was able to deploy without issue. I added it back in, and got the failed to register new resource TopRevalidationSeederCode error once again.

[VirtuaBoza]

Here's the latest failure logs:
sst.log
event.log

[fwang]

@VirtuaBoza from the log above, as well as the error message in the OP, it seems files (generated lambda package) in the .sst/artifacts folder are missing.

I'm not able to recreate it on my end.

Can you try v0.0.241 and run SST_NO_ARTIFACT_CLEANUP=1 sst deploy. This will preserve the .sst/artifacts folder and we can see what's in there. When u run into the issue, can you send me ur:

• sst.log
• event.log
• sst.config.ts
• file listing in .sst/artifacts (ie. ls -lR .sst/artifacts)

[VirtuaBoza]

@fwang So.. I've watched that artifacts directory during a (failed) deploy before, and noticed that it was getting populated and then wiped out and then populated with more stuff. My suspicion is that the rmrf that happened during the deploy is causing the issue. Running deploy just now with SST_NO_ARTIFACT_CLEANUP=1 was successful, but here is what you asked for anyway:
sst.log
event.log
sst.config.ts.zip
artifacts.txt

[VirtuaBoza]

I'm on 0.0.256 now and seeing a similar error on sst dev that appears to hang the process:

×  Failed
   LMSRevalidationSeederCode aws:s3:BucketObjectv2
   open /home/runner/work/stacks-ils-ion/stacks-ils-ion/.sst/artifacts/LMSRevalidationSeeder/code.zip: no such file or directory

sst.log
event.log
artifacts.txt
sst.config.ts.zip

[fwang]

hmm.. that's weird.. the file .sst/artifacts/LMSRevalidationSeeder/code.zip shows there in the artifacts folder according to artifacts.txt.

it was getting populated and then wiped out and then populated with more stuff

@thdxr any idea how this could happen?