-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AWS-SDK-SSM Compilation error during boot time #2826
Comments
This is very weird. The change in 1.148.0 of aws-sdk-ssm was only switching the I'm fairly certain there is something bootsnap related going on - my guess would be in the byte code caching, but I'm not an expert in bootsnap. Have you cleared your bootsnap cache(s)? |
We did cleared the cache, we took a corrupted image, as soon as we touch the file the cache is regenerate and the issue is fixed so its a bit tricky to pry inside and dig. The main reason I'm bringing this here and not bootsnap is because:
I'm a bit confused as how to solve this so I would take any bit. I don't want to remove the bootsnap integration as its providing a good performance boot to us so far. |
Yeah that makes sense - I know the SDK has had previous issues reveled by bootsnap. Did you upgrade any other gems at the same time? Could you try pinning aws-sdk-ssm to 1.147.0 and see if it fixes the issue? What other SDK gems/versions do you have? I'm also a bit uncertain how to debug this - I haven't been able to reproduce locally yet and I dont see any potential/obvious issues in the code itself that might cause this behavior. You mentioned it was occurring in staging/production - have you been able to reproducing locally/development? Can you share any information on how you're deploying? Do you have a fixed Gemfile.lock? |
This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing. |
We see the same issue in GitHub Actions when we are running CI. It started when we added the aws-sdk-ec2 gem a few days ago. It didn't happen with the aws-sdk-s3 gem which we have used for a long time.
Sometimes a different error occurs:
We haven't seen it in production or during local development. |
@yenshirak - Sorry you're running into this as well - I still haven't been able to reproduce or debug the issue. Does it happen on every run or only sometimes? Anything else you can tell me about the environment? Can you try pinning aws-sdk-ec2 to version 1.357.0 or earlier? |
@alextwoods Only sometimes. Deleting the GitHub Actions bundler cache is fixing it. I will try to downgrade to 1.357.0. |
@alextwoods Unfortunately downgrading to 1.357.0 doesn't help:
I didn't mention earlier that the error is thrown during the DB creation: |
@alextwoods This is the AWS section of our current Gemfile.lock:
and
|
@alextwoods The issue disappeared after we replaced the |
@alextwoods Unfortunately it didn't disappear fully, but it occurs less frequently and the error is different:
|
Have you tried opening an issue on the bootsnap repo for this? |
Until there's a solid reproduction case, I'm going to close this. I think the issue lies in bootsnap, but I agree it was surfaced by a change in the SDK. |
|
@mullermp No, but I will open a Bootsnap issue then. For now we disabled Bootsnap in CI as a workaround. |
Describe the bug
We are randomly getting this error while running our app in staging/production.
Note that we use
bootsnap
fromShopify
- https://github.com/Shopify/bootsnap tocompile_cache_iseq
&compile_cache_yaml
This started when we upgraded
aws-sdk-ssm
from1.147.0
to1.148.0
2 weeks ago.Stacktrace from our log
Expected Behavior
The app does not crash on boot (rails s/rails c/...)
Current Behavior
The app randomly crash because of wrong reference to
StringShape
instead ofStructureShape
Reproduction Steps
Unable to reproduce consistently.
Possible Solution
We've been trying to pin point the root cause issue but the diff from 421613e is quite tough to navigate.
Additional Information/Context
No response
Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version
aws-sdk-ssm
Environment details (Version of Ruby, OS environment)
Base image "ruby:3.2.0-slim"
The text was updated successfully, but these errors were encountered: