-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HeadObjectError in S3: no providers in chain provided credentials #428
Comments
@allada @aaronmondal |
Succeeds at 56eda36
|
@mhz5 Is this still an issue after #423 ? @allada added some an additional config options in #421 for the endpoint url, though I'm not sure whether that's actually related. Seems like it can't find the aws credentials. Note to self: It seems like we can now also add support for SSO: awslabs/aws-sdk-rust#703. |
Yeah, can you tell us what checkout you are using? It may have been fixed in recent changes. We have not had time to deep-dive the AWS config again to verify everything, because we are in the process of releasing a GCP variation and a pinned version release. We will be making more changes to AWS config in the coming months to support rule-based auto-scaling policies, so be on the lookout for those too. |
This regression occurred with this PR: |
Hmm I just played around with things and can't reproduce this issue. I can imagine two things going wrong here: The new s3 implementation enforces HTTPS by default and now requires explicit setting of the
In this case the fix would be to set that setting to Otherwise it might just be a difference in the way you pass the credentials to the store. The aws SDK uses a different config detection mechanism as the previous implementation. In my (working) case, I have credentials at I assumed that https://github.com/TraceMachina/native-link/blob/3ec203b9c17e8e4dfa7160f74e948c64e542de16/native-link-store/src/s3_store.rs#L152C11-L152C11 correctly forwards environment variables to the S3 store if they're set. If this is not the case we might need to make the detection of the credential environment variables explicit. |
Credentials can be used from a variety of sources. On AWS it is likely coming from the service account associated attached to the instance, which comes from "169.254.169.254" (iirc). It is possible that it is unable to resolve these credentials and use them. Doing what @aaronmondal said and setting them in It is possible that the new S3 SDK does not support credentials provider (I personally have not tested yet). If it is the case, we can find a work-around. |
I deployed native-link on AWS via |
Sorry for the delay on this. We are trying to get a release pinned and have some priority thrashing. This is a high priority to resolve and we'll double check everything soon. |
Is there a workaround for this? I'm also facing the same issue |
Yep, we identified that the default credentials provider does not support this and the sdk requires this be set manually. Fix is in-flight, but waiting on a regression test. |
Credential provider is not supplied when creating aws_config::from_env(), that leads to failure responses seen in #428. Pass the aws_config::default_provider::credentials.default_provider into the aws_config::from_env() builder which should pick up the proper credentials for the environment based on the resolution order.
Should be fixed by #494. This change is published in the docker pull ghcr.io/tracemachina/nativelink:v0.2.0 Please let us know if things still don't work. |
It might be worth having CI perform a dev native-link deployment in AWS, and point some builds at it. (nightly?)
To prevent regressions in the AWS deployment.
Deployed native-link on AWS (instructions) at 4cc53bc and attempted to point a build at the deployment. Encountered following error:
INFO: Invocation ID: d4859976-5154-434c-91f9-3537d8ef7d40
INFO: Analyzed target //:dummy_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
ERROR: /home/ubuntu/native-link/BUILD.bazel:42:8: Executing genrule //:dummy_test_sh failed: (Exit 34): UNAVAILABLE: Unhandled HeadObjectError in S3: Unhandled { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: "no providers in chain provided credentials" }), connection: Unknown } }), meta: ErrorMetadata { code: None, message: None, extras: None } }, retries: 7 : Failed to run has() on slow store : Inner store get in compression store failed : Compression underlying store get failed : --- : Received erroneous partial chunk: Error { code: Internal, messages: ["Writer was dropped before EOF was sent"] } : During first buf_channel::take() : Failed to read header in get_part compression store : Failed to get_part in get_part_unchunked : --- : Received erroneous partial chunk: Error { code: Internal, messages: ["Writer was dropped before EOF was sent"] } : Failed to recv first chunk in collect_all_with_size_hint : Failed to read stream to completion in get_part_unchunked
INFO: Elapsed time: 4.626s, Critical Path: 3.88s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
//:dummy_test FAILED TO BUILD
Executed 0 out of 1 test: 1 fails to build.
The text was updated successfully, but these errors were encountered: