Unable to upload artifact to Notary API when behind Corporate Proxy

[austinsasko]

When attempting to upload an artifact to the notary API behind a corporate proxy, depending on the environment, one of following errors are encountered:
Error: HTTP error: error sending request for url (https://appstoreconnect.apple.com/notary/v2/submissions): error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer

Error: error sending request for url (https://appstoreconnect.apple.com/notary/v2/submissions): error trying to connect: unsuccessful tunnel

Is there a flag or override in which the API call can leverage provided proxies / provided CA chain?

Note: I am able to access that API via POST directly from cURL or Postman on the same machine

[roblabla]

For the first error, we use rustls for the TLS, using webpki as the trust root.

I suppose we should change to use the native root trusts instead? Assuming the corporate proxy is one of those ass-backwards proxy that sets a catchall tls cert.

[austinsasko]

@roblabla That would be ideal. I think many enterprise corporate proxies will run with some level of TLS interception.

[roblabla]

@austinsasko can you try the binaries from https://github.com/indygreg/apple-platform-rs/actions/runs/5869121441 ?

[austinsasko]

That did resolve the certificate error. Thank you

[austinsasko]

Although, unfortunately I now encounter a separate issue, an error with the S3 upload stage:

credentials cache miss occurred; added new AWS credentials (took 9.666µs)
Error: s3 upload error: unhandled error

[roblabla]

Right, s3 is going through smithy, which is using a different version of rustls, and so didn't accept native TLS certs. I updated smithy so its rustls version matches, and started a new build @ https://github.com/indygreg/apple-platform-rs/actions/runs/5872829057 .

[austinsasko]

Made it past the command erroring out immediately -- it seems to be getting better and better with each build you provide, thank you.

try_op;
try_attempt;
lazy_load_credentials;
credentials cache miss occurred; added new AWS credentials (took Ok(11µs))
finally_attempt;
try_attempt;
finally_attempt;
try_attempt;
finally_attempt;
finally_op;
Error: s3 upload error: unhandled error

[austinsasko]

Is this due to awslabs/aws-sdk-rust#169?

[roblabla]

Can you try running with -vvv to get (extremely verbose) output, and provide me with the logs?

[roblabla]

Is this due to awslabs/aws-sdk-rust#169?

I don't think so. This seems to be about support for explicit proxies (ergo proxies that you connect to using one of the many proxy protocols, like HTTP CONNECT, SOCKS5 or whatever). In your case, the proxying is supposedly done by a middlebox in your connection redirecting the traffic to the proxy - without any kind of configuration on your part.

[austinsasko]

With -vvv (minor redactions)

[2023-08-17T17:15:46Z DEBUG aws_smithy_runtime::client::orchestrator] beginning attempt #2
[2023-08-17T17:15:46Z INFO  tracing::span] try_attempt;
[2023-08-17T17:15:46Z TRACE aws_smithy_runtime::client::orchestrator::endpoints] orchestrating endpoint resolution
[2023-08-17T17:15:46Z DEBUG aws_smithy_runtime::client::orchestrator::endpoints] will use endpoint Endpoint { url: "https://notary-submissions-prod.s3.us-west-2.amazonaws.com", headers: {}, properties: {"authSchemes": Array([Object({"name": String("sigv4"), "disableDoubleEncoding": Bool(true), "signingRegion": String("us-west-2"), "signingName": String("s3")})])} }
[2023-08-17T17:15:46Z TRACE aws_smithy_runtime::client::interceptors] running `modify_before_signing` interceptors
[2023-08-17T17:15:46Z TRACE aws_smithy_runtime::client::orchestrator::auth] orchestrating auth auth_scheme_option_resolver_params=AuthSchemeOptionResolverParams(TypeErasedBox[!Clone]:StaticAuthSchemeOptionResolverParams) auth_scheme_options=[AuthSchemeId { scheme_id: "sigv4" }]
[2023-08-17T17:15:46Z TRACE aws_smithy_runtime::client::orchestrator::auth] resolved auth scheme, identity resolver, and signing implementation auth_scheme=SharedAuthScheme(SigV4AuthScheme { signer: SigV4Signer }) identity_resolver=SharedIdentityResolver(CredentialsIdentityResolver { credentials_cache: SharedCredentialsCache(LazyCredentialsCache { time: SharedTimeSource(SystemTimeSource), sleeper: SharedAsyncSleep(TokioSleep), cache: ExpiringCache { buffer_time: 10s, value: RwLock { data: OnceCell { value: Some((Credentials { provider_name: "apple-codesign", access_key_id: "x", secret_access_key: "** redacted **" }, SystemTime { tv_sec: x, tv_nsec: x })) } }, _phantom: PhantomData<aws_credential_types::provider::error::CredentialsError> }, provider: SharedCredentialsProvider(Credentials { provider_name: "apple-codesign", access_key_id: "x", secret_access_key: "** redacted **" }), load_timeout: 5s, buffer_time: 10s, buffer_time_jitter_fraction: 0x104679f30, default_credential_expiration: 900s }) }) signer=SigV4Signer
[2023-08-17T17:15:46Z TRACE aws_smithy_runtime::client::orchestrator::auth] extracted auth scheme endpoint config auth_scheme_endpoint_config=AuthSchemeEndpointConfig(Some(Object({"name": String("sigv4"), "disableDoubleEncoding": Bool(true), "signingRegion": String("us-west-2"), "signingName": String("s3")})))
[2023-08-17T17:15:46Z DEBUG aws_credential_types::cache::lazy_caching] loaded credentials from cache
[2023-08-17T17:15:46Z TRACE aws_smithy_runtime::client::orchestrator::auth] resolved identity identity=Identity { data: Credentials { provider_name: "apple-codesign", access_key_id: "x", secret_access_key: "** redacted **" }, expiration: None }
[2023-08-17T17:15:46Z TRACE aws_smithy_runtime::client::orchestrator::auth] signing request
[2023-08-17T17:15:46Z TRACE aws_sigv4::http_request::sign] signing request request=SignableRequest { method: PUT, uri: https://notary-submissions-prod.s3.us-west-2.amazonaws.com/prod/x%x-7408-4997-ab30-2fef189b9013?x-id=PutObject, headers: {"content-type": "application/octet-stream", "content-length": "12729381", "user-agent": "aws-sdk-rust/0.56.0 os/macos lang/rust/1.71.1", "x-amz-user-agent": "aws-sdk-rust/0.56.0 api/s3/0.29.0 os/macos lang/rust/1.71.1"}, body: UnsignedPayload } params=SigningParams { access_key: "** redacted **", secret_key: "** redacted **", security_token: "** redacted **", region: "us-west-2", service_name: "s3", time: SystemTime { tv_sec: x, tv_nsec: x }, settings: SigningSettings { percent_encoding_mode: Single, payload_checksum_kind: XAmzSha256, signature_location: Headers, expires_in: None, excluded_headers: Some(["authorization", "user-agent", "x-amzn-trace-id"]), uri_path_normalization_mode: Disabled, session_token_mode: Include } }
[2023-08-17T17:15:46Z TRACE aws_sigv4::http_request::sign] canonical_request=PUT
    /prod/x%x-7408-4997-ab30-2fef189b9013
    x-id=PutObject
    content-length:12729381
    content-type:application/octet-stream
    host:notary-submissions-prod.s3.us-west-2.amazonaws.com
    x-amz-content-sha256:UNSIGNED-PAYLOAD
    x-amz-date:20230817T171546Z
    x-amz-security-token:xxxxxxx    x-amz-user-agent:aws-sdk-rust/0.56.0 api/s3/0.29.0 os/macos lang/rust/1.71.1
    
    content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent
    UNSIGNED-PAYLOAD
[2023-08-17T17:15:46Z TRACE aws_smithy_runtime::client::interceptors] running `modify_before_transmit` interceptors
[2023-08-17T17:15:46Z DEBUG aws_smithy_runtime_api::client::interceptors::context] entering 'transmit' phase
[2023-08-17T17:15:46Z TRACE aws_smithy_runtime::client::orchestrator] transmitting request request=Request { method: PUT, uri: https://notary-submissions-prod.s3.us-west-2.amazonaws.com/prod/x%x-7408-4997-ab30-2fef189b9013?x-id=PutObject, version: HTTP/1.1, headers: {"content-type": "application/octet-stream", "content-length": "12729381", "user-agent": "aws-sdk-rust/0.56.0 os/macos lang/rust/1.71.1", "x-amz-user-agent": "aws-sdk-rust/0.56.0 api/s3/0.29.0 os/macos lang/rust/1.71.1", "x-amz-date": "20230817T171546Z", "authorization": Sensitive, "x-amz-content-sha256": "UNSIGNED-PAYLOAD", "x-amz-security-token": Sensitive, "amz-sdk-request": "attempt=2; max=3", "amz-sdk-invocation-id": "c2169e32-x-44b8-82ab-9def91810b40"}, body: SdkBody { inner: BoxBody, retryable: true } }
[2023-08-17T17:15:46Z TRACE hyper::client::pool] checkout waiting for idle connection: ("https", notary-submissions-prod.s3.us-west-2.amazonaws.com)
[2023-08-17T17:15:46Z TRACE hyper::client::connect::http] Http::connect; scheme=Some("https"), host=Some("notary-submissions-prod.s3.us-west-2.amazonaws.com"), port=None
[2023-08-17T17:15:46Z TRACE tracing::span::active] <- invoke;
[2023-08-17T17:15:46Z DEBUG hyper::client::connect::dns] resolving host="notary-submissions-prod.s3.us-west-2.amazonaws.com"
[2023-08-17T17:15:46Z TRACE tracing::span::active] -> invoke;
[2023-08-17T17:15:46Z DEBUG hyper::client::connect::http] connecting to 52.218.235.89:443
[2023-08-17T17:15:46Z TRACE tracing::span::active] <- invoke;
[2023-08-17T17:15:49Z TRACE tracing::span::active] -> invoke;
[2023-08-17T17:15:49Z TRACE hyper::client::pool] checkout dropped for ("https", notary-submissions-prod.s3.us-west-2.amazonaws.com)
[2023-08-17T17:15:49Z DEBUG aws_smithy_runtime::client::orchestrator] encountered orchestrator error; halting
[2023-08-17T17:15:49Z TRACE aws_smithy_runtime_api::client::interceptors::context] orchestrator is transitioning to the 'failure' phase from the 'Transmit' phase
[2023-08-17T17:15:49Z TRACE tracing::span] -- try_attempt;
[2023-08-17T17:15:49Z INFO  tracing::span] finally_attempt;
[2023-08-17T17:15:49Z TRACE aws_smithy_runtime::client::interceptors] running `modify_before_attempt_completion` interceptors
[2023-08-17T17:15:49Z TRACE aws_smithy_runtime::client::interceptors] running `read_after_attempt` interceptors
[2023-08-17T17:15:49Z TRACE tracing::span] -- finally_attempt;
[2023-08-17T17:15:49Z TRACE aws_smithy_runtime_api::client::retries] "Retryable Smithy Errors" classifier classified error as Error(TransientError)
[2023-08-17T17:15:49Z DEBUG aws_smithy_runtime::client::retries::strategy::standard] attempt #2 failed with Error(TransientError); retrying after 784.517938ms
[2023-08-17T17:15:49Z TRACE aws_smithy_runtime::client::orchestrator] checking if context can be rewound for attempt #3
[2023-08-17T17:15:49Z DEBUG aws_smithy_runtime::client::orchestrator] delaying for 784.517938ms
[2023-08-17T17:15:49Z TRACE tracing::span::active] <- invoke;
[2023-08-17T17:15:50Z TRACE tracing::span::active] -> invoke;
[2023-08-17T17:15:50Z TRACE aws_smithy_runtime::client::orchestrator] attempt_timeout_config=MaybeTimeoutConfig { sleep_impl: Some(SharedAsyncSleep(TokioSleep)), timeout: None, timeout_kind: OperationAttempt }
[2023-08-17T17:15:50Z DEBUG aws_smithy_runtime::client::orchestrator] beginning attempt #3
[2023-08-17T17:15:50Z INFO  tracing::span] try_attempt;

[austinsasko]

@roblabla Any idea with the above log?

Thanks :)

[roblabla]

Well, the problem is in the S3 upload, but the logs aren't giving me enough information to figure out what's going on, unfortunately (smithy isn't being very verbose).

However, if the proxy is somehow tampering with the request, there's a chance that the failure is coming from the signature failing to validate, in which case there isn't much we can do.

[austinsasko]

@roblabla We were able to produce a version that successfully handles a proxy environment. See #88

Is it possible to get an official build off a branch with that code in the interim of requirements being met to merge?

Thanks :)

[indygreg]

The main branch should use the machine's trusted certificate store now. I think the only remaining piece here is the HTTP proxy detection for the S3 upload bits. That's #88.