RequestTimeTooSkewed even though on version 3 and correctClockSkew is true by default

[kgoedecke]

Describe the bug

We are using "@aws-sdk/client-s3": "^3.9.0" and in our production system lately we are getting a lot of RequestTimeTooSkewed errors. As per the docs correctClockSkew is true by default and deprecated in v3. So why are we still receiving this error?

Docs: https://github.com/aws/aws-sdk-js-v3/blob/eae65cded5e703306346bdbd1b5de9d23050054a/UPGRADING.md

Your environment

SDK version number

version "3.9.0"

Details of the browser/Node.js/ReactNative version

Node v14.16.0 on Ubuntu

Steps to reproduce

See: https://github.com/kgoedecke/aws-sdk-v3-timeskew-bug

Observed behavior

S3 errors are being thrown by Node

Expected behavior

No errors should be thrown and clock should be corrected automatically.

[ajredniwja]

Hi there @kgoedecke, thanks for opening the issue, can you please share the code that you use?

[kgoedecke]

Hi @ajredniwja thanks a lot for getting back to me.

  uploadToAWS(
    fileName: string,
    { data, mimetype }: UploadData,
  ): Promise<PutObjectOutput> {
    const dir = removeDot(path.extname(fileName));

    const s3Params = {
      Bucket: S3_BUCKET,
      Key: `${dir}/${fileName}`,
      ContentType: mimetype,
      ACL: 'public-read',
      Body: data,
    } as PutObjectRequest;

    return this.s3.putObject(s3Params);
  }

[amie-wilt]

hey @ajredniwja , I'm having the same issue. It's intermittent but happening about 50% of the time. I've seen reports of this error primarily with uploading but I'm only reading buckets. my code is as below:

EDIT: I should also note that the error seems to be increasing in frequency as I increase the number of calls. For example, I have a dashboard view and over time I've added new cards to the dashboard, each making their own call with their respective bucket params. It seems as the number of cards have grown, the number of errors has too. Maybe that's obvious/expected but figured I'd note it anyway.

@kgoedecke have you had any luck with anything?

[stagr]

I am seeing the same or a very similar problem and have been able to reproduce it 100% of the time by setting the clock an hour back. I hope that the below information can help accelerate this issue and give some idea on how to reproduce it and troubleshoot it.

I have also mitigated the reproduced problem by manually setting config.systemClockOffset to the corresponding offset.

After comparing the two runs I can verify that systemClockOffset is never updated in the error case (when the system clock is one hour off). In fact I edited the dist code of "aws-sdk-js-v3/packages/middleware-signing/src/middleware.ts" manually, adding some console.log printouts and i can confirm that any code after "const output = await next({..." below is not run in the case where AWS S3 responds with a 403 with "Error: RequestTimeTooSkewed: The difference between the request time and the current time is too large." It seems that an error is thrown and execution never continous as expected after the await statement. So in my case the code that is supposed to actualize the systemClockOffset is never run and the next retry will intent with the same clock offset.

export function awsAuthMiddleware<Input extends object, Output extends object>(
  options: AwsAuthResolvedConfig
): FinalizeRequestMiddleware<Input, Output> {
  return (next: FinalizeHandler<Input, Output>, context: HandlerExecutionContext): FinalizeHandler<Input, Output> =>
    async function (args: FinalizeHandlerArguments<Input>): Promise<FinalizeHandlerOutput<Output>> {
      if (!HttpRequest.isInstance(args.request)) return next(args);
      const signer = typeof options.signer === "function" ? await options.signer() : options.signer;
      const output = await next({
        ...args,
        request: await signer.sign(args.request, {
          signingDate: new Date(Date.now() + options.systemClockOffset),
          signingRegion: context["signing_region"],
          signingService: context["signing_service"],
        }),
      });
      const { headers } = output.response as any;
      const dateHeader = headers && (headers.date || headers.Date);
      if (dateHeader) {
        const serverTime = Date.parse(dateHeader);
        if (isClockSkewed(serverTime, options.systemClockOffset)) {
          options.systemClockOffset = serverTime - Date.now();
        }
      }

      return output;
    };
}

Below some more technical details:

The edited middleware.js dist version (es):

export function awsAuthMiddleware(options) {
    console.log("awsAuthMiddleware")
    return function (next, context) {
        console.log("awsAuthMiddleware, inner function")
        return function (args) {
            
             console.log("awsAuthMiddleware, inner function 2")
            return __awaiter(this, void 0, void 0, function () {
                
                console.log("awsAuthMiddleware, inner function 2 awaiter ")
                var signer, _a, output, _b, _c, headers, dateHeader, serverTime;
                var _d;
                return __generator(this, function (_e) {
                    console.log("awsAuthMiddleware, inner function 2 awaiter, generator ")
                    switch (_e.label) {
                        case 0:
                            console.log("awsAuthMiddleware, inner function 2 awaiter, generator, case 0")
                            if (!HttpRequest.isInstance(args.request))
                                return [2 /*return*/, next(args)];
                            if (!(typeof options.signer === "function")) return [3 /*break*/, 2];
                            return [4 /*yield*/, options.signer()];
                        case 1:
                            console.log("awsAuthMiddleware, inner function 2 awaiter, generator, case 1")
                            _a = _e.sent();
                            return [3 /*break*/, 3];
                        case 2:
                            console.log("awsAuthMiddleware, inner function 2 awaiter, generator, case 2")
                            _a = options.signer;
                            _e.label = 3;
                        case 3:
                            console.log("awsAuthMiddleware, inner function 2 awaiter, generator, case 3")
                            signer = _a;
                            _b = next;
                            _c = [__assign({}, args)];
                            _d = {};
                            return [4 /*yield*/, signer.sign(args.request, {
                                    signingDate: new Date(Date.now() + options.systemClockOffset),
                                    signingRegion: context["signing_region"],
                                    signingService: context["signing_service"],
                                })];
                        case 4: 
                            console.log("awsAuthMiddleware, inner function 2 awaiter, generator, case 4")
                            return [4 /*yield*/, _b.apply(void 0, [__assign.apply(void 0, _c.concat([(_d.request = _e.sent(), _d)]))])];
                        case 5:
                            console.log("awsAuthMiddleware, inner function 2 awaiter, generator, case 5")
                            output = _e.sent();
                            console.log("awsAuthMiddleware, inner function 2 awaiter, generator, case 5, output = _e.sent()")
                            headers = output.response.headers;
                            console.log("es headers: %o", headers);
                            dateHeader = headers && (headers.date || headers.Date);
                            console.log("es dateHeader: %s", dateHeader);
                            if (dateHeader) {
                                serverTime = Date.parse(dateHeader);
                                console.log("es serverTime: " + serverTime);
                                if (isClockSkewed(serverTime, options.systemClockOffset)) {
                                    console.log("es Clock is skewed, old value: " + options.systemClockOffset);
                                    options.systemClockOffset = serverTime - Date.now();
                                    console.log("es Clock corrected, new value: " + options.systemClockOffset);
                                }
                            }
                            return [2 /*return*/, output];
                    }
                });
            });
        };
    };
}

The result of running this with a 30 minute offset between server and client is that you can see three retries and all cases show the logs up until:
console.log("awsAuthMiddleware, inner function 2 awaiter, generator, case 5")
but not:
console.log("awsAuthMiddleware, inner function 2 awaiter, generator, case 5, output = _e.sent()")

instrument.js:109 awsAuthMiddleware
instrument.js:109 systemClockOffset: 0 (Printed from my code before PUT object is executed)
instrument.js:109 awsAuthMiddleware, inner function
instrument.js:109 awsAuthMiddleware, inner function 2
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 0
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 1
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 3
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 4
instrument.js:136 PUT https://s3.us-east-1.amazonaws.com....?x-id=PutObject 403 (Forbidden)
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 5
instrument.js:109 awsAuthMiddleware, inner function 2
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 0
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 1
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 3
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 4
instrument.js:136 PUT https://s3.us-east-1.amazonaws.com/...?x-id=PutObject 403 (Forbidden)
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 5
instrument.js:109 awsAuthMiddleware, inner function 2
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 0
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 1
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 3
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 4
instrument.js:136 PUT https://s3.us-east-1.amazonaws.com....?x-id=PutObject 403 (Forbidden)
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator 
instrument.js:109 awsAuthMiddleware, inner function 2 awaiter, generator, case 5
instrument.js:109 
{requestId: undefined, cfId: undefined, extendedRequestId: "EGwUvYGnPtWVlCBm3fooG2MBD7tddRsyfWch3ZPgaYvsfxUVlsIESJmg+udO48nOgZeHDa0CYKc="}
instrument.js:109 Error: RequestTimeTooSkewed: The difference between the request time and the current time is too large.
    at http://localhost:3000/static/js/bundle.js:37897:68
    at step (http://localhost:3000/static/js/bundle.js:362564:23)
    at Object.next (localhost:3000/static/js/bundle.js:362545:53)
    at fulfilled (http://localhost:3000/static/js/bundle.js:362535:58)
instrument.js:109 error.$metadata: 
{httpStatusCode: 403, requestId: undefined, extendedRequestId: "EGwUvYGnPtWVlCBm3fooG2MBD7tddRsyfWch3ZPgaYvsfxUVlsIESJmg+udO48nOgZeHDa0CYKc=", cfId: undefined, attempts: 3, …}
attempts: 3
cfId: undefined
extendedRequestId: "EGwUvYGnPtWVlCBm3fooG2MBD7tddRsyfWch3ZPgaYvsfxUVlsIESJmg+udO48nOgZeHDa0CYKc="
httpStatusCode: 403
requestId: undefined
totalRetryDelay: 249
__proto__: Object
instrument.js:109 systemClockOffset 3: 0 (Printed from my code after PUT object is executed and in this case failed, after catch)

I am running this on:

OS: Windows 10
Browser: Chrome 90.0.4430.212
And the following dependencies:
"@aws-sdk/client-cognito-identity": "3.17.0",
"@aws-sdk/client-s3": "3.17.0",
"@aws-sdk/credential-provider-cognito-identity": "3.17.0",
"@aws-sdk/lib-storage": "3.17.0",

[stagr]

Similar issues in AWS SDK JS (v2):

aws/aws-sdk-js#3676
aws/aws-sdk-js#3718

Note that this i a big problem when running the SDK in browser since we do not have control of the end users clock. Between servers that we have control we can force NTP Sync but for users out in the wild with any setup of browser, os and device it is impossible.

Does anyone have a workaround for this situation? I know that I can set the systemClockOffset manually but I haven't been able to figure out how to hook me up in the middleware stack to catch this specific error and correct the offset on a per end user basis.

In v2 there is a workaround (aws/aws-sdk-js#399 (comment)) but I don't think this is possible in v3 right?

AWS.events.on('retry', function(response) {  
      if (response.error.name === 'RequestTimeTooSkewed') {
        console.error('User time is not correct. Handling error!');
        console.log('AWS systemClockOffset:', AWS.config.systemClockOffset);
        var serverTime = Date.parse(response.httpResponse.headers['date']);
        var timeNow = new Date().getTime();
        console.log('AWS timestamp:', new Date(serverTime));
        console.log('Browser timestamp:', new Date(timeNow));
        AWS.config.systemClockOffset = Math.abs(timeNow - serverTime);
        response.error.retryable = true;
        console.log('Setting systemClockOffset to:', AWS.config.systemClockOffset);
        console.log('Retrying uploading to S3 once more...');
      }
});

[kgoedecke]

@amie-wilt we still have the issue. Weirdly after migrating to a server on AWS that sits in the US region the issue is not there anymore. The issue seems to be related to the daylight savings time. But regardless the SDK should work imo. Do you guys have a server in a country with Daylight savings?

[stagr]

I now have a working workaround for our problem.

It has the below drawbacks but serves as proof of concept:

• The correction of systemClockOffset is made after all "normal" retries (3 in the default case) have been performed
• The correction of systemClockOffset and the explicit retry is made outside of the middleware stack
• The correction have to be applied each time a S3 command is used

private _uploadToS3 = async (arrayBuffer: ArrayBuffer, key: string, contentType: string): Promise<void> => {
    const uploadParams = {
      Bucket: this.bucket,
      Key: key,
      Body: new Blob([arrayBuffer]),
      ACL: 'private',
      ContentType: contentType
    };
    try {
      await this.s3Client.send(new PutObjectCommand(uploadParams));
    } catch (error) {
      if (error.name === "RequestTimeTooSkewed") {
        console.log("Serialized error: " + JSON.stringify(error, null, 2));      
        console.log("Current systemClockOffset: " + this.s3Client.config.systemClockOffset);
        const now = Date.now();
        const serverTime = Date.parse(error.ServerTime);
        const newOffset = (serverTime - now);
        console.log("Servertime: " + error.ServerTime + ", " + serverTime);
        console.log("Now: " + now);
        console.log("newOffset: ServerTime - Date.now(): " + newOffset);
        this.s3Client.config.systemClockOffset = newOffset;
        console.log("Manual retry after setting systemClockOffset to " + newOffset);
        try {
          await this.s3Client.send(new PutObjectCommand(uploadParams));
        } catch (error) {
          console.log("Manual retry also failed, hopefully for another reason.");
          throw error;
        }
      } else {  
        //Handle other errors as usual
        throw error;
      }
    }

Suggested implementation in SDK (DISCLAIMER: not tested and my first look on the SDK code):

• Detect the relevant errors (at least RequestTimeTooSkewed, maybe all isClockSkewError(error) is better) in StandardRetryStrategy.ts
• Let the delayDecider.ts set the delay to 0 in this case or force manually the delay to 0 in StandardRetryStrategy to avoid unnecessary delay
• Pass in the options/config object to StandardRetryStrategy.ts from retryMiddleware.ts so that it can manipulate systemClockOffset
• Set the systemClockOffset on the options/config object to the difference between error.ServerTime and Date.now()
• Let the normal retry flow continue as normal

Let me know your opinions on the workaround and the suggested solution.

[kgoedecke]

@stagr I will try this and let you know in the next few days. I can't reliably reproduce the error it happens every now and then, super strange...

[amie-wilt]

I am instantiating my s3 client inside of my functions that use it. That doesn't seem like the right way but it is working. I saw an error somewhere that indicated the call wasn't authenticated (when it is, as least as far as my code is concerned), it seems like the amount of time (which isn't much) between instantiation and the function call causes this?

@kgoedecke this was happening for me before daylight savings but I appreciate the suggestion!

[stagr]

@trivikr I see that you are assigned to this issue. Can you have a look at my analysis and suggested solution?

@kgoedecke ok, great. Let me know if it solves your problem also.

@amie-wilt I am not sure I understand. Are you saying that intanciating the S3Client directly before sending the GetObjectCommand solved your issue? It sounds very strange and is not inline with my analysis but maybe there are more than one bug here.

[amie-wilt]

@stagr Yes, that's what I'm saying. Pardon the awful markup:

[ajredniwja]

Hey @stagr, I have reached out to the team internally to discuss this, and have an update soon. Meanwhile I am gonna try the workarounds mentioned and see if they work for me. I was able to reproduce the issue by changing the systems time to more than 15 minutes and running simple code like:

(async () => {

    const { S3Client, PutObjectCommand } = require("@aws-sdk/client-s3");
    // ...
    
    var params = {Bucket: 'aj', Key: 'test', Body: "jdj"};
    const client = new S3Client({region: "us-west-2"});
    const command = new PutObjectCommand(params);

    try {
        const response = await client.send(command);
        console.log(response);
    } catch (err) {
        console.log(err);
    }
    
})();

[stagr]

@ajredniwja ok, great! Let me know if you get the workarounds to work and what the team says.

FYI: We have the workaround that i presented above running in production since yesterday and we have not seen the error since. Ususally it occurs between 1 and 50 times per day. Letting it run over the weekend I believe will be enough proof that it is working in production also.

[stagr]

Workaround confirmed as working after 4 days without error occuring.

@ajredniwja @kgoedecke any updates from your side?

[ajredniwja]

I am glad to hear that, wasn't able to try those yet as I wanted to discuss it further with the team. Will have someone else from the team look at it as well.

[kgoedecke]

@stagr not yet, we haven't pushed it into production yet, frankly we are a bit scared to do so haha

[stagr]

Sending a ping on this issue...

@ajredniwja @kgoedecke any updates from your side?

[amie-wilt]

I realize my fix doesn’t seem in line with what’s going on for you guys however the fix has been in prod for weeks now and I haven’t seen the error since. Prior to that it was happening daily. Just figured I’d update!

…

On Fri, Jun 25, 2021 at 06:59 Staff ***@***.***> wrote: Sending a ping on this issue... @ajredniwja <https://github.com/ajredniwja> @kgoedecke <https://github.com/kgoedecke> any updates from your side? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2208 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB6YLAGES7YIM32AUI56FGTTUROSHANCNFSM42LFBACA> .

[kgoedecke]

We temporarily move to a new server on EC2 and that seems to have fixed the issue. Sadly we currently dont have the capacity to try out the hotfix.

[stagr]

Sending a ping on this issue...

@ajredniwja any updates from your side?

[ajredniwja]

@stagr will have an update for you soon. It was not a priority because of workaround available.

[stagr]

Ok, great. Just want to make sure it gets fixed since it is a quite serious issue.

[leematos]

I'm seeing this problem while attempting to replace v2 with v3 in a react-native iOS app. at the moment, I'm seeing strange skews and I know that system time is accurate. If I set this.s3Client.config.systemClockOffset = 0; effectively nullifying the auto skew logic, I can get successful requests, but at that means if a user's clock is actually skewed they will hit the skew error. :(

Stagr's logic here is solid #2208 (comment) and I'm gonna keep exploring it but just adding a +1 to this issue.

[kgoedecke]

@trivikr I saw you wrote some of this code around it. Do you have any idea why this is happening in the first place?

[kgoedecke]

@trivikr ping 🏓 or maybe from you @ajredniwja

[kgoedecke]

Here's instructions how to reproduce this bug: https://github.com/kgoedecke/aws-sdk-v3-timeskew-bug

One more update: It makes sense as per the implementation that this bug occurs:

aws-sdk-js-v3/packages/middleware-signing/src/middleware.ts

Line 26 in 0a251aa

const output = await next({

The skew is only corrected if the response from the amazon servers is there, but it won't even be parsed in the middleware since the await next is never executed. Because of the Skew itself this will never happen. The way back through the middleware will never be triggered.

aws-sdk-js-v3/packages/middleware-signing/src/middleware.ts

Line 39 in 0a251aa

if (isClockSkewed(serverTime, options.systemClockOffset)) {

will never be triggered.

[kgoedecke]

I've opened a first PR about this: #2664

[kgoedecke]

Btw here's how to fix it manually by hooking into the SDK middleware, also showcased here: kgoedecke/aws-sdk-v3-timeskew-bug#1

const s3 = new S3({
  region: process.env.AWS_REGION,
  maxAttempts: 3,
  systemClockOffset: 0,
  retryStrategy: new StandardRetryStrategy(() => Promise.resolve(5)),
});
const putObject = new PutObjectCommand(s3Params);

const middleware = (next, _context) => async (args) => {
    const result = await next(args).catch((error) => {
    if (error.Code === "RequestTimeTooSkewed") {
      const serverTime = Date.parse(error.ServerTime);
      const newOffset = (serverTime - Date.now());
      s3.config.systemClockOffset = newOffset;
    }
    throw error;
  });
  return result;
}
const options = { step: "finalizeRequest", tags: ["CORRECT_TIME_SKEW"] };

putObject.middlewareStack.add(middleware, options);

s3.send(putObject);

[trivikr]

PR which originally enabled ClockSkew correction by default in v3: #459
Current status: CI is failing for solution proposed in #2664

[trivikr]

The error is reproducible with any operation when the time skewed is more than 15 minutes.

Code

import { S3 } from "@aws-sdk/client-s3";

const region = "us-east-1";
const client = new S3({ region });
await client.listBuckets({});

Output with System time difference <15 mins

# No output means success

Output with System time difference >15 mins

/Users/trivikr/workspace/aws-sdk-v3-timeskew-bug/node_modules/@aws-sdk/client-s3/dist/cjs/protocols/Aws_restXml.js:7060
    return Promise.reject(Object.assign(new Error(message), response));
                                        ^

RequestTimeTooSkewed: The difference between the request time and the current time is too large.
    at deserializeAws_restXmlListBucketsCommandError (/Users/trivikr/workspace/aws-sdk-v3-timeskew-bug/node_modules/@aws-sdk/client-s3/dist/cjs/protocols/Aws_restXml.js:7060:41)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async /Users/trivikr/workspace/aws-sdk-v3-timeskew-bug/node_modules/@aws-sdk/middleware-serde/dist/cjs/deserializerMiddleware.js:6:20
    at async /Users/trivikr/workspace/aws-sdk-v3-timeskew-bug/node_modules/@aws-sdk/middleware-signing/dist/cjs/middleware.js:12:24
    at async StandardRetryStrategy.retry (/Users/trivikr/workspace/aws-sdk-v3-timeskew-bug/node_modules/@aws-sdk/middleware-retry/dist/cjs/StandardRetryStrategy.js:51:46)
    at async /Users/trivikr/workspace/aws-sdk-v3-timeskew-bug/node_modules/@aws-sdk/middleware-logger/dist/cjs/loggerMiddleware.js:6:22
    at async file:///Users/trivikr/workspace/aws-sdk-v3-timeskew-bug/test.mjs:5:1 {
  Code: 'RequestTimeTooSkewed',
  RequestTime: '20210817T124326Z',
  ServerTime: '2021-08-17T12:58:29Z',
  MaxAllowedSkewMilliseconds: '900000',
  RequestId: 'F92XPB5HYR783C5F',
  HostId: 'qECx2KXQEP2spTq5c4gOzCT14IAl8zrOo159uJ6dw6EETemfisw5BdTAmQdhOWdowzTbjlLKzok=',
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 403,
    requestId: undefined,
    extendedRequestId: 'qECx2KXQEP2spTq5c4gOzCT14IAl8zrOo159uJ6dw6EETemfisw5BdTAmQdhOWdowzTbjlLKzok=',
    cfId: undefined,
    attempts: 3,
    totalRetryDelay: 337
  }
}

[trivikr]

Behavior in other AWS SDKs:

• Mobile SDKs added automatic clock skew correction in 2013 (version 1.6.1 for Android)
• .NET SDK has been correcting clockskew since 2015
• AWS SDK for C++ added clockskew adjustment logic in 2016
• AWS SDK for Java (v2) added clockskew adjustment logic in 2019

[trivikr]

Update: The unit tests in existing middleware-signing were difficult to update, and the code was not modular. So, an PR which refactors middleware-signing was posted at #2681

Once #2681 is merged, I'll post PR trivikr#169 upstream which fixes this issue.

[kgoedecke]

@trivikr thanks for that. Looks good so far.

I also noticed that it was super difficult to fix the tests.

[trivikr]

Once #2681 is merged, I'll post PR trivikr#169 upstream which fixes this issue.

The fix for this issue is posted upstream at #2686

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.