NoSuchKey during deployment #41

gs11 · 2020-01-08T15:46:39Z

Since version 1.10.1 deployments fail with a NoSuchKey error

S3 Sync: Synced metadata.
Serverless: Invoke aws:deploy:finalize
Serverless: [AWS s3 404 0.314s 0 retries] copyObject({
  CopySource: '<redacted>//index.html',
  Key: '/index.html',
  Bucket: '<redacted>',
  Metadata: { CacheControl: 'no-cache' },
  ACL: 'private',
  MetadataDirective: 'REPLACE'
})

  No Such Key --------------------------------------------

  NoSuchKey: The specified key does not exist.
      at Request.extractError (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/services/s3.js:816:35)
      at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
      at Request.transition (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
      at AcceptorStateMachine.runTo (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
      at /usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
      at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
      at Request.transition (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
      at AcceptorStateMachine.runTo (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
      at /usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
      at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
      at callNextListener (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
      at IncomingMessage.onEnd (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:307:13)
      at IncomingMessage.emit (events.js:215:7)
      at IncomingMessage.EventEmitter.emit (domain.js:476:20)
      at endReadableNT (_stream_readable.js:1183:12)
      at processTicksAndRejections (internal/process/task_queues.js:80:21)

The text was updated successfully, but these errors were encountered:

trilom · 2020-01-11T03:06:40Z

Looking at this the only thing I see weird is the key path, and I can't see 100% the copySource.

Here is a healthy request.

[AWS s3 200 0.5s 0 retries] copyObject({ CopySource: 'bucketName/test3.js',
  Key: 'test3.js',
  Bucket: 'bucketName',
  Metadata: { CacheControl: 'public, max-age=1, must-revalidate' },
  ACL: 'private',
  MetadataDirective: 'REPLACE' })

It should be that the CopySource is the path to the resource including the bucket name in S3. So if the S3 location is s3://bucketName/test.js then CopySource should be bucketName/test.js. The key is the path to the file after the bucketName and forward slash (/). In the same example if the S3 location is s3://bucketName/test.js then they Key should be test.js.

From what I see the key has a forward slash (/), the Key is generated for this with the following code ** Key: file.name.replace(localDir, ''),** where localDir is [servicePath, s.localDir].join('/') and file.name is the path to the local file. In the custom.s3Sync[n].localDir parameter, you should add a forward slash (/) to the localDir and this should function as you expect it to.

nibynool · 2020-01-12T11:30:09Z

That fixed the issue for me @trilom (the slash on the end of the localDir value).

Any chance a future version could auto-detect for the slash there as the error is unclear without your explanation.

gs11 · 2020-01-13T11:50:39Z

In my case localDir has no trailing slash but it can't make out sure what triggers the issue.

trilom · 2020-01-15T21:39:04Z

That fixed the issue for me @trilom (the slash on the end of the localDir value). ...

@nibynool yes, it’s in my radar at some point to resolve once I have free time.

_{Sent with GitHawk}

trilom · 2020-01-15T21:39:53Z

In my case localDir has no trailing slash but it can't make out sure what triggers the issue.

@gs11 could you provide more information here, the error and sls.yml will help, snippets of course.

_{Sent with GitHawk}

gs11 · 2020-01-17T15:27:51Z

Sure @trilom

serverless.yml


provider:
  name: aws
  runtime: nodejs10.x
  stage: ${opt:stage}
  region: eu-west-1

custom:
  hostname:
    dev: <redacted>
    prod: <redacted>
  s3Sync:
    - bucketName: ${self:service}-${self:provider.stage}
      localDir: build
      defaultContentType: text/html
      params:
        - "**/*.html":
            CacheControl: "no-cache"

plugins:
  - serverless-s3-sync

resources:
  Resources:
    StaticSite:
      Type: AWS::S3::Bucket
      Properties:
        AccessControl: PublicRead
        BucketName: ${self:service}-${self:provider.stage}
        WebsiteConfiguration:
          IndexDocument: index.html
          ErrorDocument: index.html
    StaticSiteS3BucketPolicy:
      Type: AWS::S3::BucketPolicy
      Properties:
        Bucket:
          Ref: StaticSite
        PolicyDocument:
          Statement:
            - Sid: PublicReadGetObject
              Effect: Allow
              Principal: "*"
              Action:
                - s3:GetObject
              Resource:
                Fn::Join: ["", ["arn:aws:s3:::", { "Ref": "StaticSite" }, "/*"]]
    WebAppCloudFrontDistribution:
      Type: AWS::CloudFront::Distribution
      Properties:
        DistributionConfig:
          Origins:
            - DomainName: ${self:service}-${self:provider.stage}.s3-${self:provider.region}.amazonaws.com
              Id: ${self:service}-${self:provider.stage}
              CustomOriginConfig:
                HTTPPort: 80
                HTTPSPort: 443
                OriginProtocolPolicy: https-only
          Enabled: "true"
          Aliases:
            - ${self:custom.hostname.${self:provider.stage}}
          DefaultRootObject: index.html
          CustomErrorResponses:
            - ErrorCode: 404
              ResponseCode: 200
              ResponsePagePath: /index.html
          DefaultCacheBehavior:
            AllowedMethods:
              - GET
              - HEAD
            TargetOriginId: ${self:service}-${self:provider.stage}
            ForwardedValues:
              QueryString: "false"
              Cookies:
                Forward: none
            ViewerProtocolPolicy: redirect-to-https
          ViewerCertificate:
            AcmCertificateArn: <redacted>
            SslSupportMethod: sni-only
          PriceClass: PriceClass_100

  Outputs:
    WebAppCloudFrontDistributionOutput:
      Value:
        "Fn::GetAtt": [WebAppCloudFrontDistribution, DomainName]

Error

S3 Sync: Synced metadata.
Serverless: Invoke aws:deploy:finalize
Serverless: [AWS s3 404 0.273s 0 retries] copyObject({
  CopySource: '<redacted>//index.html',
  Key: '/index.html',
  Bucket: '<redacted>',
  Metadata: { CacheControl: 'no-cache' },
  ACL: 'private',
  MetadataDirective: 'REPLACE'
})

  No Such Key --------------------------------------------

  NoSuchKey: The specified key does not exist.
      at Request.extractError (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/services/s3.js:816:35)
      at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
      at Request.transition (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
      at AcceptorStateMachine.runTo (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
      at /usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
      at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
      at Request.emit (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
      at Request.transition (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
      at AcceptorStateMachine.runTo (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
      at /usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
      at Request.<anonymous> (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
      at Request.callListeners (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
      at callNextListener (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
      at IncomingMessage.onEnd (/usr/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:307:13)
      at IncomingMessage.emit (events.js:215:7)
      at IncomingMessage.EventEmitter.emit (domain.js:476:20)
      at endReadableNT (_stream_readable.js:1183:12)
      at processTicksAndRejections (internal/process/task_queues.js:80:21)

robinvw1 · 2020-01-27T14:32:20Z

@gs11 can you check if it's fixed with v1.10.2 now? @nibynool his PR is merged by @k1LoW.

gs11 · 2020-01-29T10:22:27Z

Unfortunately it failed with the very same config/output as above @robinvw1

nibynool · 2020-01-29T10:29:11Z

@gs11 I'll test it out and see if I can replicate the issue. I might not get to it until the weekend though.

If you're on the Serverless Framework Slack group (https://serverless.com/slack) feel free to message me on there and we'll get it sorted for you.

nibynool · 2020-01-29T11:15:04Z

@gs11 can you please test by updating the package.json to reference:
"serverless-s3-sync": "github:nibynool/serverless-s3-sync#d9e5eb805e8e8bdaa515cba226f2dabe6de97587",. Post merge of the PR you will be able to change this back to using the official version.

I've tested my revised code with your config file but I dropped the Cloudfront Distribution for testing (this shouldn't impact the deployment process).

I will create the revised PR now, but would love your confirmation before @k1LoW merges it.

gs11 · 2020-01-29T12:06:05Z

Thanks for swift feedback! Tested successfully with that branch version 👍 @nibynool

nibynool · 2020-02-14T22:41:26Z

@gs11 can you update to the latest release version and then close this issue if you're happy with the solution. Thanks.

gs11 · 2020-02-15T10:08:00Z

Thanks!

k1LoW added the bug label Jan 9, 2020

gs11 changed the title ~~BUG - NoSuchKey during deployment~~ NoSuchKey during deployment Jan 9, 2020

nibynool mentioned this issue Jan 23, 2020

Allow optional trailing slash on the localDir configuration value #42

Merged

nibynool mentioned this issue Jan 29, 2020

Add directory separator when replacing localDir with bucket name #45

Merged

nibynool mentioned this issue Feb 14, 2020

Bucket not found #48

Closed

gs11 closed this as completed Feb 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NoSuchKey during deployment #41

NoSuchKey during deployment #41

gs11 commented Jan 8, 2020

trilom commented Jan 11, 2020

nibynool commented Jan 12, 2020

gs11 commented Jan 13, 2020

trilom commented Jan 15, 2020

trilom commented Jan 15, 2020

gs11 commented Jan 17, 2020

robinvw1 commented Jan 27, 2020

gs11 commented Jan 29, 2020

nibynool commented Jan 29, 2020 •

edited

Loading

nibynool commented Jan 29, 2020

gs11 commented Jan 29, 2020

nibynool commented Feb 14, 2020

gs11 commented Feb 15, 2020

NoSuchKey during deployment #41

NoSuchKey during deployment #41

Comments

gs11 commented Jan 8, 2020

trilom commented Jan 11, 2020

nibynool commented Jan 12, 2020

gs11 commented Jan 13, 2020

trilom commented Jan 15, 2020

trilom commented Jan 15, 2020

gs11 commented Jan 17, 2020

robinvw1 commented Jan 27, 2020

gs11 commented Jan 29, 2020

nibynool commented Jan 29, 2020 • edited Loading

nibynool commented Jan 29, 2020

gs11 commented Jan 29, 2020

nibynool commented Feb 14, 2020

gs11 commented Feb 15, 2020

nibynool commented Jan 29, 2020 •

edited

Loading