Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'file already exists and is different' when publishing to S3 with KMS encryption #1117

Closed
joshdcox opened this issue Sep 27, 2022 · 0 comments · Fixed by #1167
Closed

'file already exists and is different' when publishing to S3 with KMS encryption #1117

joshdcox opened this issue Sep 27, 2022 · 0 comments · Fixed by #1167
Assignees

Comments

@joshdcox
Copy link

joshdcox commented Sep 27, 2022

Detailed Description

Even after #924, publishing to an S3 bucket with KMS encryption enabled results in 'file already exists and is different' errors like the following.

ERROR: unable to publish: unable to process packages: error putting file to aptly/focal/pool/main/a/a7xpg/a7xpg_0.11.dfsg1-10build3.1_amd64.deb: file already exists and is different: S3: us-east-1:orca-sre-staging-na1-repo-36812/aptly

The objects throwing the errors were PUT by Aptly 1.5.0 to S3 with KMS encryption and contain the user-defined x-amz-meta-md5 metadata with the correct MD5 checksum. The ETAG of the objects have a length of 32, which is the condition Aptly currently uses to assume the ETAG is an MD5 checksum.

AWS documentation states KMS encrypted objects do not use MD5 checksums of the content for their ETAG. It also makes no guarantees about the length of such ETAGs.

if len(destinationMD5) != 32 {

Context

Possible Implementation

Instead of assuming an ETAG of length 32 is an MD5 checksum of the content, Aptly could do one of the following:

  1. Use object metadata or upload details (Is this multi-part? Is this KMS encrypted?) to determine if the ETAG can be used for md5 checksum.
  2. In the S3PublishEndpoints portion of the Aptly config, allow users to specify whether or not to use object metadata rather than ETAGs for checksum values.
  3. Always use object metadata for checksum values on S3 (this would have a performance impact).

Option 2 is probably the simplest solution.

Your Environment

Aptly version 1.5.0
Running aptly on Ubuntu Focal LTS
Publishing Ubuntu Focal LTS mirrors to S3 buckets with KMS encryption

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants