s3 publish should perform differential publishes

[cpoole]

If I publish a snapshot with a distribution and components to a specific path the next time I publish a snapshot to the same path aptly should check for existence of the packages in the s3 pool and choose to NOT reupload already existing packages.

Detailed Description

here is my process flow:

• create a local repo
  • 1 to 1 mapping of distribution+component to local repo
• import packages to that repo
• snapshot repo
• multipublish snapshots to a location in s3
  • following this guide https://www.aptly.info/doc/feature/multi-component/
  • example command: aptly publish snapshot 1.0.5_xenial -component=main,security,updates xenial_main-snapshot xenial_security-snapshot xenial_updates-snapshot s3:main:xenial/
• create a new local repo for my next "version" of mirrors
• import packages
• snapshot repo
• perform the multicomponent snapshot publish to the same location in s3 but with different snapshots and a different distribution.

This means my "versions" are distributions that all share the same pool. However when I publish the new pool aptly will re-upload ALL of the packages even if the same package (package with the same name) already exists within the pool.

It would be nice to add a flag for aptly so that I can accept that if a package with the same name already exists in the remote I won't try and re-upload

Context

We mirror a lot of packages, if this could be implemented the total bandwidth spend and time waiting for new distributions would be significantly reduced

Possible Implementation

It looks rather simple.

here https://github.com/smira/aptly/blob/master/s3/public.go#L142 a check could be added to see if the file already exists before deciding to upload it.

This check could be accomplished by calling checking against a result from the internalFilelist function as well.

Please let me know if you're interested and I can submit a PR

[smira]

Actually aptly is never overwriting existing package files on S3. Correct entry point is LinkFromPool: https://github.com/smira/aptly/blob/master/s3/public.go#L264

It lists S3 contents and skips files which are already present in S3 package pool.

So unless there's some ugly bug I'm not aware of, aptly is already working the way you described above.

[smira]

Closing on inactivity, please feel free to reopen.