Support skipping upload if file already exists #201
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [github.com/aws/aws-sdk-go](https://togithub.com/aws/aws-sdk-go) | `v1.54.6` -> `v1.54.14` | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>aws/aws-sdk-go (github.com/aws/aws-sdk-go)</summary> ### [`v1.54.14`](https://togithub.com/aws/aws-sdk-go/blob/HEAD/CHANGELOG.md#Release-v15414-2024-07-03) [Compare Source](https://togithub.com/aws/aws-sdk-go/compare/v1.54.13...v1.54.14) \=== ##### Service Client Updates - `service/application-autoscaling`: Updates service documentation - `service/directconnect`: Updates service documentation - This update includes documentation for support of new native 400 GBps ports for Direct Connect. - `service/organizations`: Updates service API and documentation - Added a new reason under ConstraintViolationException in RegisterDelegatedAdministrator API to prevent registering suspended accounts as delegated administrator of a service. - `service/rekognition`: Updates service API and documentation - This release adds support for tagging projects and datasets with the CreateProject and CreateDataset APIs. - `service/workspaces`: Updates service API - Fix create workspace bundle RootStorage/UserStorage to accept non null values ### [`v1.54.13`](https://togithub.com/aws/aws-sdk-go/blob/HEAD/CHANGELOG.md#Release-v15413-2024-07-02) [Compare Source](https://togithub.com/aws/aws-sdk-go/compare/v1.54.12...v1.54.13) \=== ##### Service Client Updates - `service/ec2`: Updates service API and documentation - Documentation updates for Elastic Compute Cloud (EC2). - `service/fms`: Updates service API - `service/s3`: Updates service API, documentation, and examples - Added response overrides to Head Object requests. ### [`v1.54.12`](https://togithub.com/aws/aws-sdk-go/blob/HEAD/CHANGELOG.md#Release-v15412-2024-07-01) [Compare Source](https://togithub.com/aws/aws-sdk-go/compare/v1.54.11...v1.54.12) \=== ##### Service Client Updates - `service/apigateway`: Updates service API - Add v2 smoke tests and smithy smokeTests trait for SDK testing. - `service/cognito-identity`: Updates service API - `service/connect`: Updates service API, documentation, and paginators - `service/docdb`: Updates service API - Add v2 smoke tests and smithy smokeTests trait for SDK testing. - `service/eks`: Updates service API - `service/payment-cryptography`: Updates service API and documentation - `service/payment-cryptography-data`: Updates service API, documentation, and waiters - `service/states`: Updates service API - Add v2 smoke tests and smithy smokeTests trait for SDK testing. - `service/swf`: Updates service API - Add v2 smoke tests and smithy smokeTests trait for SDK testing. - `service/wafv2`: Updates service API and documentation ### [`v1.54.11`](https://togithub.com/aws/aws-sdk-go/blob/HEAD/CHANGELOG.md#Release-v15411-2024-06-28) [Compare Source](https://togithub.com/aws/aws-sdk-go/compare/v1.54.10...v1.54.11) \=== ##### Service Client Updates - `service/acm-pca`: Updates service API - `service/cloudhsmv2`: Updates service API and documentation - Added 3 new APIs to support backup sharing: GetResourcePolicy, PutResourcePolicy, and DeleteResourcePolicy. Added BackupArn to the output of the DescribeBackups API. Added support for BackupArn in the CreateCluster API. - `service/connect`: Updates service API and documentation - `service/elasticmapreduce`: Updates service API and documentation - This release provides the support for new allocation strategies i.e. CAPACITY_OPTIMIZED_PRIORITIZED for Spot and PRIORITIZED for On-Demand by taking input of priority value for each instance type for instance fleet clusters. - `service/glue`: Updates service API and documentation - Added AttributesToGet parameter to Glue GetDatabases, allowing caller to limit output to include only the database name. - `service/kinesisanalyticsv2`: Updates service API and documentation - `service/opensearch`: Updates service API and documentation - `service/pi`: Updates service API and documentation - `service/workspaces`: Updates service API and documentation - Added support for Red Hat Enterprise Linux 8 on Amazon WorkSpaces Personal. ### [`v1.54.10`](https://togithub.com/aws/aws-sdk-go/blob/HEAD/CHANGELOG.md#Release-v15410-2024-06-27) [Compare Source](https://togithub.com/aws/aws-sdk-go/compare/v1.54.9...v1.54.10) \=== ##### Service Client Updates - `service/application-autoscaling`: Updates service API and documentation - `service/chime-sdk-media-pipelines`: Updates service API and documentation - `service/cloudfront`: Updates service API and documentation - Doc only update for CloudFront that fixes customer-reported issue - `service/datazone`: Updates service API, documentation, and paginators - `service/elasticache`: Updates service API - Add v2 smoke tests and smithy smokeTests trait for SDK testing. - `service/mq`: Updates service API and documentation - This release makes the EngineVersion field optional for both broker and configuration and uses the latest available version by default. The AutoMinorVersionUpgrade field is also now optional for broker creation and defaults to 'true'. - `service/qconnect`: Updates service API, documentation, and paginators - `service/quicksight`: Updates service API and documentation - Adding support for Repeating Sections, Nested Filters - `service/rds`: Updates service API, documentation, waiters, paginators, and examples - Updates Amazon RDS documentation for TAZ export to S3. - `service/sagemaker`: Updates service API and documentation - Add capability for Admins to customize Studio experience for the user by showing or hiding Apps and MLTools. - `service/workspaces`: Updates service API and documentation - Added support for WorkSpaces Pools. ### [`v1.54.9`](https://togithub.com/aws/aws-sdk-go/blob/HEAD/CHANGELOG.md#Release-v1549-2024-06-26) [Compare Source](https://togithub.com/aws/aws-sdk-go/compare/v1.54.8...v1.54.9) \=== ##### Service Client Updates - `service/controltower`: Updates service API, documentation, and paginators - `service/eks`: Updates service API and documentation - `service/ivs-realtime`: Updates service API, documentation, and paginators - `service/kinesisanalyticsv2`: Updates service API, documentation, and paginators - `service/opensearch`: Updates service API and documentation ### [`v1.54.8`](https://togithub.com/aws/aws-sdk-go/blob/HEAD/CHANGELOG.md#Release-v1548-2024-06-25) [Compare Source](https://togithub.com/aws/aws-sdk-go/compare/v1.54.7...v1.54.8) \=== ##### Service Client Updates - `service/autoscaling`: Updates service API and documentation - Doc only update for Auto Scaling's TargetTrackingMetricDataQuery - `service/ec2`: Updates service API - This release is for the launch of the new u7ib-12tb.224xlarge, R8g, c7gn.metal and mac2-m1ultra.metal instance types - `service/networkmanager`: Updates service API and documentation - `service/workspaces-thin-client`: Updates service API and documentation ### [`v1.54.7`](https://togithub.com/aws/aws-sdk-go/blob/HEAD/CHANGELOG.md#Release-v1547-2024-06-24) [Compare Source](https://togithub.com/aws/aws-sdk-go/compare/v1.54.6...v1.54.7) \=== ##### Service Client Updates - `service/bedrock-runtime`: Updates service API - `service/customer-profiles`: Updates service API and documentation - `service/ec2`: Updates service API - Fix EC2 multi-protocol info in models. - `service/qbusiness`: Updates service API and documentation - `service/ssm`: Updates service API - Add sensitive trait to SSM IPAddress property for CloudTrail redaction - `service/workspaces-web`: Updates service API and documentation </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am" (UTC), Automerge - "before 4am" (UTC). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/woodpecker-ci/plugin-s3). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MjEuOSIsInVwZGF0ZWRJblZlciI6IjM3LjQyMS45IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
6543
approved these changes
Jul 9, 2024
](https://renovatebot.com){kind=link}
6543
pushed a commit
that referenced
this pull request
Jul 18, 2024
This PR was opened by the [ready-release-go](https://github.com/woodpecker-ci/plugin-ready-release-go) plugin. When you're ready to do a release, you can merge this pull-request and a new release with version `1.3.0` will be created automatically. If you're not ready to do a release yet, that's fine, whenever you add more changes to `main` this pull-request will be updated. ## [1.3.0](https://github.com/woodpecker-ci/plugin-s3/releases/tag/v1.3.0) - 2024-07-18 ### 📈 Enhancement - Support skipping upload if file already exists [[#201](#201)] ### 🐛 Bug Fixes - Fix env var parsing with urfave/cli v2 [[#198](#198)] ### Misc - fix(deps): update module github.com/aws/aws-sdk-go to v1.54.20 [[#203](#203)]
|
🎉 This PR is included in version 1.3.0 🎉 The release is now available here Thank you for your contribution. ❤️📦🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
Currently, files are always uploaded to the remote side, even if they didn't change between runs. This causes new versions to be created, incurring potentially unneeded costs and resource waste.
In some cases overwrite might be desired, e.g. for backups, as the files might have changed on disk.
In other scenarios, the CI might be building idempotent assets, in which case "overwrite" is not desired, and the upload should be skipped.
The default will still be
overwrite=true, i.e. it won't break any workflow in use.Solution
Add param
overwrite=true/falsewhich skips the upload in case the file already exists. This saves bandwith and storage space on the remote side.Implementation
Before doing anything, a call to
HeadObjectInputis done which checks if the file already exists. If so, the upload is skipped.The implementation could potentially be improved by checking the sha of the object on both sides and only skip the upload if both are identical. The current solution is sufficient for my use case, though.
Testing
I am using a custom image with this branch since weeks already - didn't encounter any issues so far.