Skip to content

Improve promotional copy#213

Merged
matthiasL-scality merged 1 commit intomainfrom
PTFE-3086-improve-copy-s3-to-s3
Mar 16, 2026
Merged

Improve promotional copy#213
matthiasL-scality merged 1 commit intomainfrom
PTFE-3086-improve-copy-s3-to-s3

Conversation

@matthiasL-scality
Copy link
Contributor

Purpose

Improve copy s3 to s3 (promote command)
This pull request updates the object copying logic in lua/copy_build.lua to improve efficiency and performance by introducing batch processing. Instead of copying objects one at a time, the code now processes them in batches using ngx.location.capture_multi, which allows multiple requests to be sent concurrently.

Performance improvements to object copying:

  • Refactored the object copying loop to collect all object names up front and process them in batches of 16 using ngx.location.capture_multi for concurrent HTTP requests, instead of copying objects one at a time.
  • Updated progress output and flushing logic to reflect batch processing and ensure correct progress reporting.

@bertranddemiddelaer
Copy link
Contributor

Nice to have: I did not see a test regarding the promotion of a build. This would be really nice to have in order to fully validate those optimizations.

@matthiasL-scality matthiasL-scality force-pushed the PTFE-3086-improve-copy-s3-to-s3 branch from 62c00b8 to 3d9e99a Compare March 13, 2026 13:49
@matthiasL-scality
Copy link
Contributor Author

Nice to have: I did not see a test regarding the promotion of a build. This would be really nice to have in order to fully validate those optimizations.

https://github.com/scality/artifacts/pull/213/changes#diff-019e68f115c410de17a78789e0eaaf5260ae05195c2e21714eb07983f8998ab4R74-R88 should do the job

@matthiasL-scality matthiasL-scality marked this pull request as ready for review March 13, 2026 14:21
@matthiasL-scality matthiasL-scality requested a review from a team as a code owner March 13, 2026 14:21
@matthiasL-scality matthiasL-scality force-pushed the PTFE-3086-improve-copy-s3-to-s3 branch from 3d9e99a to 1ac00c8 Compare March 13, 2026 14:51
Copy link
Contributor

@bertranddemiddelaer bertranddemiddelaer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@matthiasL-scality matthiasL-scality merged commit b93809e into main Mar 16, 2026
4 checks passed
@matthiasL-scality matthiasL-scality deleted the PTFE-3086-improve-copy-s3-to-s3 branch March 16, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants