Skip to content

feat(CLI): add --parallel flag to phrase pull for concurrent downloads#1067

Merged
Sven Dunemann (forelabs) merged 2 commits intophrase:mainfrom
tetienne:feat/pull-parallel
Apr 1, 2026
Merged

feat(CLI): add --parallel flag to phrase pull for concurrent downloads#1067
Sven Dunemann (forelabs) merged 2 commits intophrase:mainfrom
tetienne:feat/pull-parallel

Conversation

@tetienne
Copy link
Copy Markdown
Contributor

@tetienne Thibaut Etienne (tetienne) commented Mar 19, 2026

We run phrase pull in CI pipelines with 379 locale files across 3 projects. Sequential downloads take over a minute per run, and the latency adds up across branches and environments. This PR adds parallel downloads to dramatically reduce pull time.

Pairs well with #1066 (--cache for ETag-based conditional requests): parallel mode makes the initial pull fast, and caching makes subsequent pulls nearly free by returning 304 Not Modified (which don't count against rate limits).

Summary

  • Add --parallel flag (-p) to phrase pull that downloads locale files using up to 4 concurrent requests (matching the Phrase API concurrency limit)
  • Uses errgroup with SetLimit(4) for bounded parallelism
  • Results are collected and printed in original locale order after all downloads complete (clean, deterministic output)
  • A shared mutex coordinates rate-limit pauses across all workers
  • Only supported in sync mode; --parallel with --async warns and ignores
  • Extracts buildDownloadOpts helper to eliminate code duplication between sequential and parallel paths

Usage

# Parallel download (4 concurrent requests)
phrase pull --parallel

# Short flag
phrase pull -p

# Without flag: sequential as before (no regression)
phrase pull

Real-world results

Tested against a production project with 379 locale files across 3 Phrase projects:

$ time phrase pull
Downloaded en to locales/en/tradfile.json
Downloaded fr to locales/fr/tradfile.json
...
(379 files downloaded sequentially in ~1m 10s)

$ time phrase pull --parallel
Downloaded en to locales/en/tradfile.json
Downloaded fr to locales/fr/tradfile.json
...
(379 files downloaded in parallel in ~11s)
Mode Time Speedup
phrase pull 1m 10s baseline
phrase pull --parallel 11.5s ~6x faster

Design decisions

  • 4 concurrent requests: Matches the Phrase API's documented concurrency limit. Going higher would result in rejected requests.
  • errgroup with SetLimit: Standard Go concurrency pattern. Provides bounded parallelism, fail-fast error propagation via context cancellation, and clean Wait() semantics.
  • Ordered output: Results are stored in a pre-allocated slice indexed by position. Output is printed after all downloads complete, preserving the original locale file order regardless of goroutine completion order.
  • Rate-limit mutex: When any worker hits HTTP 429, it holds a shared mutex while waiting for the rate limit window to reset. Other workers block on the mutex before making their next request, preventing a thundering herd.
  • Shared buildDownloadOpts: Extracted the download options preparation (params, file format, tags) into a helper used by both sequential and parallel paths, eliminating ~20 lines of duplication.
  • Timeout: Parallel path uses context.WithTimeout matching the existing 30-minute timeout from the sequential path.

Test plan

  • go build ./cmd/internal/... compiles
  • go test ./cmd/internal/... passes
  • go vet ./cmd/internal/ clean
  • Manual test: phrase pull --parallel downloads 379 files in 11.5s
  • Manual test: phrase pull without --parallel behaves identically
  • phrase pull --parallel --async warns and ignores parallel

@tetienne Thibaut Etienne (tetienne) force-pushed the feat/pull-parallel branch 2 times, most recently from e4541d8 to 2fa4000 Compare March 19, 2026 14:44
@tetienne Thibaut Etienne (tetienne) marked this pull request as ready for review March 19, 2026 14:53
Copy link
Copy Markdown
Collaborator

@jablan jablan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks really solid to me, a non-go developer. I have a question regarding the tests though.

Copy link
Copy Markdown
Collaborator

@jablan jablan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thibaut Etienne (@tetienne) thanks a lot! we're going to merge it only on Monday, though. I presume you are already using your home baked version?

@tetienne
Copy link
Copy Markdown
Contributor Author

jablan not on our main ci but locally I played with it yes

@forelabs
Copy link
Copy Markdown
Member

Thibaut Etienne (@tetienne) due to the merge of #1066 we have conflicts on this PR now. Thank you a lot for your contribution! Would you mind resolving the conflicts? We appreciate your support ❤️

Download locale files using up to 4 concurrent requests (matching the
Phrase API concurrency limit) via errgroup.

Results are collected and printed in order after all downloads complete
for clean output. A shared mutex coordinates rate-limit pauses across
all workers.

Only supported in sync mode; --parallel with --async warns and ignores.
Drop TestRateGate_* tests that only verified sync.RWMutex
standard library behavior without testing any PR code.
@tetienne
Copy link
Copy Markdown
Contributor Author

Sven Dunemann (@forelabs) Many thanks for this. I have updated my PR to resolve the conflicts.

@forelabs Sven Dunemann (forelabs) merged commit d93d302 into phrase:main Apr 1, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants