Skip to content

perf(s3): LIST-based existence + parallel tile upload (MAPCO-11322) - #257

Open
shimoncohen wants to merge 2 commits into
masterfrom
logic-3-io-async
Open

perf(s3): LIST-based existence + parallel tile upload (MAPCO-11322)#257
shimoncohen wants to merge 2 commits into
masterfrom
logic-3-io-async

Conversation

@shimoncohen

@shimoncohen shimoncohen commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

LOGIC-3 of MAPCO-11317. Scoped S3 I/O wins.

Scope decision

"Async end-to-end" taken literally would convert the whole tile pipeline (IDataUtils/IData/ITileMerger/Process/TaskExecutor) to async — a large rewrite that collides with SVC-1 and the other PRs. This PR delivers the self-contained I/O wins and leaves full pipeline-async as a follow-up.

Changes

  • LIST-based existenceS3Client.GetTileKey (used by TileExists) did a full GetObjectAsync, downloading the entire object body just to read its .Key. Now a single prefixed ListObjectsV2 with MaxKeys=1 — existence + real extension, no body transfer.
  • Parallel uploadS3.InternalUpdateTiles uploaded tiles one blocking PutObject at a time. Now Parallel.ForEach, materializing tiles first so the single-threaded grid/origin projection completes before uploads fan out. (chore: update logs + add parallel.for when putting updating tiles to S3 #112 attempted this via Task.Run(...).Wait(), which isn't actually parallel; this supersedes it.)
  • Removed no-op connection cap — dropped a ServicePointManager.DefaultConnectionLimit line that was added alongside the parallel upload. ServicePointManager governs the legacy HttpWebRequest stack; on net6 the AWS SDK's HttpClient/SocketsHttpHandler ignores it, so it never capped anything. Pure dead-code removal.

Deferred (follow-ups) — tracked in MAPCO-11364

  • P4 — full end-to-end async of the read/write path (sync-over-async .Result).
  • P5rGetTile still probes Jpeg-then-Png (up to 2 full GETs/tile); the speculative probe was not dropped. (Only GetTileKey/TileExists moved to LIST here.)
  • P10r — download parallelism; only uploads are parallelized in this PR.
  • P11 — ctor issues 25 LIST calls (FolderExists loop 0..24) to discover the zoom set → single ListObjectsV2 with Delimiter="/".
  • P14 — HTTP/FS clients' own async/existence tuning (kept out to bound this PR to S3).
  • P15 — upload-concurrency bound. With the no-op cap gone, the upload Parallel.ForEach has no deliberate ceiling: concurrency tracks ThreadPool sizing (blocking PutObject bodies) and socket count is unbounded (SocketsHttpHandler.MaxConnectionsPerServer default = int.MaxValue). Follow-up adds a config-driven MaxDegreeOfParallelism paired with AmazonS3Config.MaxConnectionsPerServer (the real net6 lever), value from a load test.

Note: parent ticket P5 specified HEAD (GetObjectMetadataAsync) for existence; this PR chose LIST instead. HEAD-vs-LIST to be settled consistently in MAPCO-11364.

Testing

Full suite: 1141 passed, 0 failed. TileExists client test drives ListObjectsV2; the S3 UpdateTiles data-type test no longer sequences UpdateTile (uploads are unordered) but still asserts one upload per tile.

Supersedes / relates

🤖 Generated with Claude Code

Scoped I/O wins for the S3 path (full end-to-end async deferred to its own
change).

- S3Client.GetTileKey: existence/key lookup did a full GetObjectAsync, which
  downloads the whole object body just to read its key. Use a single prefixed
  ListObjectsV2 (MaxKeys=1) instead — no body transfer.
- S3.InternalUpdateTiles: upload the batch's tiles with Parallel.ForEach
  instead of one blocking PutObject at a time. Tiles are materialized first so
  the single-threaded grid/origin projection runs before the uploads fan out;
  the S3 client and PutObject are independent per tile.
- Raise ServicePointManager.DefaultConnectionLimit (default 2) so parallel
  PUTs are not serialized on the connection pool.

TileExists client test now drives ListObjectsV2; the S3 UpdateTiles data-type
test no longer sequences UpdateTile (uploads are unordered) but still asserts
one upload per tile.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@shimoncohen shimoncohen self-assigned this Aug 5, 2026
ServicePointManager.DefaultConnectionLimit governs the legacy HttpWebRequest
stack; on net6 the AWS SDK's HttpClient/SocketsHttpHandler ignores it, so the
line never capped anything. Drop it and its misleading comment — pure dead-code
removal, no behavior change.

Bounding the S3 upload fan-out (config-driven MaxDegreeOfParallelism +
AmazonS3Config.MaxConnectionsPerServer, value from a load test) is deferred to
MAPCO-11364 (P15).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@shimoncohen
shimoncohen marked this pull request as ready for review August 5, 2026 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants