feat: Phase 6 — push sources, upload path and Paper plugin - #8
Closed
TheMeinerLP wants to merge 17 commits into
Closed
feat: Phase 6 — push sources, upload path and Paper plugin#8TheMeinerLP wants to merge 17 commits into
TheMeinerLP wants to merge 17 commits into
Conversation
A running Paper server can now push its own world to Apus instead of only being polled: paper-worldpush pauses autosave, forces one save, then copies and uploads only region files whose mtime or checksum actually changed since the last cycle, entirely off the main thread via Paper's AsyncScheduler/GlobalRegionScheduler. Uploads land in a per-tenant S3 staging prefix, authenticated with a tenant-bound world:push service token kept in the plugin's own config rather than tied to any user login, and a completion report goes to the Apus API. New module, own release track (pinned to paper-api 26.2.build.111, independent of this repo's own version line, matching telemetry-addon's existing precedent for foreign-version dependencies).
Both push-style sources (paper-worldpush writing directly, the UI's presigned multipart upload) stage their payload as a single object under a prefix in S3 before an ingest starts, so unlike S3SourceConnector neither reports versions via discover(). PushSourceConnector and UploadSourceConnector share that fetch logic through a new AbstractStagedSourceConnector; only the WorldSourceSpec.type discriminator differs between them. Covered by MinIO-backed tests (Testcontainers), excluded from the default test task and run via :ingest:integrationTest like the existing S3SourceConnectorTest.
POST /api/uploads initiates a presigned S3 multipart upload into a
tenant-scoped staging prefix (design spec §11.1): CreateMultipartUpload,
ListParts, CompleteMultipartUpload and AbortMultipartUpload all run
backend-side with the platform's own staging credentials, never
presigned -- only UploadPart is. POST /api/uploads/{uploadId}/complete
finalises it, summing the real S3-recorded part sizes via ListParts and
aborting rather than completing an upload whose actual total exceeds
the configured maximum. The staged object's key is always derived from
the caller's JWT-resolved namespace, never from request input.
POST /api/push/{token} is the one endpoint in this module that
authenticates via a tenant-bound service token instead of a JWT, per
design spec §10.3 -- looked up against labelled Secrets and compared
constant-time (MessageDigest.isEqual, exhaustive scan, no early return)
so neither the token nor a tenant's existence leaks through timing.
Creates one WorldIngest per configured world on the target push source,
mirroring WorldSourceReconciler's per-world loop for pull sources.
Adds the AWS SDK v2 dependency to the api module (S3Presigner ships
inside the s3 artifact itself in this SDK version, not a separate
s3-presigner module).
PushSourceConnector and UploadSourceConnector existed but IngestConfig still rejected APUS_SOURCE_TYPE=push/upload outright, so an ingest of either type could never start. Add both to the supported source types, select the matching connector in IngestMain, and add the shared staging env-var contract (APUS_SOURCE_STAGING_*) both connectors need. Proven end to end against real MinIO with a new PushIngestEndToEndTest: stage a world archive in a staging prefix, run IngestMain for push and upload, assert a valid bundle and manifest come out the other side.
Nothing created the Secret FabricPushTokenRepository validates push tokens against, so the push path was only usable by hand. Tokens are tenant-bound, not per-WorldSource (design spec §10.3, and the existing resolveNamespace already assumes this): TenantReconciler now creates one cryptographically random, URL-safe token per tenant, in a fixed-name Secret alongside the tenant's namespace, and never regenerates it on later reconciles so an already-configured paper-worldpush server never gets silently locked out. The raw token never appears in status, an event, or a log line -- only the Secret's fixed, non-secret name does. api's FabricPushTokenRepository now shares the Secret-shape constants with the operator instead of duplicating them, and its Javadoc documents the RBAC trade-off the current cluster-wide, label-scoped lookup implies, plus a narrower alternative left as a follow-up.
HttpPushNotifier sent {tenant, worldName, fileCount, bytesUploaded} as
the completion report body, but PushController/PushReportRequest only
ever deserialize {sourceName, version} -- every real push report from
this plugin would have been rejected with 400, only ever noticeable in
production. The token-as-path-segment transport itself was already
correct on both sides; only config.yml's comment wrongly called it a
bearer token, fixed too.
Adds the required world-source-name config key (a tenant may run more
than one push source, so the tenant-bound token alone can't pick one),
generates a per-cycle version identifier, and sends exactly the two
fields the API contract expects. New HttpPushNotifierTest locks the
wire shape in against a local HTTP stub.
…tract test IngestRenderContractTest never set APUS_BUNDLE_SOURCE_NAME, a required IngestConfig field since before phase 6, so 🏃integrationTest failed at the very first assertion whenever it actually ran. Add it and fold it into the expected bundle path, matching BundlePath's real tenant/sourceName/worldId/version shape. Unrelated to the phase 6 push path; found while verifying IngestConfig's other callers.
- New "Stand der Umsetzung" section up front: all six phases built, sharding deliberately not built after the phase 4 spike, and the three remaining open points (identity broker unselected, OIDC never tested against a real broker, paper-worldpush's save window untested). - §4 module table: Java 21 -> 25 everywhere (the root toolchain applies uniformly), world-ingest/runner-image -> the actual ingest/runner directory names, added hosting, corrected operator's stack (JOSDK + fabric8, no Micronaut) and api/ui to their current form. - §13.2: CRD generation marked done, per-module test coverage corrected. - §15: connector order and CRD generation marked resolved; bucket notifications corrected (a direct completion callback was built instead, not notifications or polling); two new open items (Paper save window, push-token RBAC broader than ideal).
Closes out the phase 6 push-source work: the ingest wiring, tenant push-token provisioning, and the paper-worldpush/api request-contract fix, plus verification results and remaining concerns.
BundleWriter has written level.dat (and, where present, per-dimension entities/poi) as part of every bundle since phase 2b, exactly as the design spec documents. IngestRenderContractTest's bucket-listing assertion still expected only the six region files plus manifest.json, so it failed on the now-legitimate extra level.dat object. Replace the exact-set comparison with two checks: every mandatory object (manifest.json, every region file) must still be present, and anything beyond that must match the spec-documented sidecar content (level.dat, entities/, poi/) rather than being an unconstrained allow-everything check. This keeps the test's power to catch missing or stray objects while not breaking again the next time a fixture exercises entities/poi.
…f hardcoding them Same fix as feat/phase-1-render-kern's MinioFixtures and feat/phase-2b-ingest's S3SourceConnectorTest: these three container-based integration tests (introduced in this phase's push/upload feature work) hardcoded the well-known MinIO default access/secret key pair. Generate a fresh, random pair per test run instead, long enough to satisfy MinIO's own minimum key lengths.
…onfig The strings the secret scanner flagged were not only in old commits: the phase 1 plan carried them in a code sample, and .gitguardian.yaml listed them in plaintext, which made the exemption file a finding of its own. The plan sample now shows a placeholder, and the exemptions are SHA256 digests.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last phase of the Apus spec. Stacked on PR #7.
Until now Apus pulled worlds — from an S3 bucket or a Pterodactyl panel. This adds the reverse: a running Paper server pushes its own world, and a user can upload one through the UI.
What this delivers
paper-worldpush— a Paper plugin that copies a live world consistently (pause autosave, save once, copy, resume) and incrementally, uploading only region files whose mtime or checksum changed. All copying and uploading happens off the server thread; only the save itself touches the main thread, because Bukkit requires it.POST /api/uploadswith presigned multipart, so world files never travel through the API.POST /api/push/{token}— the one endpoint that authenticates with a tenant-bound service token rather than a user login, because a server plugin must not stop working when a person leaves the team.Upload restrictions, measured rather than claimed
A presigned URL is a transferable credential: what it permits, it permits to anyone holding it. So these were verified empirically against real MinIO rather than asserted.
Caveat stated plainly: verified against MinIO, not independently against Ceph RGW. Same SigV4 mechanism, but that is an inference, not a second measurement.
An unresolved design conflict, stated rather than hidden
The plugin pushes many individual raw region files incrementally. The staged connector expects one object per version. The HTTP call between them is now correct, but an ingest triggered that way would likely find nothing usable.
The upload path (single archive) works end to end and is proven by
PushIngestEndToEndTestagainst real MinIO. The plugin path needs its own design pass.This surfaced because the plugin and the API endpoint were built simultaneously in separate worktrees — each side's assumption was reasonable, and neither could see the other's.
Also in this PR
level.datwas legitimately added by the writer. The check now requires every mandatory object and still rejects genuinely unexpected ones, without breaking on spec-documented sidecars.Known gaps