Use TUS to do replication and make it non-blocking when uploading by rickyrombo · Pull Request #99 · OpenAudio/go-openaudio

rickyrombo · 2026-01-14T17:36:34Z

No description provided.

raymondjacobson

seems good, one high level comment.
not 100% sure what the benefit is of having tus do server-server replication, but i dont see it hurting either

raymondjacobson · 2026-01-14T18:33:53Z

pkg/mediorum/server/replicate_worker.go

+		return fmt.Errorf("failed to create TUS request: %w", err)
+	}
+
+	createReq.Header.Set("Tus-Resumable", "1.0.0")


what's this version from? maybe we can lift it from version.go?

raymondjacobson · 2026-01-14T18:35:48Z

pkg/mediorum/server/serve_upload_grpc.go

-	}
+	// Handle image uploads immediately with synchronous replication
+	if upload.Template == JobTemplateImgSquare || upload.Template == JobTemplateImgBackdrop {
+		upload.Mirrors, err = ss.replicateFileParallel(ctx, formFileCID, filePath, upload.PlacementHosts)


why not handle all replication async?

i agree. unclear why the AI did this

I think this is why: #99 (comment)

pkg/mediorum/server/replicate_worker.go

…ons from normal uploads

Copilot

Pull request overview

This pull request refactors the file replication system to use TUS (Tus Resumable Upload Protocol) and makes replication non-blocking. The main goal is to improve upload responsiveness by decoupling replication from the upload request path.

Changes:

Adds asynchronous replication using worker pools that process uploads from a buffered channel
Implements TUS-based replication with authentication for peer-to-peer file transfers
Modifies upload processing to queue replication asynchronously instead of blocking on it

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 16 comments.

File	Description
pkg/mediorum/server/tusd.go	Adds authentication for TUS replication uploads and handles incoming replication requests with CID validation
pkg/mediorum/server/server.go	Adds replicationWork channel and starts replication workers as a managed routine
pkg/mediorum/server/serve_upload_grpc.go	Changes upload processing to queue replication asynchronously instead of blocking, marks images as done immediately
pkg/mediorum/server/replicate_worker.go	New file implementing replication worker pool with TUS client, periodic retry logic, and authenticated uploads

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-06T01:07:38Z

pkg/mediorum/server/serve_upload_grpc.go

+	// For images, mark as done immediately (no transcoding needed)
 	if upload.Template == JobTemplateImgSquare || upload.Template == JobTemplateImgBackdrop {
 		upload.TranscodeResults["original.jpg"] = formFileCID
 		upload.TranscodeProgress = 1
 		upload.TranscodedAt = time.Now().UTC()
 		upload.Status = JobStatusDone
-		if shouldCreate {
-			return ss.crud.Create(upload)
-		}
-		return ss.crud.Update(upload)
 	}


Images are marked as JobStatusDone but are still queued for replication. The findMissedReplications function only looks for uploads with status=JobStatusNew and template='audio', so images with status=JobStatusDone will never be picked up by the periodic replication job if they're missed. Either include JobStatusDone in the query, or ensure images are also replicated synchronously, or change the logic so images stay in JobStatusNew until replication completes.

pkg/mediorum/server/tusd.go

pkg/mediorum/server/replicate_worker.go

pkg/mediorum/server/server.go

pkg/mediorum/server/serve_upload_grpc.go

pkg/mediorum/server/replicate_worker.go

pkg/mediorum/server/serve_upload_grpc.go

pkg/mediorum/server/server.go

Copilot · 2026-02-06T01:07:42Z

pkg/mediorum/server/replicate_worker.go

+func (ss *MediorumServer) findMissedReplications() {
+	// Find uploads that don't have enough replicas
+	uploads := []*Upload{}
+	ss.crud.DB.Where("status = ? AND template = 'audio'", JobStatusNew).Find(&uploads)
+
+	for _, upload := range uploads {
+		if len(upload.Mirrors) < ss.Config.ReplicationFactor {
+			select {
+			case ss.replicationWork <- upload:
+				ss.logger.Debug("queued upload for replication", zap.String("uploadID", upload.ID))
+			default:
+				// Channel full, skip for now
+			}
+		}
+	}
+}


Multiple workers may process the same upload concurrently when findMissedReplications queues uploads. If an upload with insufficient mirrors is found by findMissedReplications, it could be processed by multiple workers simultaneously, leading to redundant replication attempts and potential race conditions when updating the Mirrors field. Consider adding a locking mechanism or status tracking to prevent concurrent processing of the same upload.

pkg/mediorum/server/tusd.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…ding (#99)" This reverts commit 06f61fb.

…ding (#99)" (#117) This reverts commit 06f61fb.

…ading (#99)" (#117) This reverts commit c4f1cca.

…ading (#99)" (#120) * Reapply "Use TUS to do replication and make it non-blocking when uploading (#99)" (#117) This reverts commit c4f1cca. * fix panic * fix panic * refresh upload struct before writes * Update pkg/mediorum/server/replicate_worker.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * don't replicate to every node * better error handling * buffer managed workers * better error management * better error management * validate computed cid matches replication filename * prevent mirror overwrites * prevent crash on malformed header * support placementHosts when replicating * persist transcode results * only add self to mirrors if self is supposed to be a mirror * test wrapper so that vscode "run tests" run in container * log panics in handlers, add logs for failed validations, and add retries for querying upload (fixes race condition in test) * fix replications to placement hosts * queue transcode replication _after_ saving upload record * generate private keys for testing * add tests for tus uploads * Update pkg/mediorum/server/replicate_worker.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * replicate using old method for now * exit if error setting upload status * fix transcode progress getting overwritten --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

rickyrombo added 3 commits January 8, 2026 17:23

Allow custom httpo client in SDK

5ca0f13

prevent transcode jobs from trying to run on incomplete uploads

b2dde27

Make replication non-blocking and use tus protocol w/ batching

af3e106

rickyrombo requested a review from raymondjacobson January 14, 2026 17:36

Base automatically changed from mjp-custom-http-client to main January 14, 2026 17:57

raymondjacobson approved these changes Jan 14, 2026

View reviewed changes

alecsavvy reviewed Jan 14, 2026

View reviewed changes

pkg/mediorum/server/replicate_worker.go Outdated Show resolved Hide resolved

rickyrombo added 7 commits January 14, 2026 14:30

use tus client instead of manual uploads. differentiate tus replicati…

c040833

…ons from normal uploads

add auth to replication

b4169ab

cleanup ai

eb7dae5

handle image replication via async worker as well

f7d5c00

Merge remote-tracking branch 'origin/main' into mjp-tus-replication

3372014

Merge remote-tracking branch 'origin/main' into mjp-tus-replication

0cfc6cb

Merge remote-tracking branch 'origin/main' into mjp-tus-replication

1276874

rickyrombo requested a review from Copilot February 6, 2026 00:51

Copilot started reviewing on behalf of rickyrombo February 6, 2026 00:51 View session

Copilot AI reviewed Feb 6, 2026

View reviewed changes

rickyrombo and others added 9 commits February 5, 2026 17:20

Update pkg/mediorum/server/replicate_worker.go

bb0c18b

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update pkg/mediorum/server/serve_upload_grpc.go

24b5f02

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update pkg/mediorum/server/server.go

ca43379

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update pkg/mediorum/server/tusd.go

a19f590

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update pkg/mediorum/server/replicate_worker.go

94d9284

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Apply suggestion from @Copilot

bc1f404

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Apply suggestion from @Copilot

f83a188

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

pr comments

433fd2a

add back validation

46ca1fd

rickyrombo merged commit 06f61fb into main Feb 6, 2026
4 checks passed

rickyrombo deleted the mjp-tus-replication branch February 6, 2026 03:03

rickyrombo added a commit that referenced this pull request Feb 6, 2026

Revert "Use TUS to do replication and make it non-blocking when uploa…

adc0dd6

…ding (#99)" This reverts commit 06f61fb.

rickyrombo mentioned this pull request Feb 6, 2026

Revert "Use TUS to do replication and make it non-blocking when uploading" #117

Merged

rickyrombo added a commit that referenced this pull request Feb 6, 2026

Revert "Use TUS to do replication and make it non-blocking when uploa…

c4f1cca

…ding (#99)" (#117) This reverts commit 06f61fb.

rickyrombo added a commit that referenced this pull request Feb 6, 2026

Reapply "Use TUS to do replication and make it non-blocking when uplo…

fbb3af1

…ading (#99)" (#117) This reverts commit c4f1cca.

rickyrombo mentioned this pull request Feb 6, 2026

Reapply "Use TUS to do replication and make it non-blocking when uploading (#99)" #120

Merged

Conversation

rickyrombo commented Jan 14, 2026

Uh oh!

raymondjacobson left a comment

Choose a reason for hiding this comment

Uh oh!

raymondjacobson Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

raymondjacobson Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

rickyrombo Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

rickyrombo Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants