perf(parser): improve performance and progress reporting for NZBs with 430 Not Found - #394
Merged
Conversation
…h 430s - Increase goroutine pool sizes from NumCPU to min(n,20) for I/O-bound operations in fetchAllFirstSegments and ParseFile - Cache segment IDs that returned 430 Not Found and skip redundant NNTP calls in normalizeSegmentSizesWithYenc for permanently-missing segments - Parallelize the 16KB completion loop using errgroup instead of sequential fetches - Add IsArticleNotFound to FirstSegmentData to distinguish permanent 430s from transient timeouts - Advance UpdateDownloadProgress on all error paths in fetchYencHeaders and known-404 shortcuts so the metrics tracker never stalls - Add ProgressTracker parameter to ParseFile; fetchAllFirstSegments ticks it atomically per file so the broadcaster-driven UI bar advances from 0→10% during parsing instead of jumping only at completion Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…gment bytes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
yoshitaka420
pushed a commit
to yoshitaka420/altmount
that referenced
this pull request
Jun 1, 2026
…h 430 Not Found (kipsilabs#394) * perf(parser): improve performance and progress reporting for NZBs with 430s - Increase goroutine pool sizes from NumCPU to min(n,20) for I/O-bound operations in fetchAllFirstSegments and ParseFile - Cache segment IDs that returned 430 Not Found and skip redundant NNTP calls in normalizeSegmentSizesWithYenc for permanently-missing segments - Parallelize the 16KB completion loop using errgroup instead of sequential fetches - Add IsArticleNotFound to FirstSegmentData to distinguish permanent 430s from transient timeouts - Advance UpdateDownloadProgress on all error paths in fetchYencHeaders and known-404 shortcuts so the metrics tracker never stalls - Add ProgressTracker parameter to ParseFile; fetchAllFirstSegments ticks it atomically per file so the broadcaster-driven UI bar advances from 0→10% during parsing instead of jumping only at completion
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fetchAllFirstSegmentsandParseFilechanged fromruntime.NumCPU()tomin(n, 20)— these operations are network-bound, not CPU-boundnotFoundIDsset and short-circuited innormalizeSegmentSizesWithYenc, saving 1–3 NNTP round-trips per missing fileerrgroupfetch + in-order assemblyFirstSegmentData.IsArticleNotFoundis now populated so future retry logic can treat permanent and transient failures differentlyUpdateDownloadProgressis called on every error path infetchYencHeadersand known-404 shortcuts;ParseFilenow accepts aprogress.ProgressTrackerand ticks it atomically per file so the broadcaster UI bar advances incrementally (0→10%) during parsing instead of jumping only when the entire parse finishesTest plan
go test ./internal/importer/parser/...passesgo test ./internal/importer/...passesmake(full build) passes"known not found, skipping"instead of repeated NNTP calls for the same segment IDs🤖 Generated with Claude Code