Skip to content

feat(health): optimize validation, add automatic metadata regeneration and enforce strict linking - #411

Merged
javi11 merged 11 commits into
kipsilabs:mainfrom
drondeseries:main
Mar 14, 2026
Merged

feat(health): optimize validation, add automatic metadata regeneration and enforce strict linking#411
javi11 merged 11 commits into
kipsilabs:mainfrom
drondeseries:main

Conversation

@drondeseries

Copy link
Copy Markdown
Contributor

Summary

This PR introduces several critical improvements to the health monitoring and repair system to prevent stalling, reduce bandwidth usage, and automate the recovery of physically corrupted metadata.

🚀 Key Features & Fixes

  1. Optimized Usenet Validation (Bandwidth & Speed)

    • Issue: Health checks were downloading full segments even after confirmation, leading to high bandwidth usage (73GB+) and worker stalls.
    • Fix: Modified validation logic to immediately cancel the segment download as soon as the first non-zero byte is detected.
    • Result: Health check cycles now complete in under 1 minute with minimal data usage.
  2. Automatic Metadata Regeneration

    • New Feature: Added which automatically searches the persistent .nzbs folder when a file has corrupted metadata and no library path.
    • Benefit: Allows the system to "self-heal" by rebuilding the .meta file from the original NZB without requiring a redownload or ARR intervention.
  3. Improved Corruption Handling in Library Sync

    • Issue: Files with physically corrupted metadata were ignored by the sync process and never reached the health database.
    • Fix: Updated LibrarySync to detect unmarshal errors and register them as corrupted entries in the database.
    • Result: Corrupted files are now automatically caught, hidden from the mount (moved to corrupted_metadata/), and queued for repair.
  4. Strict ARR Linking

    • Removed legacy filename parsing fallbacks for Sonarr and Radarr. Repairs are now strictly triggered only when a direct path match or known relative path suffix is confirmed by the ARR instance, preventing "zombie" or incorrect repair triggers.
  5. Configurable Health Timeouts

    • Added read_timeout_seconds to the health configuration (defaulting to 30s) to allow users to tune the system for slow Usenet providers.

🛠 Implementation Details

  • Updated internal/usenet/validation.go with context cancellation logic.
  • Extended internal/importer/service.go with NZB search and re-processing capabilities.
  • Enhanced internal/database/health_repository.go query to pick up metadata-corrupted files even without a library_path.
  • Wired importerService into HealthWorker via internal/health/worker.go and setup commands.

@drondeseries
drondeseries force-pushed the main branch 3 times, most recently from c616e9f to 3808cbe Compare March 14, 2026 03:36
@javi11
javi11 merged commit 43a5bd4 into kipsilabs:main Mar 14, 2026
2 checks passed
yoshitaka420 pushed a commit to yoshitaka420/altmount that referenced this pull request Jun 1, 2026
…n and enforce strict linking (kipsilabs#411)

Co-authored-by: drondeseries <drondeseries@users.noreply.github.com>
Co-authored-by: Javier <javicuba11@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants