Refactor media browsing around stable media records#33
Merged
Conversation
Introduce manifest-backed media ordering and record hydration as the new canonical media layer. Replace media-session/list assumptions with stable media IDs across viewer navigation, streaming, gallery, search, continue watching, sync, chat view sharing, TV casting, progress, and GhostStream playback paths. Refactor streaming and gallery into smaller data sources and lifecycle-owned components, including row rendering, pagination, what's new, continue watching, gallery timeline/month views, lazy loading, progress bars, and subfolder navigation. Rework event propagation for uploads, batch uploads, deletes, renames, hidden-content changes, mount changes, thumbnail progress, indexing updates, and layout refreshes so client state follows backend media/storage changes without stale cards or wrong viewer targets. Harden app behavior around admin/visibility handling, session/profile state, rate limiting, payload size, upload concurrency, Cloudflare tunnel URL capture, TV/kiosk progress persistence, and low-memory media browsing. Refresh GhostStream integration docs, architecture/QA docs, stress tooling, and Python/JavaScript coverage for the refactored media and storage surfaces. Update README.md Hydrate and append manifest records when missing Instead of paginating through ordering to find missing media, hydrate individual records from the media manifest and append them to the view's orderedIds. Applied to openCategoryViewer (media/loader.js), navigateToState (sync/manager.js) and resolveStoredCastRecord (sync/tvPlayerModal.js). This removes page-append loops/guards, uses manifest.hydrate to fetch canonical records, ingests the updated view ordering, and updates warning text for hydration failures. Improves reliability when opening viewers or resolving cast records that are not present on the current pages. Improve media sync, playback & navigation fallbacks Several fixes and hardening for media viewer, playback and sync: - navigation.js: Always broadcast host viewer updates when sync is enabled and host (removed previousIndex check) to avoid missed syncs. - thumbnailHandler.js: Prevent autoplay attempts for sync guests; retry playback with mute fallback and attach readiness event handlers to reliably start videos on capable clients. - sync/manager.js: Make view ordering requests more robust (try/catch), request paged results with append until media is found, and fall back to category-based loading when view-based load fails or viewer session is missing/hidden. Use startRecordId for category viewer navigation. - sync playback logic: Only retry play when video readyState is sufficient, use safeSeek return value to track seeks, and attach an onseeked handler to resume playback after long seeks to avoid race conditions. These changes improve reliability of synchronized viewing across hosts/guests and handle edge cases where media pages or viewer state are out-of-sync. Batch delete/visibility, rate limiter and UI fixes Introduce batch file operations, improve rate limiting semantics, and adjust streaming UI/behaviour. Key changes: - Add /api/storage/media/batch-delete and storage_media_file_service.delete_files to delete multiple files atomically and emit a single STORAGE_FILES_DELETED bus event; cleanup and cache invalidation consolidated. - Add /api/admin/files/batch-visibility and AdminVisibilityController._batch_file_visibility to hide/unhide many files with grouped category invalidation. - Storage event handler now listens for STORAGE_FILES_DELETED and emits grouped category updates; client socket handlers handle files_deleted and defer refreshes during active uploads. - StorageUploadController now blocks-and-waits via wait_for_upload_capacity for both chunked and non-chunked uploads (avoids mid-stream aborts); helper to determine file size added. - Rate limiter: fix Mbps->bytes conversion, introduce a BURST_WINDOW, token refund on global rejection, blocking wait_for_upload_capacity helper, and get_effective_upload_limit_mbps for reporting; default upload limits increased in config and runtime defaults. - storage_path_service now prefers registered categories over auto-derived subfolder IDs to ensure uploads and media URLs resolve to real categories. - Add ProxyFix to app init so remote_addr reflects real client when behind local reverse proxy; include effective upload limit in system tunnel health response. - Streaming layout refactor: separate subfolder and media strips, mount virtual scroller into media strip, CSS tweaks for layout, and client JS updated to use batched endpoints and improved error/notification handling. Tests updated accordingly. These changes reduce noisy per-file events, prevent corrupted uploads caused by 429s during chunked uploads, and make UI updates more robust and efficient. Improve TV casting progress persistence Persist and commit TV cast progress more reliably and improve related UI/DOM handling. Key changes: - Progress persistence: process_tv_progress_update accepts a force flag and skips when state is empty; fills missing video_duration and thumbnail_url from existing DB progress; avoid broadcasting when nothing was saved. - DB fallback: save_video_progress reads existing video_progress row to populate missing duration, thumbnail, and category_id before insert/replace. - TV cast lifecycle: stop_cast now accepts an optional payload (currentTime/duration) and commits progress before clearing state; playback actions (play/pause/seek) trigger progress commits. Added _commit_tv_progress helper and progress dependency on TVCastService. - Frontend sync/UI: emit stop payload with current playback time and duration from the client; track currentPlaybackTime and restore active viewer file selection more robustly; update progress selectors to handle data-video-url and include timestamp/duration in Continue Watching cache keys; progress updates compute effective duration/thumbnail using stored progress and update card progress bars. - TV modal UX: show loading spinner and clearer "Starting/Loading/Casting" subtitle states and defer leaving loading until first playback state is received. - Misc: added safe coercion for numeric times, minor selector/behavior fixes in profile selector and progress bar DOM querying. These changes reduce lost or incomplete progress saves when stopping casts or when the client omits duration/thumbnail, and provide clearer feedback in the TV player modal. Remove media session tracking and session_id Remove legacy media session tracker and related session_id usage across backend and frontend. Deleted media_runtime_store and media_session_service, removed clear_session_tracker calls and session resolution helpers (_resolve_media_session_id/get_session_id) from controllers and services, and updated media ordering to no longer accept or propagate session_id. Simplified TV/cast state to prefer mediaId (removed fallback media_id fields), removed search view/params handling and client-side order-hash/reset logic (lastSentOrderHash/resetOrderHash), and adjusted tests to match the new behavior. Doc: expand media API; layout refresh fixes Docs: expand media browsing API section with endpoint table, new /api/media/orders batched ordering, canonical view types (CANON_VIEW_TYPES), and stable media id format. JS: File manager now consumes deferred refresh flags when closed and calls refreshAllLayouts after hide/unhide operations. Socket handlers treat file_hidden/file_unhidden as layout-invalidating events. CategoryRowComponent ensures virtual scroller is unmounted/rebuilt to avoid stale nodes when shape/status changes or rendering shells/empty states. lazyLoad requeues disconnected images (instead of dropping) so images from detached virtual-scroller chunks get observed. liveVisibility defers layout refresh while file-manager modal is open and updates log message. Add whats_new ordering and integrate frontend Introduce a new media-ordering view 'whats_new' and wire it through backend and frontend. MediaOrderingService gains _order_whats_new and media_discovery now delegates newest-media requests to the ordering registry. Frontend streaming modules were updated to consume the ordering view: fetchNewestMedia now requests an ordering viewKey, pins/hydrates the manifest, and sets whatsNewViewKey; WhatsNewRowComponent subscribes to the view so manifest/order invalidations reconcile in-place. MediaInvalidationModule now snapshots entries referencing invalid ids and recognizes global category views (including whats_new) to drive proper refetch behavior. CategoryRowComponent adds append-only optimization and avoids unnecessary rebuilds; rowShell prefetch margin increased to reduce mid-scroll re-renders. Native TV runtime and various JS modules replace hard-coded socket event strings with SOCKET_EVENTS/TV_EVENTS constants. Removed an unused "On This Day" helper and made several test updates to mock the new ordering API and viewKey behavior. These changes unify ordering, reduce DOM churn on updates, and standardize socket event usage. Refactor socket events and upload flow Remove legacy CONTENT_VISIBILITY_CHANGED and FILE_RENAMED socket constants and update code to import/use centralized SOCKET_EVENTS in JS modules. Replace the ad-hoc file-save loop in SystemTransferController with standard_upload_service.upload_files, simplify drive/subfolder handling, and stop emitting per-upload batch bus events from the controller. Add rename-handling in media invalidation (dedupe rename notifications, update progress DB and emit APP_EVENTS.FILE_RENAMED_UPDATED) and remove the now-redundant local rename emit in quick actions. Misc: minor docstring/import cleanups and test update to reflect removed constants. Standardize storage events and gallery UI fixes Unify and harden storage-related event handling and polish gallery UI behaviors. Key changes: emit centralized bus/library events (with timestamps, media URLs and invalidated IDs) for uploads, deletes, renames and folder operations instead of ad-hoc socket emits; improve category resolution for filesystem paths; add defensive logging around filesystem operations; propagate media URL/stable-id info when invalidating library caches. Gallery and media frontend updates: CSS fix for VirtualScroller sentinels, refactor lazy-loader usage, simplify overlay lazy-load logic, improve timeline/media-data handling (avoid unnecessary grouping, use timelineProjectionVersion for subscriptions), and make viewer opening prefer source view ordering (openViewer/openViewerFromGalleryCard changes). Misc: remove unused socket-emission helpers, add safer error/debug logs, and attach timestamps to several storage events for better tracing. Revert "fix streaming and gallery shimmer state bugs" This reverts commit 6b3f83a325eb8b96955e843bfe878aa73a4b13b4. fix streaming and gallery shimmer state bugs request coalescing gallery and streaming migration fix: cloudflare tunnel capture
The backend rejects /api/profiles/select with 409 when a profile is
already owned by another session, but the client swallowed the error
behind a generic catch. On auto-restore (quiet mode) this left the
stored profile id in localStorage and silently re-opened the picker
with no explanation, so users didn't realize they had been bumped.
Handle 409 explicitly in selector.js:
- If the conflicting id matches the locally stored profile (the
"log back in" case), clear localStorage via clearStoredProfile(),
null out the active profile, dispatch PROFILE_SELECTED_EVENT, and
force the picker open with { required: true }.
- If a different profile was clicked (race during manual switch),
warn and refresh lock state without clearing the user's own
stored profile.
- Always surface a toast — including during quiet auto-restore —
so the user knows why they were signed out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR overhauls GhostHub’s media browsing pipeline around manifest-backed media ordering and record hydration.
The old media-session/list-index assumptions have been replaced with stable media IDs across viewer navigation, Streaming, Gallery, Search, Continue Watching, Sync Mode, chat view sharing, TV casting, progress tracking, and GhostStream playback paths. This makes media identity consistent across pagination, sorting, filtering, shuffled views, uploads, deletes, renames, hidden-content changes, and partially loaded views.
Streaming and Gallery were refactored into smaller lifecycle-owned modules and data sources. Row rendering, pagination, What's New, Continue Watching, Gallery timeline/month views, lazy loading, progress bars, and subfolder navigation now use the shared media ordering/manifest flow instead of layout-local full media lists.
Event propagation was also reworked. Uploads, batch uploads, deletes, renames, hidden-content changes, USB mount changes, thumbnail progress, indexing updates, and layout refreshes now invalidate scoped media records/views so client state follows backend storage changes without stale cards or wrong viewer targets.
This PR also hardens related app flows:
trycloudflare.comURLs more reliably./api/configno longer exposespython_configto non-admin users, preventing sensitive Python-side settings such as Pinggy tunnel tokens from leaking through the client config response.Validation
venv/bin/python scripts/run_all_tests.pypasses locally1082 passed1543 passed, 23 skippedStress Test Results
GhostHub stress suite completed successfully in quick mode.
READY FOR RELEASE00:04:57121101Worst Case Scenario also passed with all major workloads running simultaneously for
61.3s.521 requests,546,308,096 bytes,0 errors1,197 messages,0 errors3,732 requests,0 errors166 navigations,0 errors250 saves,250 reads,0 errors258 scans,774 media fetches47% avg,53% max16% avg,16% max42°C avg,44°C maxResult:
System handled worst case scenario successfully.Skipped: long stability test was intentionally skipped in quick mode.
Pi hardware notes
This refactor is intended to improve Pi-class behavior for large media libraries.
Media browsing now requests ordered ID windows and hydrates only missing records instead of loading entire categories, months, or search result sets into frontend memory. The shared media manifest and ordering cache use bounded retention so large libraries do not become unbounded client-side arrays.
Upload handling now waits for rate-limit capacity instead of rejecting chunks mid-stream, which should reduce partial uploads and retry churn on slower storage. Layout refreshes are deferred while uploads are active to avoid unnecessary reload work.
Storage/media events are grouped or scoped where possible so deletes, hides/unhides, renames, uploads, and mount changes do not trigger excessive per-file refresh churn.