Subtitles get out of sync when resuming a movie, or the audio/video is out of sync with the subtitles.

[MrRoza]

Describe the bug

Reproduce steps:

1. Watch a movie until like 50% of it.
2. Notice that the subtitles are (probably) in sync.
3. Shut down jellyfin app.
4. Start Jellyfin again, and try to resume the movie.
5. Notice that the subtitles are out of sync. However this doesn't happen at every movie.

I also noticed something else odd, which could explain why this bug happens.

If I go to timestamp 1:00:00 on the androidtv client (chromecast), and I go to 1:00:00 on the Windows 10 client, it seems like the androidtv client is like 5-10 seconds behind the Windows 10 client. The subtitles are 100% exactly on the same timestamp, so those appear at the identical time if you sync the two clients up, but the video and audio on the androidtv client seems to be 5-10 seconds behind, which could explain the subtitle sync issue.
Long story short, it looks like the audio/video get's out of sync with the time, but the subtitles are actually correct.

For me this is a big issue since I have trouble hearing voices with background noise, especially if it's not my native tongue.

If any more info is needed, I am able to (privately) provide a test-account to my JF server to a developer, with sftp access to my library.

Logs

[2024-04-08 13:03:05.430 +02:00] [INF] [74] Jellyfin.Api.Helpers.MediaInfoHelper: User policy for "joost". EnablePlaybackRemuxing: True EnableVideoPlaybackTranscoding: True EnableAudioPlaybackTranscoding: True
[2024-04-08 13:03:05.431 +02:00] [INF] [74] Jellyfin.Api.Helpers.MediaInfoHelper: StreamBuilder.BuildVideoItem( Profile="AndroidTV-libVLC", Path="/data/media/movies/Harry Potter and the Deathly Hallows Part 2 (2011)/Harry.Potter.and.the.Deathly.Hallows.Part.2.2011.1080p.10bit.BluRay.6CH.x265.HEVC-PSA.mkv", AudioStreamIndex=null, SubtitleStreamIndex=null ) => ( PlayMethod=DirectPlay, TranscodeReason=0 ) "media:/videos/7a35d69c-9514-7a12-6085-859f3e9f0d8b/stream.mkv?MediaSourceId=7a35d69c95147a126085859f3e9f0d8b&Static=true&VideoCodec=hevc&AudioCodec=aac&AudioStreamIndex=2&api_key=<token>&Tag=bca4acb4a71f9ec7e60db48d1994d1f6"
[2024-04-08 13:03:05.581 +02:00] [INF] [36] Jellyfin.Api.Helpers.MediaInfoHelper: User policy for "joost". EnablePlaybackRemuxing: True EnableVideoPlaybackTranscoding: True EnableAudioPlaybackTranscoding: True
[2024-04-08 13:03:05.581 +02:00] [INF] [36] Jellyfin.Api.Helpers.MediaInfoHelper: StreamBuilder.BuildVideoItem( Profile="AndroidTV-ExoPlayer", Path="/data/media/movies/Harry Potter and the Deathly Hallows Part 2 (2011)/Harry.Potter.and.the.Deathly.Hallows.Part.2.2011.1080p.10bit.BluRay.6CH.x265.HEVC-PSA.mkv", AudioStreamIndex=null, SubtitleStreamIndex=null ) => ( PlayMethod=Transcode, TranscodeReason=AudioChannelsNotSupported ) "media:/videos/7a35d69c-9514-7a12-6085-859f3e9f0d8b/master.m3u8?MediaSourceId=7a35d69c95147a126085859f3e9f0d8b&VideoCodec=hevc,h264,hevc&AudioCodec=aac&AudioStreamIndex=2&VideoBitrate=99616000&AudioBitrate=384000&AudioSampleRate=48000&MaxFramerate=23.976025&api_key=<token>&RequireAvc=false&Tag=bca4acb4a71f9ec7e60db48d1994d1f6&SegmentContainer=ts&BreakOnNonKeyFrames=False&hevc-level=120&hevc-videobitdepth=10&hevc-profile=main10&hevc-audiochannels=6&aac-profile=he-aac&aac-audiochannels=2&TranscodeReasons=AudioChannelsNotSupported"
[2024-04-08 13:03:05.955 +02:00] [INF] [36] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Adding playback tracker : 2e352282e0d23b46b66a01bd96896342f814fbbe-342c3b788c2e49dfb6226644434d66ef-7a35d69c95147a126085859f3e9f0d8b
[2024-04-08 13:03:05.955 +02:00] [INF] [36] Jellyfin.Plugin.PlaybackReporting.Data.PlaybackTracker: PlaybackTracker : Adding Start Event : 04/08/2024 13:03:05
[2024-04-08 13:03:05.955 +02:00] [INF] [36] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Creating StartPlaybackTimer Task
[2024-04-08 13:03:05.955 +02:00] [INF] [36] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : Entered
[2024-04-08 13:03:06.836 +02:00] [INF] [73] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Processing playback tracker : "2e352282e0d23b46b66a01bd96896342f814fbbe-342c3b788c2e49dfb6226644434d66ef-7a35d69c95147a126085859f3e9f0d8b"
[2024-04-08 13:03:07.946 +02:00] [INF] [56] Jellyfin.Api.Helpers.TranscodingJobHelper: "/usr/lib/jellyfin-ffmpeg/ffmpeg" "-analyzeduration 200M -fflags +genpts  -i file:\"/data/media/movies/Harry Potter and the Deathly Hallows Part 2 (2011)/Harry.Potter.and.the.Deathly.Hallows.Part.2.2011.1080p.10bit.BluRay.6CH.x265.HEVC-PSA.mkv\" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 copy -tag:v:0 hvc1 -bsf:v hevc_mp4toannexb -start_at_zero -codec:a:0 libfdk_aac -ac 2 -ab 384000 -ar 48000 -af \"volume=2\" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 6 -hls_segment_type mpegts -start_number 0 -hls_segment_filename \"/config/data/transcodes/b0c2898c3a5c96e5cb336ce3ed084331%d.ts\" -hls_playlist_type vod -hls_list_size 0 -y \"/config/data/transcodes/b0c2898c3a5c96e5cb336ce3ed084331.m3u8\""
[2024-04-08 13:03:09.122 +02:00] [INF] [69] Jellyfin.Api.Helpers.TranscodingJobHelper: Stopping ffmpeg process with q command for "/config/data/transcodes/b0c2898c3a5c96e5cb336ce3ed084331.m3u8"
[2024-04-08 13:03:09.189 +02:00] [INF] [69] Jellyfin.Api.Helpers.TranscodingJobHelper: FFmpeg exited with code 0
[2024-04-08 13:03:09.190 +02:00] [INF] [69] Jellyfin.Api.Helpers.TranscodingJobHelper: "/usr/lib/jellyfin-ffmpeg/ffmpeg" "-analyzeduration 200M -ss 01:18:16.567 -fflags +genpts  -i file:\"/data/media/movies/Harry Potter and the Deathly Hallows Part 2 (2011)/Harry.Potter.and.the.Deathly.Hallows.Part.2.2011.1080p.10bit.BluRay.6CH.x265.HEVC-PSA.mkv\" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 copy -tag:v:0 hvc1 -bsf:v hevc_mp4toannexb -start_at_zero -codec:a:0 libfdk_aac -ac 2 -ab 384000 -ar 48000 -af \"volume=2\" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 6 -hls_segment_type mpegts -start_number 782 -hls_segment_filename \"/config/data/transcodes/b0c2898c3a5c96e5cb336ce3ed084331%d.ts\" -hls_playlist_type vod -hls_list_size 0 -y \"/config/data/transcodes/b0c2898c3a5c96e5cb336ce3ed084331.m3u8\""
[2024-04-08 13:03:25.954 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: session.RemoteEndPoint : "192.168.1.254"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : event_playing_id     = "7a35d69c95147a126085859f3e9f0d8b"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : event_user_id        = "342c3b788c2e49dfb6226644434d66ef"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : event_user_id_int    = 1
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : session_playing_id   = "7a35d69c95147a126085859f3e9f0d8b"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : session_user_id      = "342c3b788c2e49dfb6226644434d66ef"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : play_method          = "Transcode (v:direct a:aac)"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : e.ClientName         = "Android TV"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : e.DeviceName         = "Joost"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : ItemName             = "Harry Potter and the Deathly Hallows: Part 2"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : ItemId               = "7a35d69c95147a126085859f3e9f0d8b"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : ItemType             = "Movie"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : All matches, playback registered
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Playback tracker found, adding playback info : "2e352282e0d23b46b66a01bd96896342f814fbbe-342c3b788c2e49dfb6226644434d66ef-7a35d69c95147a126085859f3e9f0d8b"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Saving playback tracking activity in DB
[2024-04-08 13:03:25.977 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : Exited
[2024-04-08 13:03:27.078 +02:00] [INF] [36] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Processing playback tracker : "2e352282e0d23b46b66a01bd96896342f814fbbe-342c3b788c2e49dfb6226644434d66ef-7a35d69c95147a126085859f3e9f0d8b"
[2024-04-08 13:03:49.305 +02:00] [INF] [73] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Processing playback tracker : "2e352282e0d23b46b66a01bd96896342f814fbbe-342c3b788c2e49dfb6226644434d66ef-7a35d69c95147a126085859f3e9f0d8b"

Application version

jellfyin-androidtv 0.16.7 release

Where did you install the app from?

Google Play

Device information

Chromecast HD

Android version

Android TV 12 (STTL.240206.002)

Jellyfin server version

10.8.13 (docker)

[VampiricAlien]

Any chance the media is being transcoded?

[MrRoza]

Iirc it was a direct stream. But I'll be double checking later today and report back.

…

________________________________ From: VampiricAlien ***@***.***> Sent: Tuesday, April 9, 2024 2:51:12 PM To: jellyfin/jellyfin-androidtv ***@***.***> Cc: Joost R. ***@***.***>; Author ***@***.***> Subject: Re: [jellyfin/jellyfin-androidtv] Subtitles get out of sync when resuming a movie, or the audio/video is out of sync with the subtitles. (Issue #3492) Any chance the media is being transcoded? — Reply to this email directly, view it on GitHub<#3492 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFAMG55BKC2YSBVHNXHAFE3Y4PP4BAVCNFSM6AAAAABF4PM5KCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBVGEYTANBZG4>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[MrRoza]

Sorry, something came up yesterday.
Anyways, it isn't being transcoded:

However, I tried to force transcoding by putting the quality to 1Mbit/s, and this "fixes" the sync issues.
"fixes" in quotes because it makes the movie look bad obviously.

Is there any other way to force encoding, so I can temp-fix this until a proper fix is out?

[VampiricAlien]

In playback settings, Hardware encoding options might be a temp fix. Your CPU supports HEVC encoding.

[MrRoza]

Hi VampiricAlien,

Thanks for the suggestion, however it did not temp-fix my issue.
Are there other ways to force transcoding without losing resolution / on higher bitrates?

In case it is important, my setup is like this:
i5 9500T running Proxmox
Proxmox has a Debian VM with iGPU pass-through
Debian vm runs Docker with GPU device mapping

And this are my JF settings for hardware decoding

[VampiricAlien]

In the app, menu, playback, under video there should be max streaming bitrate, set that to highest but don't select auto.

Can you post the video details such as:
Video Title: 4K - HEVC - SDR Codec: HEVC Profile: Main 10 Level: 156 Resolution: 3840x2160 Aspect ratio: 16:9 Interlaced: No Framerate: 23.976025 Bitrate: 6072 kbps Bit depth: 10 bit Video range: SDR Video range type: SDR Color space: bt709 Color transfer: bt709 Color primaries: bt709 Pixel format: yuv420p10le Ref frames: 1
That's just an example.
and also the device the client is using if you can.

[MrRoza]

I've set it manually to 120mbps, but it's still direct stream and out of sync

I didn't know how to get that video detail in the format how you have it, I hope this screenshot is enough:

If it's not, please let me know how I can get more information.

edit: Just noticed the last property is cut off in the last screenshot. It was "Ref frames" with value "1"

[VampiricAlien]

Silly question but are you sure the subtitle are correct? Have you tested on Jellyfin web, VLC, android client?

[MrRoza]

Yes, on my phone and on my pc using the JellyFin windows client the subtitles are correct.

[VampiricAlien]

Have you looked at the option Allow subtitle extraction on the fly under playback?

[MrRoza]

I haven't seen that option anywhere yet, but I'll see if I can find it tomorrow, and report back. Thanks for the suggestion.

[VampiricAlien]

You'll find the option in Jellyfin web.

[MrRoza]

I just enabled it, rebooted JF and tried it again without luck.
I think what I said in the original post is what is actually happening, which is the video/audio getting out of sync, but the subtitles actually being "correct".

windows:

Chromecast:

Notice that both frames are at 35:30, have a different scene, but the exact same subtitles. This confirms for me that the subtitles are actually correct, but the video itself is somewhere else than JF expects, resulting in a sync issue between video and subtitles.

[VampiricAlien]

My last suggest would be to create a new audio track.
ffmpeg -i inputfile.mkv -map 0:a:0 -ac 2 outputfile.aac
ffmpeg -i input.mp4 -i outputfile.aac -map 0:v -map 0:s -map 1:a -c:v copy -c:s copy -c:a aac -strict experimental movie.mp4
Something like that or you can use a program to create a new 2 channel audio track.
This doesn't fix the issue with the app but does better help you.