PS4 Media Player does not list files with uncompatible video or audio format

[Eniff]

Describe the bug
Playstation 4 Media Player over DLNA can't see any files which have unsupported video or audio codec. The only media that shows up are natively supported formats. This way transcoding over DLNA is impossible to select.

System (please complete the following information):

• OS: Windows
• Clients: PS4 Media Player DLNA
• Jellyfin Version: 10.7.7
• Playback: Direct Play, Remux, Direct Stream, Transcode
• Hardware Acceleration: NVENC
• Installed Plugins: none
• Reverse Proxy: none
• Storage: local

To Reproduce

1. Open media player on PS4 and navigate to Jellyfin media server
2. Can't find any videos that need remuxing, transcoding(video, audio or both) to play on PS4, only H264-AC3 etc... files appear which the PS4 natively supports.

Expected behavior
When the media library has a file that consists of video or audio format that the PS4 cannot play it should still show up in the PS4 Media player and it should transcode and/or remux the format which is unsupported into one that works. If for example only the audio is unsupported it should only encode that and copy the video.

Additional context
I have recently switched from UMS to Jellyfin and I really like the fact that you can easily use Nvenc to re-encode media, but sadly none of the videos that acutally need transcoding appear in the PS4 media player application so this bug made this application useless for me. I have tried playing around with the settings but nothing seems to work. In UMS for example you can select every trancoding option from the PS4 media player and everything shows up. But that app has other problems which made me look for alternatives. I was just wanting to let you know this bug still exists.

[jellyfin-bot]

This issue has gone 120 days without comment. To avoid abandoned issues, it will be closed in 21 days if there are no new comments.

If you're the original submitter of this issue, please comment confirming if this issue still affects you in the latest release or master branch, or close the issue if it has been fixed. If you're another user also affected by this bug, please comment confirming so. Either action will remove the stale label.

This bot exists to prevent issues from becoming stale and forgotten. Jellyfin is always moving forward, and bugs are often fixed as side effects of other changes. We therefore ask that bug report authors remain vigilant about their issues to ensure they are closed if fixed, or re-confirmed - perhaps with fresh logs or reproduction examples - regularly. If you have any questions you can reach us on Matrix or Social Media.

[FarFetchd]

I'm also affected. My PS4 can play some things from Jellyfin, but others (looks like definitely all H265 and avi) don't even show up in their directories.

I first found the issue jellyfin/jellyfin#4806, but it's not clear if their guess that the TS isn't supported by PS4 turned out to be true; sounds like the audio codec thing also linked there might have been the real problem.

Unfortunately I can't figure out how to play around with this on my own: I found /etc/jellyfin/dlna/system/Sony PlayStation 4.xml, but when I edit it (to change transcode container from ts to mp4) and systemctl restart jellyfin, the file just gets reverted back to its original state - so I guess it's not the actual source of truth on this stuff.

Is there an easy way for me to play around with the codec and container settings for this profile, ideally without having to compile Jellyfin myself?

[FarFetchd]

Looking closer at the xml file and the videos I'm having trouble with, it seems that I'm running into problems both with ones that ought to be directly playable and ones that need transcoding.

At least one not-seen file is of a type that's listed (IIUC) as able to be directly played, i.e. has a DirectPlayProfile entry. The file is an avi container, mp4 video ("MSMPEG4V3"), mp3 audio.

On the other hand, another not-seen file is mkv container, h265/HEVC video, AAC audio. That combination does not match the DirectPlayProfile entries. Also, when I enter its directory on the PS4, systemctl status jellyfin shows messages that look like:
StreamBuilder.BuildVideoItem( Profile=Sony PlayStation 4, Path=... AudioStreamIndex=null, SubtitleStreamIndex=null ) => ( PlayMethod=Transcode, TranscodeReason=VideoCodecNotSupported ) media:/videos/(someuuid)/stream.ts?MediaSourceId=(someuuid)&VideoCodec=h264&AudioCodec=aac,ac3,mp2&AudioStreamIndex=1&VideoBitrate=139904000&AudioBitrate=96000&MaxFramerate=25&api_key=<token>&RequireAvc=false&Tag=somehexnumber&msmpeg4v3-level=-99&msmpeg4v3-videobitdepth=8&TranscodeReasons=VideoCodecNotSupported
...so it looks like Jellyfin knew to kick off transcoding the first chunk of each of the files that might be about to be played.
(Maybe worth noting: that api_key=<token> is literally that. But I assume that's just a placeholder used when you're on your own local machine and don't need to show a key to someone.)

mp4 or mkv container, mp4 video, and AAC audio does work.

All that leads me to think that there are some codecs/containers that it was mistakenly thought PS4 supports, and they are used both in some DirectPlayProfile entries (affecting matching files) as well as in the video TranscodingProfile (affecting ALL videos that would need to be transcoded). I would like to see what would happen if TranscodingProfile was changed to H264+AAC contained in mp4, and the avi DirectPlayProfile was deleted.

[FarFetchd]

I recompiled the DLNA dll to use an updated profile for the PS4. I switched the video transcoding target to mkv container (also tried mp4, same results), h264 video, aac audio. Here's where it gets weird: I found that if ffmpeg could finish before the transcoding kill timer killed it (like 15 or 20 seconds), then the video could start playing, otherwise it would just say "Cannot play the media". Unfortunately, for the video I tried, the PS4 client would then freeze (like, crash) several seconds into the playback, always at the same point.

I tried manually running Jellyfin's exact ffmpeg transcode command to get a pre-transcoded version to feed to the PS4. Jellyfin still had to remux that (the log file was named FFmpeg.Remux rather than .Transcode) and it went back to "cannot play the media", rather than the few seconds of playback.

Also worth noting: this change allowed the files to show up in the list at all. So it does seem like the ts container, or maybe ts with the particular codecs used, is a problem for PS4.

But anyways, I'm stopping here. I get the impression the PS4 media client is very particular and fragile: I'm having Jellyfin transcode to formats that AFAICT are the same as what the PS4 can play natively (and, sanity check, my laptop can get Jellyfin's web client to play videos that Jellyfin has to transcode, so it's not a problem with my Jellyfin). Maybe someone with deeper understanding of media codec/protocol stuff could find a workaround.