Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add software tonemap filter support #12270

Merged
merged 4 commits into from
Jul 15, 2024
Merged

Conversation

gnattu
Copy link
Member

@gnattu gnattu commented Jul 14, 2024

This adds a fast SIMD optimized software tonemap filter support to the server. This filter currently does not support dolby vision reshaping and only support HDR10 and HLG inputs. With current veryfast libx264 settings we can expect 4K60 transcoding+tone mapping on a modern processor with 8 cores equivalent to Zen3/Golden Cove/Firestorm level performance.

Changes

  • Add tonemapx filter support to software transcode pipeline
  • Add tonemapx filter support to thumbnail generation pipeline

Issues

Depends on ffmpeg pr jellyfin/jellyfin-ffmpeg#407

The perf figures in the original post only reflects the initial implementation. The finalized perf figures are here: jellyfin/jellyfin-ffmpeg#407 (comment)

gnattu added 4 commits July 12, 2024 23:15
Signed-off-by: gnattu <gnattuoc@me.com>
Signed-off-by: gnattu <gnattuoc@me.com>
Signed-off-by: gnattu <gnattuoc@me.com>
Signed-off-by: gnattu <gnattuoc@me.com>
|| !options.EnableTonemapping
|| GetVideoColorBitDepth(state) != 10
|| !_mediaEncoder.SupportsFilter("tonemapx")
|| !(string.Equals(state.VideoStream?.ColorTransfer, "smpte2084", StringComparison.OrdinalIgnoreCase) || string.Equals(state.VideoStream?.ColorTransfer, "arib-std-b67", StringComparison.OrdinalIgnoreCase)))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these checks duplicated?

|| !(string.Equals(state.VideoStream?.ColorTransfer, "smpte2084", StringComparison.OrdinalIgnoreCase) || string.Equals(state.VideoStream?.ColorTransfer, "arib-std-b67", StringComparison.OrdinalIgnoreCase)))
            return state.VideoStream.VideoRange == VideoRange.HDR
                   && state.VideoStream.VideoRangeType is VideoRangeType.HDR10 or VideoRangeType.HLG or VideoRangeType.DOVIWithHDR10 or VideoRangeType.DOVIWithHLG;

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The former checks for all supported HDR curves and the latter filters out Dolby vision videos without fallback profiles. Only checks for the latter should be enough but we are doing this double check for hardware tone mapping as well.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

public (VideoRange VideoRange, VideoRangeType VideoRangeType) GetVideoColorRange()
{
if (Type != MediaStreamType.Video)
{
return (VideoRange.Unknown, VideoRangeType.Unknown);
}
var codecTag = CodecTag;
var dvProfile = DvProfile;
var rpuPresentFlag = RpuPresentFlag == 1;
var blPresentFlag = BlPresentFlag == 1;
var dvBlCompatId = DvBlSignalCompatibilityId;
var isDoViProfile = dvProfile == 5 || dvProfile == 7 || dvProfile == 8;
var isDoViFlag = rpuPresentFlag && blPresentFlag && (dvBlCompatId == 0 || dvBlCompatId == 1 || dvBlCompatId == 4 || dvBlCompatId == 2 || dvBlCompatId == 6);
if ((isDoViProfile && isDoViFlag)
|| string.Equals(codecTag, "dovi", StringComparison.OrdinalIgnoreCase)
|| string.Equals(codecTag, "dvh1", StringComparison.OrdinalIgnoreCase)
|| string.Equals(codecTag, "dvhe", StringComparison.OrdinalIgnoreCase)
|| string.Equals(codecTag, "dav1", StringComparison.OrdinalIgnoreCase))
{
return dvProfile switch
{
5 => (VideoRange.HDR, VideoRangeType.DOVI),
8 => dvBlCompatId switch
{
1 => (VideoRange.HDR, VideoRangeType.DOVIWithHDR10),
4 => (VideoRange.HDR, VideoRangeType.DOVIWithHLG),
2 => (VideoRange.SDR, VideoRangeType.DOVIWithSDR),
// While not in Dolby Spec, Profile 8 CCid 6 media are possible to create, and since CCid 6 stems from Bluray (Profile 7 originally) an HDR10 base layer is guaranteed to exist.
6 => (VideoRange.HDR, VideoRangeType.DOVIWithHDR10),
// There is no other case to handle here as per Dolby Spec. Default case included for completeness and linting purposes
_ => (VideoRange.SDR, VideoRangeType.SDR)
},
7 => (VideoRange.HDR, VideoRangeType.HDR10),
_ => (VideoRange.SDR, VideoRangeType.SDR)
};
}
var colorTransfer = ColorTransfer;
if (string.Equals(colorTransfer, "smpte2084", StringComparison.OrdinalIgnoreCase))
{
return (VideoRange.HDR, VideoRangeType.HDR10);
}
else if (string.Equals(colorTransfer, "arib-std-b67", StringComparison.OrdinalIgnoreCase))
{
return (VideoRange.HDR, VideoRangeType.HLG);
}
return (VideoRange.SDR, VideoRangeType.SDR);
}

ColorTransfer is normalized to VideoRange and VideoRangeType. We should eventually avoid using ColorTransfer directly.

Copy link
Member

@nyanmisaka nyanmisaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Speaking of Golden Cove, let's hope that the flagship models in Raptor Lake don't crash in our AVX2 code.

@thornbill thornbill added feature Adding a new feature, or substantial improvements on existing functionality release-highlight A major new feature for the next release labels Jul 14, 2024
@crobibero crobibero merged commit 1eb2b6e into jellyfin:master Jul 15, 2024
18 checks passed
@gnattu gnattu deleted the sw-tonemap branch July 17, 2024 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Adding a new feature, or substantial improvements on existing functionality release-highlight A major new feature for the next release
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

6 participants