Skip to content

Tune variance partition thresholds in RTC - #5256

Merged
urvangjoshi merged 1 commit into
AOMediaCodec:av2-encfrom
jianj-g:rtc-vbp-tune-thresh
Sep 3, 2026
Merged

Tune variance partition thresholds in RTC#5256
urvangjoshi merged 1 commit into
AOMediaCodec:av2-encfrom
jianj-g:rtc-vbp-tune-thresh

Conversation

@jianj-g

@jianj-g jianj-g commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Comprehensive overhaul of variance-based partition (VBP) logic and
thresholds for real-time (RTC) encoding mode across resolutions
(low-res, 360p, VGA, 720p, 1080p+) and content types (camera video
and screen content). This only affects RTC mode with speed >= 6.

  1. Exponential quantizer modeling:
    Update threshold_base calculation to properly model AV2's exponential
    quantizer step size across QP, replacing the linear AC quantizer scale.

  2. Resolution-adaptive thresholds and high-QP scaling:
    Refine thresholds for <= 288p, 360p, 720p, and 1080p+, introducing
    smooth high-QP threshold slopes to avoid over-partitioning fine textures
    and UI elements at coarse quantizers.

  3. Sub-block variance spread evaluation & partition gating:

    • Evaluate sub-block variance spread in get_part_eval_based_on_sub_blk_var
      to avoid unnecessary 8x8 splits on flat/uniform areas while preserving
      splits for fine details.
    • Extend 32x32 sub-block variance difference checking to non-screen
      resolutions > 360p (VGA, 720p) guarded by max sub-block variance,
      avoiding trapped texture details in 32x32 blocks.
    • Guard 64x64 block splitting based on maximum sub-block variance.
  4. Screen content tuning (is_screen):

    • Guard 32x32 variance-diff force-split with !is_screen to retain large
      32x32 and 16x16 blocks needed for Palette Mode and IntraBC efficiency.
    • Tailor sub-block split thresholds and high-QP scaling specifically for
      screen content.

Benchmark results vs upstream baseline:

rtc (28 clips, VGA/720p camera video):
vmaf: -5.3% | ovr_psnr: -3.7% | speedup: +4.8%

rtc_derf (20 clips, <= 288p low-res):
vmaf: -3.6% | ovr_psnr: -2.1% | speedup: +1.1%

rtc_screen (9 clips, screen content):
vmaf: -8.4% | ovr_psnr: -3.3% | speedup: +2.2%

STATS_CHANGED for RTC mode

@jianj-g
jianj-g marked this pull request as draft August 12, 2026 19:08
@jianj-g
jianj-g requested a review from marpan-max August 12, 2026 19:09
@jianj-g
jianj-g force-pushed the rtc-vbp-tune-thresh branch 7 times, most recently from 7367c88 to f74f37b Compare August 31, 2026 17:04
@jianj-g
jianj-g marked this pull request as ready for review August 31, 2026 17:36
@jianj-g jianj-g changed the title Tune variance partition thresholds for low resolutions in RTC Tune variance partition thresholds in RTC Sep 1, 2026
@jianj-g
jianj-g requested a review from yunqingwang1 September 2, 2026 14:39
@jianj-g
jianj-g force-pushed the rtc-vbp-tune-thresh branch from f74f37b to 8bb4925 Compare September 3, 2026 15:20

@yunqingwang1 yunqingwang1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thanks.

Comprehensive overhaul of variance-based partition (VBP) logic and
thresholds for real-time (RTC) encoding mode across resolutions
(low-res, 360p, VGA, 720p, 1080p+) and content types (camera video
and screen content):

1. Exponential quantizer modeling:
   Update threshold_base calculation to properly model AV2's exponential
   quantizer step size across QP, replacing the linear AC quantizer scale.

2. Resolution-adaptive thresholds and high-QP scaling:
   Refine thresholds for <= 288p, 360p, 720p, and 1080p+, introducing
   smooth high-QP threshold slopes to avoid over-partitioning fine textures
   and UI elements at coarse quantizers.

3. Sub-block variance spread evaluation & partition gating:
   - Evaluate sub-block variance spread in get_part_eval_based_on_sub_blk_var
     to avoid unnecessary 8x8 splits on flat/uniform areas while preserving
     splits for fine details.
   - Extend 32x32 sub-block variance difference checking to non-screen
     resolutions > 360p (VGA, 720p) guarded by max sub-block variance,
     avoiding trapped texture details in 32x32 blocks.
   - Guard 64x64 block splitting based on maximum sub-block variance.

4. Screen content tuning (is_screen):
   - Guard 32x32 variance-diff force-split with !is_screen to retain large
     32x32 and 16x16 blocks needed for Palette Mode and IntraBC efficiency.
   - Tailor sub-block split thresholds and high-QP scaling specifically for
     screen content.

Benchmark results vs upstream baseline:

rtc (28 clips, VGA/720p camera video):
  vmaf: -5.3% | ovr_psnr: -3.7% | speedup: +4.8%

rtc_derf (20 clips, <= 288p low-res):
  vmaf: -3.6% | ovr_psnr: -2.1% | speedup: +1.1%

rtc_screen (9 clips, screen content):
  vmaf: -8.4% | ovr_psnr: -3.3% | speedup: +2.2%

STATS_CHANGED for RTC mode

TAG=agy
CONV=fe11822c-44c1-4487-85e3-edf6281da4a9
@jianj-g
jianj-g force-pushed the rtc-vbp-tune-thresh branch from 8bb4925 to ce4dfff Compare September 3, 2026 17:12
@urvangjoshi
urvangjoshi merged commit c7cf6dd into AOMediaCodec:av2-enc Sep 3, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants