Couple of issues in RDO

[Jovasa]

Hello,
I've been using VVenC as a reference while fixing issues in RDO of uvg266. I've also managed to spot couple of potential issues in VVenC

1. If transform skip is disabled the following the cabac context is not reset properly during intra chroma search, which cause minor degradation when LFNST is enabled, at least on my cursory tests.

   vvenc/source/Lib/EncoderLib/IntraSearch.cpp

   Lines 917 to 920 in a361f59

   	if (cs.sps->transformSkip)
   	{
   	m_CABACEstimator->getCtx() = ctxStart;
   	}

2. The contexts for split flags are updated only after the split search is performed.
3. When transform skip is disabled, the following check is true for MTS_DST7_DST7 when it probably shouldn't.

   vvenc/source/Lib/CommonLib/TrQuant.cpp

   Line 771 in a361f59

   const bool testTr = itC->first <= (itC->second == 1 ? thrTS : thr) && numTests <= maxCand;

   Also just wondering, shouldn't the modes be sorted according to the cost instead of just disabling the last mode(s) when all of the modes fulfill the cost threshold?
4. And related to the previous issue; --MTSIntraMaxCand=4 segfaults.

Though overall I did find many more issues in uvg266 😅

[adamjw24]

Thanks, we'll look into that, especially the SegFault issue.

About 2., that is a behavior ported from VTM already. I remember doing this test with some early version of VTM or maybe even JEM without much improvement. But we'll try that again, after all the context derivation has changed since than.

[Jovasa]

Yeah I suspect the issue with 2. is that because the MTT prohibits certain conditions it's impossible to know the actual splitting pattern before the whole CU is searched.
Also there was one other issue I forgot to mention originally.

vvenc/source/Lib/EncoderLib/IntraSearch.cpp

Line 1735 in a361f59

cuCtx.mtsLastScanPos = false;

violatesMtsCoeffConstraint should also be reset here.

[Jamaika1]

Compatibility of codecs from different developers.
uvg266e 0.2.3-5ea9132
vvdec 1.5.0-c9fb1c5

uvg266e.exe -i "113.yuv" -o "output_10bit.266" --no-info --input-bitdepth 8 --threads 4 --input-res 1280x720 --input-fps 30000/1001 --input-format P420 --gop=16 --ref=4 --bipred --gop lp-g16d4t1 -p 256 --bitrate 3000000 -n 200 --preset medium
VVDecoderApp.exe -b output_10bit.266 -o yyy.yuv -t 4

Fraunhofer VVC/H.266 Decoder VVdeC, version 1.5.0 [THREADS=4; PARSE_DELAY=4; SIMD=SCALAR]
POC    0 LId:  0 TId: 0 ( I-SLICE, QP 20 ) [DT  0.014] [L0 ] [L1 ]
POC   16 LId:  0 TId: 0 ( P-SLICE, QP 20 ) [DT  0.048] [L0  0c] [L1 ]
POC    8 LId:  0 TId: 0 ( B-SLICE, QP 27 ) [DT  0.025] [L0  0c] [L1 16 ]
POC    4 LId:  0 TId: 0 ( B-SLICE, QP 23 ) [DT  0.026] [L0  0c] [L1 8 16 ]
POC    2 LId:  0 TId: 0 ( B-SLICE, QP 34 ) [DT  0.015] [L0  0c] [L1 4 8 16 ]
POC    1 LId:  0 TId: 0 ( B-SLICE, QP 29 ) [DT  0.020] [L0  0c] [L1 2 4 8 16 ]
POC    6 LId:  0 TId: 0 ( B-SLICE, QP 20 ) [DT  0.021] [L0  4c0 ] [L1 8 16 ]
POC    3 LId:  0 TId: 0 ( B-SLICE, QP 26 ) [DT  0.020] [L0  2c0 ] [L1 4 8 16 ]
POC    5 LId:  0 TId: 0 ( B-SLICE, QP 24 ) [DT  0.017] [L0  4c0 ] [L1 6 8 16 ]
POC    7 LId:  0 TId: 0 ( B-SLICE, QP 24 ) [DT  0.024] [L0  6c4 0 ] [L1 8 16 ]
POC   12 LId:  0 TId: 0 ( B-SLICE, QP 19 ) [DT  0.030] [L0  8c0 ] [L1 16 ]
POC   10 LId:  0 TId: 0 ( B-SLICE, QP 20 ) [DT  0.016] [L0  8c0 ] [L1 12 16 ]
POC    9 LId:  0 TId: 0 ( b-SLICE, QP 23 ) [DT  0.014] [L0  8c0 ] [L1 10 12 16 ]
POC   11 LId:  0 TId: 0 ( b-SLICE, QP 20 ) [DT  0.019] [L0  10c8 0 ] [L1 12 16 ]
POC   14 LId:  0 TId: 0 ( B-SLICE, QP 20 ) [DT  0.019] [L0  12c8 0 ] [L1 16 ]
vvdecapp [error]: decoding failed:  (decoder requires restart) detail: caught unknown exception
ERROR: In function "void vvdec::DecSlice::parseSlice(vvdec::Slice*, vvdec::InputBitstream*, int)" in DecSlice.cpp:179: Expecting a terminating bit
ERROR CONDITION: !binVal

VVCSoftware: VTM Decoder Version 16.2 [Windows][GCC 13.0.0][64 bit]
POC    0 LId:  0 TId: 0 ( IDR_N_LP, I-SLICE, QP 20 ) [DT  0.047] [L0] [L1] [Checksum:07036092,01cbdb60,01cd3e00,(OK)]
POC   16 LId:  0 TId: 0 ( TRAIL, P-SLICE, QP 20 ) [DT  0.078] [L0 0c] [L1] [Checksum:06fd03a4,01b810f2,01d9f2a9,(OK)]
POC    8 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 27 ) [DT  0.047] [L0 0c] [L1 16] [Checksum:0703025b,01bedb94,01ded136,(OK)]
POC    4 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 23 ) [DT  0.067] [L0 0c] [L1 8 16] [Checksum:07032fd8,01c60315,01d7059e,(OK)]
POC    2 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 34 ) [DT  0.031] [L0 0c] [L1 4 8 16] [Checksum:0701feed,01c788e5,01cce173,(OK)]
POC    1 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 29 ) [DT  0.031] [L0 0c] [L1 2 4 8 16] [Checksum:0703ade6,01caf364,01cd3dec,(OK)]
POC    3 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 26 ) [DT  0.047] [L0 2c 0] [L1 4 8 16] [Checksum:0703cec2,01c785c2,01d8ed72,(OK)]
POC    6 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 20 ) [DT  0.062] [L0 4c 0] [L1 8 16] [Checksum:0704bebd,01c36752,01deeb63,(OK)]
POC    5 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 24 ) [DT  0.031] [L0 4c 0] [L1 6 8 16] [Checksum:070268e5,01c64c84,01d8d861,(OK)]
POC    7 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 24 ) [DT  0.047] [L0 6c 4 0] [L1 8 16] [Checksum:07017439,01be99a4,01e3fe0d,(OK)]
POC   12 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 19 ) [DT  0.063] [L0 8c 0] [L1 16] [Checksum:0701b4fd,01b76ab0,01dd6bad,(***ERROR***)] [rxChecksum:0701b3d5,01b770c8,01dd7e39]
POC   10 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 20 ) [DT  0.063] [L0 8c 0] [L1 12 16] [Checksum:07055dcb,01bbc4c3,01ddba4e,(***ERROR***)] [rxChecksum:07055d28,01bbc986,01ddc73b]
POC    9 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 23 ) [DT  0.062] [L0 8c 0] [L1 10 12 16] [Checksum:07054d2b,01ba28da,01dcfa0d,(***ERROR***)] [rxChecksum:07054bec,01ba2c4e,01dd0371]
POC   11 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 20 ) [DT  0.078] [L0 10c 8 0] [L1 12 16] [Checksum:0705e90a,01b67b66,01dbcd29,(***ERROR***)] [rxChecksum:0705e20f,01b6803f,01dbd6ff]
POC   14 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 20 ) [DT  0.063] [L0 12c 8 0] [L1 16] [Checksum:07020553,01b8396f,01db7d04,(***ERROR***)] [rxChecksum:070201f4,01b83d73,01db8a1d]
POC   13 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 22 ) [DT  0.046] [L0 12c 8 0] [L1 14 16] [Checksum:0700430a,01b70dd0,01dcbe5d,(***ERROR***)] [rxChecksum:07003d89,01b71417,01dccb57]
POC   15 LId:  0 TId: 0 ( TRAIL, B-SLICE, QP 21 ) [DT  0.046] [L0 14c 12 8 0] [L1 16] [Checksum:06ff8988,01b72702,01dba108,(***ERROR***)] [rxChecksum:06ff9aed,01b72737,01dbb57f]

ERROR: In function "decompressSlice" in DecSlice.cpp:259: Expecting a terminating bit

[adamjw24]

Compatibility of codecs from different developers. uvg266e 0.2.3-5ea9132 vvdec 1.5.0-c9fb1c5

I will give you some advice here, please follow it. If you encoded stuff with uvg266 and it doesn't decode with vvdec nor VTM, its a bug in uvg266 and should be posted there. If it decodes with VTM and doesn't with vvdec, its a bug with vvdec and should be reported in the vvdec repo. None of those cases should be reported in this repo. Enc/Dec for international standards don't have compatibilities. If a pair is incompatible, one of those has a bug.