Unable to decode HEVC containing single frame

[limitz]

Hi,

I am using HEVC for a sparse set of frames, sometimes only a single frame. Encoding works fine, but it seems PyNvDecoder.DecodeSingleSurface doesn't return any frames if there is only 1 single frame in the file.
A comment in a sample says that with the built_in demuxer the function should be synchronized and flushing isn't necessary. Even so, I tried a flush but it still did not help.

I made a notebook where I tried to include the bare minimum to reproduce this error.

NvDecoderBug.pdf

[rarzumanyan]

Hi @limitz

WRT your notebook:

def encode(t:torch.Tensor, path:str) -> None:
    n,c,h,w = t.shape
    print(t.shape)
    assert t.dim() == 4
    
    encoded = np.ndarray(shape=(0), dtype=np.uint8)
    surface = nvc.Surface.Make(nvc.PixelFormat.NV12, w, h, 0)
    plane = surface.PlanePtr()
    written = 0

    with open(path, "wb") as stream:
        nvenc = nvc.PyNvEncoder({
        "preset":"P7",
        "tuning_info":"high_quality",
        "codec":"hevc",
        "rc":"constqp",
        "s": f"{w}x{h}"}, 0)

        for yuv in t:
            yuv = yuv.cuda()
            y,uv = yuv[[0]], TF.resize(yuv[[1,2]], (h//2, w//2))
            uv = uv.view(2,-1).permute(1,0).reshape(1,-1,w)
            nv12 = torch.cat((y, uv), dim=1)
            pnvc.TensorToDptr(nv12, plane.GpuMem(), plane.Width(), plane.Height(), 
                              plane.Pitch(), plane.ElemSize())
            ok = nvenc.EncodeSingleSurface(surface, encoded)

            if ok:
                stream.write(bytearray(encoded))
                written += 1
                print(f"Encoded frame {written}/{n}")
            
            while written < n:
                ok = nvenc.FlushSinglePacket(encoded)
                
                if ok:
                    stream.write(bytearray(encoded))
                    written += 1
                    print(f"Encoded frame {written}/{n} (flush)")

IDK what's the underlying pixel format of your input tensor. Torch tensors are usually RGB, are you sure its YUV420?
Also it would be helpful if you could share single frame video file.

[limitz]

Well, the underlying pixelformat is YUV444, 3 planes @ full resolution, which get transformed to NV12 by the code in the notebook. But it's just 3 channels of random data so it doesn't really matter, as long as the plane sizes are correct.

I have attached a single frame HEVC encoded by PyNvEncoder.
singleframe.zip

Thanks!

[rarzumanyan]

@limitz
Looks like something is wrong with your input frame:

How did you encode it?

[limitz]

no, sorry this confuses even more, I'm working on specific cameras and it's just a way to encode hyperspectral data. It's 5(+1 black) channels, Y,U,V, _,P0,P1 rearranged as 2 x 3 channel side by side, with a custom nonlinear color space encoded as NV12)

The point is though: I'm encoding an NV12 plane of size 3840x2160 (Y:3840x1440 + UV:3840x720) . It don't believe the content of the plane plays any part in not being able to decode it if it is the only frame in the video?

Please just refer to the notebook, it creates 2 hevc files, one with 2 frames and one with 1 frame. The single frame HEVC isn't properly decoded.

It actually decodes like this, after postprocessing of the raw NV12:

[rarzumanyan]

Thanks for the update @limitz

I confirm that I can repro this on my machine.
Now debugging PyNvDecoder, will update you later as I find something.

[rarzumanyan]

Hi @limitz

I've found the issue. The reason was bug in decoder reconfigure counter.
It caused false positive decoded frames queue flush which erased decoded frame. Please check out issue_375 feature branch.

P. S.
I've tested fix with python3 ./SampleDemuxDecode.py 0 singleframe.hevc singleframe.yuv command.