Encoding surface throws CUDA_ERROR_ILLEGAL_ADDRESS

[anklebreaker]

Is it possible to encode surface without downloading or uploading a frame? My current pipeline is Decoder -> Color Conversion -> Surface Remaper -> Color Conversion -> Encoder. However, with everything I've tried, I run into an illegal memory access error on the encoder (and sometimes in the second color conversion). I tried passing in the result of the surface remaper directly. The samples only show decoding/remaping then downloading or reading from file, uploading, and encoding. Is a CPU step necessary?

[gedoensmax]

The sample is much too complicated, but e.g. cvcuda is doing a full gpu pipeline. https://github.com/CVCUDA/CV-CUDA/blob/release_v0.2.x/samples/segmentation/python/inference.py
Also see their vpf utils here: https://github.com/CVCUDA/CV-CUDA/blob/release_v0.2.x/samples/common/python/vpf_utils.py

The segmentations faults you are seeing sound buggy. Can you share a minimal sample ?

[anklebreaker]

Thanks for the reference and your response.

Below replicates the error for me:

import sys
sys.path.insert(0, "/home/ubuntu/VideoProcessingFramework/install")
import numpy as np
import PyNvCodec as nvc

def remap_func(im_surface):
    #1080p map
    map_x, map_y = np.meshgrid(np.arange(1920, dtype=np.float32), np.arange(1080, dtype=np.float32))

    #remap
    nv_remap = nvc.PySurfaceRemaper(map_x, map_y, nvc.PixelFormat.RGB, 0)
    remapped = nv_remap.Execute(cvtSurface_rgb)
    print('Remapped')
    return remapped

#intialize decode
filepath = 'test.mov' #4k video
_decoder = nvc.PyNvDecoder(filepath, 0)
_to_yuv420 = nvc.PySurfaceConverter(3840, 2160, _decoder.Format(), nvc.PixelFormat.YUV420, 0)
yuv420_rgb = nvc.PySurfaceConverter(3840, 2160, nvc.PixelFormat.YUV420, nvc.PixelFormat.RGB, 0)
cc_to_yuv420 = nvc.ColorspaceConversionContext(nvc.ColorSpace.BT_601, nvc.ColorRange.MPEG)
cc_yuv420_rgb = nvc.ColorspaceConversionContext(nvc.ColorSpace.BT_601, nvc.ColorRange.MPEG)

#initialize encode
codec = "hevc"
bitrate = "4000000"
config = {"preset": "ll_hq", "codec": codec, "s": "1920x1080", "bitrate": bitrate, "gop": "60", "fps": "30"}
_encoder = nvc.PyNvEncoder(config, 0)
_encode_file = open('test_out.h265', "wb")
_rgb_to_yuv420 = nvc.PySurfaceConverter(1920, 1080, nvc.PixelFormat.RGB, nvc.PixelFormat.YUV420, 0)
_yuv420_to_nv12 = nvc.PySurfaceConverter(1920, 1080, nvc.PixelFormat.YUV420, nvc.PixelFormat.NV12, 0)
cc_rgb_yuv420 = nvc.ColorspaceConversionContext(nvc.ColorSpace.BT_601, nvc.ColorRange.MPEG)
cc_yuv420_nv12 = nvc.ColorspaceConversionContext(nvc.ColorSpace.BT_601, nvc.ColorRange.MPEG)

for i in range(1000):
    #Decode
    rawSurface = _decoder.DecodeSingleSurface()
    print('Decoded')
    
    #Convert to RGB
    cvtSurface_yuv420 = _to_yuv420.Execute(rawSurface, cc_to_yuv420)
    cvtSurface_rgb = yuv420_rgb.Execute(cvtSurface_yuv420, cc_yuv420_rgb)

    remapped = remap_func(cvtSurface_rgb)

    #convert to nv12
    yuv420_surface = _rgb_to_yuv420.Execute(remapped, cc_rgb_yuv420)
    nv12_surface = _yuv420_to_nv12.Execute(yuv420_surface, cc_yuv420_nv12)
    
    #encode
    encoded_frame = np.ndarray(shape=(0), dtype=np.uint8)
    success = _encoder.EncodeSingleSurface(nv12_surface, encoded_frame)
    if success:
        bits = bytearray(encoded_frame)
        _encode_file.write(bits)

Interestingly, this code works if I take the remap portion out of the function and paste it directly in the main loop. Let me know if there's something I'm doing wrong with respect to memory access here. Curious why it would pass the initial surface in fine but not be able to encode the surface passed back. When I add a download and upload step before the encode, it works again with correct output. The code you linked to seems to do ok with passing objects to functions, but it similarly adds an intermediate step to convert from Tensor before encoding.

[gedoensmax]

This sounds to me a lot like a CPU/GPU synchronization issue. First of all it makes most sense to hold on to nv_remap = nvc.PySurfaceRemaper(map_x, map_y, nvc.PixelFormat.RGB, 0) instead of reinstantiating it as it will allocate a buffer und upload it.
PySurfaceRemaper instantiates a RemapSurface:

VideoProcessingFramework/src/TC/src/Tasks.cpp

Lines 1506 to 1517 in 6fc8aff

	RemapSurface_Impl(const float* x_map, const float* y_map, uint32_t remap_w,
	uint32_t remap_h, Pixel_Format format, CUcontext ctx,
	CUstream str)
	: cu_ctx(ctx), cu_str(str), map_w(remap_w), map_h(remap_h)
	{
	xMapPlane = CudaBuffer::Make((const void*)x_map, sizeof(float),
	map_w * map_h, ctx, str);
	yMapPlane = CudaBuffer::Make((const void*)y_map, sizeof(float),
	map_w * map_h, ctx, str);
	SetupNppContext(cu_ctx, cu_str, nppCtx);
	}

I agree that this should not SEGFAULT in an ideal world but it sounds to me like nv_remap is destroyed before it is no longer needed. Do you need a dynamic meshgrid or is this really just subsampling ? In that case i would actually recommend moving away from VPF Taks and prefer CVCUDA ops over that.

[anklebreaker]

My warping is dynamic and changes frame to frame, and the codebase for generating that warp is in other methods, hence the calling of remap outside the main loop. I could try and reorganize the code to make the remap happen in the main loop body, but would take some effort.

Is there a more efficient way than reinstantiating a Remaper instance every frame?

[gedoensmax]

I mean you could use torch.

[anklebreaker]

Unfortunately grid_sample doesn't support Lanczos interpolation (NPPI does) which is preferred for my use case and was a bit slower than calling the VPF function. Your comment did make me realize that I could perform my warp map generation in PyTorch (was processing with OpenCV CUDA and downloading/uploading before). Was able to hack the format of the ptr to surface conversion function in VPF and make remap accept Surface inputs for maps each frame so I could avoid continuous memory creation and stay on GPU. Made my pipeline 2x faster!

I'll work on reworking the codebase to call all the VPF methods in one location to avoid the original error. Thanks for the help!

[gedoensmax]

If you make the solution less hacky but expose it as an API I would be happy to approve a PR for it ! Have you thought about exposing a "reconfigure" function on the surface remapper that just exchanges it's map instread of recreating the whole thing?

Also it is likely that nsight systems will tell you why exactly the Remapper get freed before it can be. Feel free to share a report !

[anklebreaker]

The hacky part was transferring a float pointer map into a surface since both the method to convert only accepts uint8 buffers (the CVCUDA example says float tensor but it seems to be passing a uchar buffer and there's an explicit check for uchar in the code) and there's no single channel float format (e.g. PixelFormat.Y32F). I solved it by faking my data like it was RGB_PLANAR image by viewing the float buffer as a byte buffer of 4x width and making it three channel by dividing the height by 3 like so:

pnvc.TensorToDptr(map_x.view(torch.uint8).reshape(3, self.height // 3, self.width * 4), surf_x_plane.GpuMem(), surf_x_plane.Width(), surf_x_plane.Height(), surf_x_plane.Pitch(), surf_x_plane.ElemSize())
pnvc.TensorToDptr(map_y.view(torch.uint8).reshape(3, self.height // 3, self.width * 4), surf_y_plane.GpuMem(), surf_y_plane.Width(), surf_y_plane.Height(), surf_y_plane.Pitch(), surf_y_plane.ElemSize())

This worked ok for my use case since I'm working with 16:9 aspect ratio videos which have heights divisible by 3. Making the remap method accept Surface inputs was a bit easier, just a heavy amount of copy pasting through the various levels of abstraction. If you'd like I can post that as PR for you to take a look, but if there's a better way to uploading the float Tensor let me know (there's 2 copies currently, one for the reshape in PyTorch and one for the CUDA copying in ptr to Surface conversion). I'm overall a bit new to GPU code.

[gedoensmax]

Ok if it does the job for you that's fine but it sounds like this is not a solution we want to integrate in generel.