Torchcodec example

[bhack]

Do you plan to have examples with Torchcodec?
https://github.com/meta-pytorch/torchcodec

[mthrok]

Hi

We don't have such a plan, but what kind of task do you have in your mind?

[bhack]

Using the video hw accelerated pipeline and seeking without going to reintegrate twice with the ffmpeg low level API natively in pytorch. E.g. In the video example here we need to manipulate a lot of low level API I think it is a little bit less practical from a classical ML/AI pipeline.

https://github.com/facebookresearch/spdl/blob/main/examples/streaming_nvdec_decoding.py

[mthrok]

I see. Unfortunately, I am not familiar with Torchcodec. Maybe @NicolasHug or someone else from the team can help on that.

[NicolasHug]

Hi folks, I'm happy to help review a PR that adds a TorchCodec-based tutorial, if that's in scope for SPDL.

[bhack]

Is there any relationship between the official data/datalodaer refactoring and this project?

This could help to understand a TorchCodec role in the integration.

[mthrok]

Is there any relationship between the official data/datalodaer refactoring and this project?

By "the official data/datalodaer refactoring", I assume you are referring to activities in PyTorch org.

SPDL was developed at Meta Reality Lab to solve the data loading bottleneck of production system.
The codebase comes directly from the work we apply to AI model pipelines used for Reality Lab products.
We made the code generic and compatible with OSS.
SPDL saw huge success at Reality Lab, which inspired PyTorch data project and they created torchdata.nodes.

So in a sense PyTorch data nodes is a derivative of SPDL.
Because of this, there are similarity in the interface, but the similarity and relationship stops there.
Technical details and UX are vastly different.

[bhack]

I was mainly talking about the mega thread at meta-pytorch/data#1196 but it is hard today to track things on this topic on the current STOA.

For sure the problem related to efficiently handling video/images pipelines it is still quite important but it is not clear on what repo investing as a third party.

[mthrok]

Yes. The data loading bottleneck is everywhere and it hinders other efforts like model optimization. It is extremely important problem to solve, and it gets harder to solve each year when new generation of GPU appears. Unfortunately, the architecture of PyTorch's DataLoader is not suited for solving the task, so we built SPDL while meeting the important criteria and keeping the solution as generic as possible.

What PyTorch wants to do with their projects and what they envision in ecosystem is on them, so I can only hope for their best.

[bhack]

Yes I totally agree on your points this is really one of the main data pipeline priority for the ecosystem IMHO.

This is why I have asked here about TorchCodec as I suppose it could potentially remove some boil-operate relatively low level/recurrent ffmpeg interfacing code at least.

[mthrok]

Are you interested in if we provide a simpler interface?

The potential problem with TorchCodec's GPU decoder is that it relies on GPU-enabled ffmpeg (libav).
This is the same approach I took when I wrote IO modules in TorchAudio, and there are certain performance-related concerns.
(TorchCodec team might have found a workaround for these issues, or maybe it's not so much of a problem par se.)

1. FFmpeg creates a separate CUDA context than PyTorch.
2. FFmpeg allocates CUDA memory when decoding.
3. You cannot use reconfigure to reuse the GPU decoder.

For 1, FFmpeg creates CUDA context when initializing a CUDA decoder. This consumes 500MB of GPU memory. When attempting to use all the NVDEC ships (for example 7 chips on H100), it can consume 3.5 GB of GPU memory. This memory consumption might be acceptable depending on the application, but I feel it is unnecessary.
(IIRC, FFmpeg6 or 7 introduced an option to share the CUDA context but I am not sure how it can be integrated)

For 2, FFmpeg's CUDA decoder plugin calls cudaMalloc and such directly, while PyTorch manages the CUDA memory through CUDA caching allocator. Letting NVDEC use the PyTorch's CUDA caching allocator is more efficient.

For 3, What makes difficult to take advantage of GPU video decoder is its initialization time. It takes about 300ms in my benchmark. There are couple of ways to avoid re-initialization. NVDEC has reconfigure mechanism, which can avoid the expensive decoder initialization cost when the only difference of videos are the resolutions.

As a bonus, SPDL I/O does not require GPU-enabled FFmpeg to use NVDEC, which is handy for OSS users.

[NicolasHug]

Moto just so you know we have addressed these in TorchCodec's new "beta" CUDA interface we released recently, which doesn't rely on FFmpeg: https://github.com/meta-pytorch/torchcodec/releases/tag/v0.8.0

For point 3, we're not re-configuring the CUvideodecoder, we're literally just caching it. That's what the pynvvidecodec folks do as well (https://www.nvidia.com/en-us/on-demand/session/gtc25-s73178/?start=1136). DALI has a cache too, they also have some re-configuration logic but it doesn't seem to be compiled by default. I'd love to know which technique to use when, but I haven't had time to do any benchmarking on that yet.

[mthrok]

@bhack I briefly looked at the documentation of torchcodec, and I see that all the decoding logic is put behind of VideoDecoder class.

This suggests that all you can do / have to do to integrate it with SPDL is just to write a wrapper function and pass it to PipelineBuilder.pipe method.

Something like the following.

def decode(input_data: T) -> Tensor:
    # Use `input_data` and `VideoDecoder` to decode video
    ...

pipeline = (
    PipelineBuilder()
    .add_source(sampler)
    .pipe(decode)
    ...
    .add_sink(...)
    .build(...)
)

with pipeline.auto_stop():
    for batch in pipeline:
        ...