Review of frame tensor creation / allocation

[NicolasHug]

We create/allocate output frame tensors in different places. In particular, we determine the height and width of the output tensor from different sources:

• For batch APIs (CPU and GPU): from the stream metadata, which itself comes from the CodecContext
• For single frame APIs:
  • CPU: swscale and filtergraph: from the AVFrame
  • GPU: from the CodecContext

The info from the metadata / CodecContext are available as soon as we add a stream, e.g. right after we instantiate a Python VideoDecoder. The AVFrame is only available once we have decoded the frame with ffmpeg (this is the "raw output").

The source of truth really is the AVFrame. CondecContext may be wrong, and in particular we now know that some streams may have variable height and width #312.

---

Details:

• For batch APIs:

https://github.com/pytorch/torchcodec/blob/41c6491d81e00f47529074ca1c78217cc03fc0ea/src/torchcodec/decoders/_core/VideoDecoder.cpp#L165-L187

• For single frames APIs: https://github.com/pytorch/torchcodec/blob/41c6491d81e00f47529074ca1c78217cc03fc0ea/src/torchcodec/decoders/_core/VideoDecoder.cpp#L887-L890
  and https://github.com/pytorch/torchcodec/blob/41c6491d81e00f47529074ca1c78217cc03fc0ea/src/torchcodec/decoders/_core/VideoDecoder.cpp#L1279-L1286

• For CUDA APIs:
  https://github.com/pytorch/torchcodec/blob/41c6491d81e00f47529074ca1c78217cc03fc0ea/src/torchcodec/decoders/_core/CudaDevice.cpp#L156-L161

[NicolasHug]

I'll close this issue. I've addressed this in a bunch of successive PRs, the state isn't 100% where we want it to be yet, but at least now we have assumption checks in place, and all strategies are documented

https://github.com/pytorch/torchcodec/blob/84cef500b5865c44fe5c799a2f468c2067d31220/src/torchcodec/decoders/_core/VideoDecoder.h#L430-L473