TorchCodec 0.16 - Image decoding and encoding

TorchCodec 0.16 is out! It is compatible with torch >= 2.11. The headline feature of this release is image decoding and encoding: TorchCodec now natively decodes and encodes JPEG (CPU and CUDA), PNG, WebP, GIF, AVIF and HEIC. These image decoders and encoders replace their torchvision counterparts, which are now deprecated.

TorchCodec is the recommended way to decode and encode images in the PyTorch ecosystem. If you are coming from torchvision, we wrote a migration guide

Image decoding

TorchCodec exposes one entry-point per format, plus a generic decode_image() that automatically detects the format. The API is largely backward-compatible with TorchVision:

from torchcodec.decoders import decode_image, decode_jpeg, decode_png

img = decode_image("image.jpg")  # CHW uint8 tensor, format auto-detected
img = decode_image("image.avif")
img = decode_image("image.heic")

# Or use the format-specific decoders for format-specific options
img = decode_jpeg("image.jpg", device="cuda")

Sources can be a path (str or pathlib.Path), bytes, or a 1D uint8 tensor of encoded bytes:

img = decode_image(open("image.png", "rb").read())
img = decode_image(torch.frombuffer(encoded_bytes, dtype=torch.uint8))

Animated and multi-image formats (WebP, GIF, AVIF, HEIC) decode into an (N, C, H, W) tensor:

from torchcodec.decoders import decode_gif

frames = decode_gif("animated.gif")  # (N, C, H, W)

JPEG decoding is also supported on CUDA, through nvJPEG. For CUDA, prefer passing a batch of sources: the whole batch is decoded in a single nvJPEG call, which is much faster than decoding images one at a time.

from torchcodec.decoders import decode_jpeg

imgs = decode_jpeg(["a.jpg", "b.jpg", "c.jpg"], device="cuda")  # list of CUDA tensors

Read more in our image decoding tutorial

Image encoding

Image encoders follow the same class-based design as our video and audio encoders: build the encoder from a CHW uint8 tensor, then choose where the encoded bytes go: a file, a file-like object, or a tensor.

from torchcodec.encoders import JpegEncoder, PngEncoder

JpegEncoder(img).to_file("image.jpg", quality=90)
PngEncoder(img).to_file("image.png", compression_level=9)

# ... or to a file-like object
import io
buffer = io.BytesIO()
JpegEncoder(img).to_file_like(buffer)

# ... or to a 1D uint8 tensor of encoded bytes
encoded = PngEncoder(img).to_tensor()

JPEG encoding is supported on CUDA as well: pass a CUDA tensor and the encoding happens on the GPU with nvJPEG, with to_tensor() returning a CUDA tensor (no host round-trip).

encoded = JpegEncoder(img_on_cuda).to_tensor(quality=90)  # CUDA uint8 tensor

Read more in our image encoding tutorial

Improvements over torchvision's decoders / encoders

The image decoders and encoders were migrated from torchvision and torchvision-extra-decoders, with the same performance, and they are significantly more capable:

• All color modes for every codec: UNCHANGED, GRAY, GRAY_ALPHA, RGB, RGB_ALPHA. torchvision only supports GRAY for PNG and JPEG, and rejects or ignores it elsewhere.
• Animation and multi-image support: animated WebP, GIF and AVIF, and multi-image HEIC, all decode to (N, C, H, W). torchvision rejects animated WebP, errors on multi-image AVIF, and only decodes the primary HEIC image.
• EXIF orientation applied by default, for JPEG (CPU and CUDA), PNG, WebP, AVIF and HEIC. In torchvision it is opt-in, PNG/JPEG-only, and ignored on CUDA.
• output_dtype control (torch.uint8, torch.uint16, or "auto") on every decoder. torchvision has no equivalent: the output dtype is dictated by the source.
• scalability of JPEG encoding and decoding: TorchCodec allows multiple NVJPEG decoders and encoders instances per process, allowing to scale decoding and encoding throughput in multi-threaded pipelines.
• decode_image() auto-detects all six formats, including AVIF and HEIC. torchvision only handles four.
• Richer inputs: str/Path/bytes/Tensor everywhere, non-contiguous encoded input accepted, and batched input for JPEG on both CPU and CUDA.
• No extra package needed: AVIF works out of the box (libavif is bundled). HEIC works if libheif is found at runtime. We don't bundle it because it is LGPL, so install it yourself (e.g. conda install -c conda-forge libheif). Torchvision required the separate torchvision-extra-decoders package for both, and its decode_image couldn't dispatch to them.
• file-like support for encoders - not supported by torchvision.

Along the way we fixed a number of correctness bugs inherited from torchvision, among them: PNG palette and tRNS transparency handling, GIF frame disposal (now aligned with Pillow), truncated JPEGs erroring instead of returning garbage, correct CMYK/YCCK handling, real grayscale for WebP, progressive AVIF stills, and full-range >8-bit HEIC output.

If you are coming from torchvision, we wrote a migration guide

FFmpeg is now an optional dependency

import torchcodec no longer fails at import time if FFmpeg cannot be found. FFmpeg is still required for video and audio decoding and encoding, but the image decoders and encoders don't need FFmpeg and work in FFmpeg-free environments.

FFmpeg 9 support

TorchCodec now support the recently released FFmpeg 9!

Bug Fixes

• Audio resampling correctness. Decoding a resampled audio stream in chunks now returns exactly the same samples as decoding it in one go. (#1604, #1614, #1615, #1616).
• MPEG-PS seeking. Fixed AudioDecoder seeks on MPEG-PS files (#1619).
• Encoders: to_tensor() no longer emits a spurious warning (#1510).

TorchCodec 0.15

TorchCodec 0.15 is out! This is a small release compatible with torch >= 2.11, with the following improvements:

• #1503 and #1504 improved decoding coverage on some videos, where a premature "end of file" would otherwise be raised.
• #1489 optimizes forward seeks - you should see better performance on sparse decoding scenarios, especially on CPU when num_ffmpeg_threads is high.
• Free-threaded wheels are now shipped for MacOS (and Linux, but that was already supported).

TorchCodec 0.14: HDR Video Decoding for CPU & CUDA, and Fast Wav Decoder

TorchCodec 0.14 is out! It is compatible with torch >= 2.11. It comes with two major additions: a fast audio WavDecoder, and support for HDR video decoding!

Fast wav decoder

Inspired by SDPL's fast wav decoder, TorchCodec now has a dedicated WavDecoder for decoding WAV files. It bypasses FFmpeg entirely and reads WAV data directly, resulting in significantly faster decoding. It supports multiple sample formats (int16, int32, float32, etc.), and can decode from files, bytes, or file-like objects.

from torchcodec.decoders import WavDecoder

decoder = WavDecoder("audio.wav")
samples = decoder.get_all_samples()  # AudioSamples with data and sample_rate

Read more in our docs.

HDR Video Decoding

VideoDecoder now supports HDR (High Dynamic Range) video decoding without losing precision. When output_dtype=torch.float32 is specified, the decoder outputs RGB float32 frames in [0, 1], preserving the full HDR color range. This is supported for both CPU and CUDA!

import torch
from torchcodec.decoders import VideoDecoder

decoder = VideoDecoder("hdr_video.mp4", output_dtype=torch.float32)
frame = decoder[0]  # Full HDR precision in float32

Read more in our docs.

⚠️ This feature is in beta stage, so behavior may slightly change depending on user feedback. Let us know if you encounter any issue!

Other Improvements

• Improved audio seeking: AudioDecoder seeking is now much faster (#1449)
• Dropped NPP dependency: TorchCodec no longer depends on NVIDIA's NPP library, which will simplify installing and using TorchCodec for CUDA decoding.

Bug Fixes

• Fix a rare crash scenario during process teardown with the CUDA decoder (#1441)
• Fix CUDA decoding of videos with odd dimensions(#1462)

TorchCodec 0.13

TorchCodec 0.13 is out! It is compatible with torch >= 2.11, and it is packed with new features.

Multi-stream iterative Encoder

This release comes with a new major feature: the multi-stream Encoder! The Encoder supports multiple streams and incremental encoding. Frames and samples can be added progressively, which is useful when data is generated on-the-fly or when encoding both audio and video into the same container.

from torchcodec.encoders import Encoder

encoder = Encoder()

video_stream = encoder.add_video(height=256, width=256, frame_rate=30)
audio_stream = encoder.add_audio(sample_rate=16000, num_channels=1)

with encoder.open_file("output.mp4"):
	video_stream.add_frames(frames_tensor)
	audio_stream.add_samples(samples_tensor)
	# Add more frames by calling video_stream.add_frames again
	# Add more samples by calling audio_stream.add_samples again

The Encoder also supports CUDA encoding! Read more in our docs!

Broader support for aarch64 and Windows

Based on popular requests, we are now shipping aarch64 CPU wheels and Windows CUDA wheels. Both are in beta status, so let us know if you encounter any issue. See our installation instructions for more details.

TorchCodec now officially supports the following platforms:

• Linux x86 (CUDA and CPU)
• Linux aarch64 (CUDA and CPU)
• Windows (CUDA and CPU)
• MacOs

CUDA wheels now shipped on PyPI by default

pip install torchcodec should now install the CUDA wheels by default on Linux x86 and aarch64. Those wheels should still work even if you do not have a CUDA GPU, or if you are missing CUDA dependencies. Let us know if you encounter any issue.

To install the CPU-only wheels, please refer to our installation instructions.

Bug fixes

• Fix the CUDA decoder on MPEG-4 Part 2 videos (#1352)
• Fix decoding of wav files which would sometimes crash when decoding from bytes (#1379)
• Fix the CUDA decoder on yuv444 videos (#1415)

TorchCodec 0.12

TorchCodec 0.12 is out! This is a small release that focuses on completing the stable ABI migration and switching our default cuda backend to the faster backend. In 0.12, we are also aligning with pytorch repo’s cuda support by dropping cuda 12.8 and adding support for cuda 13.2.

Faster Cuda Backend is the new default

Starting in TorchCodec 0.12, the faster CUDA backend (previously known as ‘beta’) becomes the default backend. This will be a transparent and backward-compatible change.

# Previously, this used the slower 'FFmpeg' backend.
# Now this uses the faster backend by default.
decoder = VideoDecoder(..., device="cuda")

Users who want to stay on the less efficient FFmpeg backend should explicitly use set_cuda_backend:

with set_cuda_backend("ffmpeg"):
    decoder = VideoDecoder(..., device="cuda")

(#1350, #1359)

ABI Stability

TorchCodec 0.12 will be ABI stable from torch 2.11 (yes, 2.11)! Previously, each new version of torch required a corresponding version of TorchCodec, which made dependency management complex for users. From 0.12, TorchCodec should be largely forward-compatible with future versions of torch, simplifying the installation and dependency management process.

Bug Fixes

• Fixed a bug that caused libx265 stale frame returned after backward seek (#1349)
• The default CUDA backend now supports videos with odd dimensions (#1364)

TorchCodec 0.11.1

We are releasing TorchCodec 0.11.1 as a bug-fix release, and it is compatible with torch 2.11.

The fix:

In 0.11, we began uploading CUDA 13.0 linux wheels to PyPI, so pip install torchcodec would install a CUDA enabled wheel. This caused an error at import time for users without the necessary CUDA libraries, or for users who only wanted to use CPU decoding.

Torchcodec 0.11.1 reverts this change, so pip install torchcodec will install the CPU wheel. We will resume uploading CUDA wheels as torch does in torchcodec version 0.12!

As before, CUDA wheels are available by specifying the index-url argument:

pip install torchcodec --index-url --index-url https://download.pytorch.org/whl/cu130

TorchCodec 0.11

TorchCodec 0.11 is out! This release brings CUDA decoding improvements and improved HDR metadata and rotation support in VideoDecoder, as well as output fps support!

CUDA decoder performance improvements

We made significant improvements to the CUDA decoder throughput, available via the “beta” backend:

• #1232 fixed a bug in the decoder cache, allowing more than one decoder instance to be cached per video configuration. The fix doesn’t affect single-threaded pipelines, but drastically improves the throughput of multi-threaded decoding.
• #1227 improved cache performance further.
• #1243 added an LRU eviction policy for the cache, which will improve the cache hit when decoding lots of different video configurations.
• #1246 added the set_nvdec_cache_capacity() which allows the user to control the cache size. Larger cache sizes are typically more performant, and more memory consuming.

These are available via the “beta” backend”.

⚠️ Note that in the next release, the “beta” backend will become the default backend. This will be a transparent and backward-compatible change. Users who want to stay on the less efficient FFmpeg backend should use:

with set_cuda_backend("ffmpeg"):
    decoder = VideoDecoder(..., device="cuda")

Read more about this in the CUDA utilities section!

FPS Resampling

get_frames_played_in_range() now accepts a fps parameter to resample video at a target frame rate, duplicating or dropping frames as necessary to match the desired output FPS:

decoder = VideoDecoder(path)
# If a source video is 25 fps, a 1-second range will contain 25 frames.
# We can use the fps argument to resample to 5 fps, which gives us 5 frames:
frames_5fps = decoder.get_frames_played_in_range(start_seconds=1, stop_seconds=2, fps=5)

Read the VideoDecoder docs for more details!

(#1148)

Rotation Support

TorchCodec now automatically applies rotation metadata during video decoding on CPU and Beta Cuda backend.

decoder = VideoDecoder(path)
print(decoder.metadata.rotation)  # e.g. 90.0, or None

⚠️ Note that this is a BC-breaking change since we consider it a bug fix. Read more about this in the VideoStreamMetadata docs!

(#1173, #1235)

HDR & Color Metadata

Video Decoder metadata now exposes color-related metadata and pixel format, making it easy to identify HDR content:

metadata = VideoDecoder(path).metadata
print(metadata.color_primaries)     # e.g. "bt2020"
print(metadata.color_space)         # e.g. "bt2020nc"
print(metadata.color_transfer)      # e.g. "smpte2084"
print(metadata.pixel_format)        # e.g. "yuv420p10le"

Read more about these fields in the VideoStreamMetadata docs!

(#1271, #1261, #1267)

Installation enhancements

• On Linux, pip install torchcodec now defaults to the CUDA 13.0 wheel to match the behavior of pip install torch. See updated instructions in our README.
• Additionally, we have added aarch64 CUDA wheels to PyPI!

Bug Fixes

• Fixed audio decoding issue when decoding audio with more than 8 channels. (#1166)
• Fixed MKV decoding being up to 42x slower than MP4 in approximate seek mode in some situations. (#1259)
• Fixed frame indexing for videos with non-zero start times in approximate seek mode. (#1209)
• Fixed time-based samplers to use float64 instead of float32 to avoid precision errors. (#1294)
• Improved BT.709 full-range CUDA color conversion on CUDA 12. (#1265)
• Improved BT.601 CUDA color conversion accuracy. (#1270)
• Fixed AudioDecoder thread oversubscription with certain codecs (FLAC, TAK, wavpack). (#1254)
• Fix SwScale error with non-32 aligned input. (#1295)

TorchCodec 0.10

TorchCodec 0.10 is out! It is compatible with torch 2.10, and comes with exciting new features.

Decoder Transforms

Decoder Transforms are available! We have released Resize, CenterCrop, RandomCrop, which can be used in VideoDecoder to transform data during preprocessing:

resize_decoder = VideoDecoder(
    video_path,
    transforms= [
        torchcodec.transforms.RandomCrop(size=(1280, 1664)),
        torchcodec.transforms.Resize(size=(480, 640)),
    ]
)
resized_frame = resize_decoder[5]

Read more about this in the tutorial!

Let us know any transforms you want to see added in #1134!

Video Encoding on GPU

VideoEncoder now supports encoding on GPU! This can improve performance by ~3x!
To use it, simply move the input frames onto the CUDA device before encoding:

encoder = VideoEncoder(frames=frames.cuda(), frame_rate=frame_rate)
encoder.to_file(dest="output.mp4", codec="h264_nvenc")

Performance Tips guide

Check out our new performance tips guide to read about best practices to improve performance!
The guide covers batch APIs, decoding seek modes, multi-threading, GPU decoding, and checking for CPU fallback during decoding.

Enhancements

• We've added a detailed stack trace when FFmpeg is not found. This should help debug installation issues on various development environments. (#1138)
• On MacOS, we've improved Homebrew FFmpeg discoverability. (#1152, #1175, #1177)

TorchCodec 0.9.1

TorchCodec 0.9.1 is out! This version is compatible with torch 2.9.

This is primarily a bug-fix release which should resolve issues on Windows where FFmpeg couldn't be found.

TorchCodec 0.9

TorchCodec 0.9 is out! This comes with a new highly requested feature: video encoding!

Video Encoding

Video encoding on CPU is available. It provides a simple API to encode video frames to tensors or bytes, and optionally enables a set of key parameters.

from torchcodec.encoders import VideoEncoder

encoder = VideoEncoder(frames=frame_tensor, frame_rate=frame_rate)

encoder.to_file(dest="output.mp4") # encode to mp4 file
encoded_bytes = encoder.to_tensor(format="mp4") # encode to tensor of bytes

Additionally, several key parameters are exposed to control the encoded video:

# Utilize a specific codec, choose a pixel format to control quality
encoder.to_file(dest="output.mp4", codec="libx264", pixel_format="yuv420p")
 
# Set quality parameter `crf` to 0 for lossless encoding, use fast `preset`
encoded_bytes = encoder.to_tensor(format="mp4", crf=0, preset="fast")

Read more about the available features in the video encoding tutorial!

Enhancements

• This release adds support for Python 3.14!
• #989: Improved VideoDecoder metadata, enabling seek_mode=approximate for some videos with missing metadata.
• #1028: Enhanced video decoding speed up to 1.5x when decoding frames sequentially with seek_mode="approximate".
• #1078: Updated guidance on when to use approximate mode in the tutorial.

Bug fixes

• #1025: Fixed bug: passing device=None in VideoDecoder now uses the current torch device.