torchvision 0.10.0

Breaking changes

• model_maskrcnn_*() now return image-sized mask probabilities in masks, shape (N, H, W) (@sebffischer, #412). Code that used the per-box (N, 28, 28) probabilities should read output$detections[[1]]$mask_probs instead of output$detections[[1]]$masks. For image overlays, use output$detections[[1]]$masks > 0.5.
• State dictionaries saved with torchvision 0.9.0 need migration for model_vit_*(), model_facenet_inception_resnet_v1(), model_convnext_*_upernet(), model_fasterrcnn_resnet50_fpn_v2() and model_maskrcnn_resnet50_fpn_v2(): parameters were added, removed or renamed, so these checkpoints cannot be loaded unchanged with strict = TRUE (#405, #406, #410, #412). RF-DETR checkpoints from earlier development versions may also need migration after changes to decoder depths and resolutions (#413). There is no automatic migration for user-saved checkpoints. Keep the package version that created them, or explicitly migrate and validate the weights. model_maxvit() retains its state-dict keys and shapes, but its corrected forward computation can change predictions (#405). Support for the hosted pretrained weights does not imply compatibility with user-saved checkpoints.
• When supplied, draw_bounding_boxes() labels and colors must have length 1 or match the number of original boxes. Partial recycling is no longer supported (#415).

New models

• Added RF-DETR model family: model_rfdetr_*, with * being nano(), small(), medium(), base(), base_2(), base_o365() and large() for real-time object detection with DINOv2 backbone and deformable-attention decoder (@DerrickUnleashed, #327).
• Added LW-DETR model family: model_lw_detr_*, with * being tiny(), small(), medium() and large() for real-time object detection with ViT backbone and deformable-attention decoder (@srishtiii28, #328).

New datasets

• Added cityscapes_dataset() relying on manual download due to usage license (#279, @ANAMASGARD).

New features

• Added target_transform_resize to manage bounding_box resizing in parallel to transform_resize for images (#337).
• Added target_transform_rotate() for converting bounding boxes from xyxy to xyxyr format with a rotation angle (@DerrickUnleashed, #338). Both operate on detection item targets (the y part of a dataset item) for composable target transform pipelines.
• Added a new draw_bounding_boxes S3 method for image_with_rotated_box that renders rotated boxes as polygons (@DerrickUnleashed, #338).
• Added item_transform_hflip() for horizontally flipping dataset items, with support for detection and segmentation item types and datasets (@DerrickUnleashed, #366).
• Added item_transform_vflip() for vertically flipping dataset items, with support for detection and segmentation item types and datasets (@DerrickUnleashed, #369).
• Added SAHI (Slicing Aided Hyper Inference) support via a three-function pipeline: prepare_sahi_split() precomputes overlapping crop windows, then transform_sahi_crop() slices images and target_transform_sahi_crop() adjusts detection targets per crop. (@DerrickUnleashed, #324)
• Added item_transform_affine() and target_transform_affine() for applying an affine transformation to a dataset item and its detection target, with support for detection and segmentation items and datasets (#367).
• Added item_transform_center_crop() for cropping images from the center to a specified size for dataset items, with support for detection and segmentation item types and datasets (@DerrickUnleashed, #370).
• Added item_transform_random_horizontal_flip() and item_transform_random_vertical_flip() for random geometric augmentation of dataset items with probability p, with support for detection and segmentation item types and datasets (@DerrickUnleashed, #381).
• Added item_transform_resize() for resizing dataset items, with support for detection and segmentation item types and datasets (@srishtiii28, #362).
• item_transform_rotate() now supports segmentation items and datasets, rotating the masks alongside the image (@srishtiii28, #379).
• Added item_transform_crop() for cropping dataset items at a specified location and size, with support for detection and segmentation item types and datasets (@DerrickUnleashed, #371).
• Added item_transform_pad() for padding dataset items on all sides, with support for detection, segmentation and rotated-box item types and datasets (@DerrickUnleashed, #373).
• Added item_transform_perspective() for applying a perspective transform to dataset items, with support for detection and segmentation item types and datasets, and implemented transform_perspective() for tensor images (@DerrickUnleashed, #377).
• Added item_transform_random_resize_crop() for cropping dataset items to a random area and aspect ratio before resizing them to a given size, with support for detection, segmentation and rotated-box item types and datasets (@srishtiii28, #386).
• Added item_transform_random_crop() for cropping dataset items at a random location with optional padding, with support for detection and segmentation item types and datasets (@DerrickUnleashed, #387).
• Added item_transform_random_erasing() for randomly erasing a rectangular region of dataset items with probability p, with support for detection, segmentation and rotated-box item types and datasets (#389).
• Added item_transform_random_affine() for applying an affine transformation drawn from the given ranges to dataset items, with support for detection, segmentation and rotated-box item types and datasets (@srishtiii28, #390).
• Added item_transform_random_perspective() for applying a random perspective transformation to dataset items with probability p, with support for detection, segmentation and rotated-box item types and datasets (@DerrickUnleashed, #393).
• Detection datasets (coco_detection_dataset(), pascal_detection_dataset() and the rf100_*_collection()s) now inherit the object_detection_dataset class and their item target y the object_detection_target class. Segmentation datasets (coco_segmentation_dataset(), pascal_segmentation_dataset(), cityscapes_dataset(), oxfordiiitpet_segmentation_dataset() and rf100_peixos_segmentation_dataset()) now inherit the segmentation_dataset class and their item target y the segmentation_target class. Target transforms now dispatch on those classes: target_transform_resize(), target_transform_rotate(), target_transform_affine() and target_transform_sahi_crop() take an object_detection_target, and target_transform_coco_masks() and target_transform_trimap_masks() a segmentation_target. Bare-list targets remain supported and retain their list class (@srishtiii28, #391).
• Added a "Visualization utilities" article covering vision_make_grid(), draw_bounding_boxes(), draw_segmentation_masks() and draw_keypoints() on the output of model_rfdetr_base() and model_fcn_resnet50() (@srishtiii28, #400).
• Added a "torchvision classes" vignette sketching what each class holds: the image_with_bounding_box, image_with_rotated_box and image_with_segmentation_mask items, their object_detection_target and segmentation_target, the sahi_split returned by prepare_sahi_split(), and which transform and drawing functions dispatch on each (@srishtiii28, #396).

Bug fixes and improvements

• draw_bounding_boxes() now keeps labels and colors aligned with their boxes when lazy = TRUE filters out degenerate boxes (#415).
• Fixed box alignment in padded center crops and random crops, and corrected five- and ten-crop geometry (#414).
• model_vit_*() predictions now depend on the input image: attention runs over the patches of each image, the positional embedding is registered so it is loaded from the pretrained weights, and layer norm uses eps = 1e-6 (@sebffischer, #405).
• model_maxvit() is re-ported from PyTorch and now reproduces its predictions with the pretrained weights (@sebffischer, #405).
• draw_bounding_boxes() now accepts degenerate bounding boxes by default with the lazy = TRUE parameter (#400).
• vision_make_grid() now accepts multiple 3D tensors with mixed uint8 and float dtype, while preserving positional scale, num_rows, padding and pad_value arguments (#398). Use per_row for the number of images per row; named num_rows remains supported as a deprecated alias.
• transform_random_affine() now accepts a bare number for shear. It used to widen degrees instead of shear, which left the shear range incomplete and made the sampling fail (#390).
• item_transform_rotate() no longer truncates the rotation angle to a whole number of degrees when the target stores its boxes as an integer tensor, as pascal_detection_dataset() does (#360).
• transform_random_resized_crop() no longer loses the last row and column of the image, and no longer pads with a black edge when it falls back to a central crop (#386).
• transform_crop() now pads the result with zeros when the crop leaves the image, so that the output always has the requested size. It previously returned only the part of the image the crop covered, which could even be empty.
• transform_center_crop() now pads correctly when the requested size is not square. Height and width were used in the wrong order in the padding branch, so e.g. cropping a (16, 20) image to c(24, 30) returned a (22, 0) image.
• transform_rgb_to_grayscale() now returns an image with one channel instead of dropping the channel dimension, i.e. (3, H, W) becomes (1, H, W) and not (H, W). This also fixes transform_grayscale() for batched input.
• hsv2rgb() computed the fractional part of the hue sector and the t component incorrectly, so converting an image to HSV and back did not return the original image. This affected transform_adjust_hue() and transform_color_jitter(hue = ...).
• transform_adjust_hue() now handles negative hue_factors, which produced a black image before, and rejects images that do not have 1 or 3 channels instead of silently dropping the additional ones.
• nms() now uses torchvisionlib::ops_nms() when torchvisionlib is installed, speeding up inference for model_fasterrcnn_*() and model_maskrcnn_*() (#321, #322).
• Lists and vectors are now preallocated to their target size instead of being grown one element at a time (@srishtiii28, #335).
• model_convnext_small_22k1k() now defaults to num_classes = 1000. It is pretrained on Imagenet 22k and fine-tuned on Imagenet 1k, so its published weights have a 1000-class head, and the previous default of 21841 made pretrained = TRUE fail to load the state dict. (@sebffischer, #376)
• model_inception_v3(pretrained = TRUE, aux_logits = FALSE) now actually removes the auxiliary classifier. Previously aux_logits = FALSE was silently ignored for pretrained models, so the network returned a list of both heads instead of a single tensor in training mode. (@sebffischer, #375)
• model_rfdetr_*() now match the reference rf-detr implementation numerically, accept padding masks, and use the official configurations of the small, medium and base variants (@sebffischer, #413).
• model_fasterrcnn_*() and model_maskrcnn_*() inference now reproduces torchvision numerically, supports batches, and loads pretrained weights strictly (@sebffischer, #412).
• model_inception_v3() and model_resnet*() without pretrained weights now initialise all layers as in PyTorch, and zero_init_residual = TRUE takes effect (@sebffischer, #411).
• The progress argument of the model constructors is now honoured (@sebffischer, #411).
• model_convnext_*_upernet() now apply the per-stage layer norms of the pretrained backbone and default to num_classes = 150 when pretrained = TRUE (@sebffischer, #410).
• model_convnext_*_detection() default to num_classes = 90 and pass score_thresh, nms_thresh and detections_per_img to the detection head (@sebffischer, #410).
• model_convnext_*() blocks now apply stochastic depth in training mode (@sebffischer, #410).
• model_efficientnet_*() and model_efficientnet_v2_*() now match torchvision (SiLU squeeze-excitation, batch norm settings), apply stochastic depth in training, and accept dropout without shifting other arguments (@sebffischer, #409).
• model_mobilenet_v3_*() use torchvision's batch norm settings, gain a norm_layer argument and stay in training mode when pretrained = FALSE; model_mobilenet_v2() handles stages with n = 0 (@sebffischer, #409).
• model_fcn_resnet*() now use a dilated backbone as in PyTorch, fixing the pretrained segmentations, and no longer download the backbone weights when pretrained = TRUE (@sebffischer, #408).
• model_deeplabv3_*() keep the auxiliary classifier when pretrained = TRUE and aux_loss = NULL, as documented (@sebffischer, #408).
• model_lw_detr_*() now match the reference LW-DETR implementation numerically, which strongly improves the pretrained detections (@sebffischer, #407).
• model_facenet_pnet(), model_facenet_rnet(), model_facenet_onet() and model_mtcnn() take the face softmax over the class dimension instead of the batch dimension (@sebffischer, #406).
• model_facenet_inception_resnet_v1() now matches facenet-pytorch (missing repeat_1 stage, Mixed_6a paddings); pretrained = TRUE loads "vggface2" and num_classes defaults to NULL (@sebffischer, #406).