Skip to content


Folders and files

Last commit message
Last commit date

Latest commit



14 Commits

Repository files navigation

HyperSeg - Official PyTorch Implementation

Teaser Example segmentations on the PASCAL VOC dataset.

This repository contains the source code for the real-time semantic segmentation method described in the paper:

HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation
Conference on Computer Vision and Pattern Recognition (CVPR), 2021
Yuval Nirkin, Lior Wolf, Tal Hassner

Abstract: We present a novel, real-time, semantic segmentation network in which the encoder both encodes and generates the parameters (weights) of the decoder. Furthermore, to allow maximal adaptivity, the weights at each decoder block vary spatially. For this purpose, we design a new type of hypernetwork, composed of a nested U-Net for drawing higher level context features, a multi-headed weight generating module which generates the weights of each block in the decoder immediately before they are consumed, for efficient memory utilization, and a primary network that is composed of novel dynamic patch-wise convolutions. Despite the usage of less-conventional blocks, our architecture obtains real-time performance. In terms of the runtime vs. accuracy trade-off, we surpass state of the art (SotA) results on popular semantic segmentation benchmarks: PASCAL VOC 2012 (val. set) and real-time semantic segmentation on Cityscapes, and CamVid.


Install the following packages:

git clone
cd hyperseg
conda env create -f hyperseg_env.yml
conda activate hyperseg
pip install -e .    # Alternatively add the root directory of the repository to PYTHONPATH.

Next, download the models and datasets:


Template Dataset Resolution mIoU (%) FPS Link
HyperSeg-L PASCAL VOC 512x512 80.6 (val) - download
HyperSeg-M CityScapes 1024x512 76.2 (val) 36.9 download
HyperSeg-S CityScapes 1536x768 78.2 (val) 16.1 download
HyperSeg-S CamVid 768x576 78.4 (test) 38.0 download
HyperSeg-L CamVid 1024x768 79.1 (test) 16.6 -

The models FPS was measured on an NVIDIA GeForce GTX 1080TI GPU.

Either download the models under <project root>/weights or adjust the model variable in the test configuration files.


Dataset # Images Classes Resolution Link
PASCAL VOC 10,582 21 up to 500x500 auto downloaded
CityScapes 5,000 19 2048x1024 download
CamVid 701 12 960x720 download

Either download the datasets under <project root>/data or adjust the data_dir variable in the configuration files.


To train the HyperSeg-M model on Cityscapes, set the exp_dir and data_dir paths in and run:

python configs/train/


Testing a model after training

For example testing the HyperSeg-M model on Cityscapes validation set:

python 'checkpoints/cityscapes/cityscapes_efficientnet_b1_hyperseg-m' \
-td "hyperseg.datasets.cityscapes.CityscapesDataset('data/cityscapes',split='val',mode='fine')" \
-it "seg_transforms.LargerEdgeResize([512,1024])"

Testing a pretrained model

For example testing the PASCAL VOC HyperSeg-L model using the available test configuration:

python configs/test/


  title={{HyperSeg}: Patch-wise Hypernetwork for Real-time Semantic Segmentation},
  author={Nirkin, Yuval and Wolf, Lior and Hassner, Tal},
  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},