Hierarchical Neural Memory Network

This repo is a PyTorch implementation of HMNet proposed in our paper: Hierarchical Neural Network for Low Latency Event Processing.

Results and models

The pre-trained weights are released under the Creative Commons BY-SA 4.0 License.

DSEC-Semantic (Semantic Segmentation)

Model	size	mIoU [%]	latency V100 [ms]	latency V100 x 3 [ms]	weights
HMNet-B1	640 x 440	51.2	7.0	-	github
HMNet-L1	640 x 440	55.0	10.5	-	github
HMNet-B3	640 x 440	53.9	9.7	8.0	github
HMNet-L3	640 x 440	57.1	13.9	11.9	github

GEN1 (Object Detection)

Model	size	mAP [%]	latency V100 [ms]	latency V100 x 3 [ms]	weights
HMNet-B1	304 x 240	45.5	4.6	-	github
HMNet-L1	304 x 240	47.0	5.6	-	github
HMNet-B3	304 x 240	45.2	7.0	5.9	github
HMNet-L3	304 x 240	47.1	7.9	7.0	github

MVSEC day1 (Monocular Depth Estimation)

Model	size	AbsRel	RMS	RMSElog	latency V100 [ms]	latency V100 x 3 [ms]	weights
HMNet-B1	346 x 260	0.385	9.088	0.509	2.4	-	github
HMNet-L1	346 x 260	0.310	8.383	0.393	4.1	-	github
HMNet-B3	346 x 260	0.270	7.101	0.332	5.0	4.1	github
HMNet-L3	346 x 260	0.254	6.890	0.319	6.9	5.4	github
HMNet-B3 w/ RGB	346 x 260	0.252	6.972	0.318	5.4	4.1	github
HMNet-L3 w/ RGB	346 x 260	0.230	6.922	0.310	7.1	5.4	github

Requirements

• PyTorch >= 1.12.1
• torch_scatter
• timm
• hdf5plugin

Installation

Create a new conda environment

conda create -n hmnet python=3.7
conda activate hmnet

Install dependencies

pip install -r requirements.txt
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch
pip install torch-scatter -f https://data.pyg.org/whl/torch-1.12.1+cu113.html
pip install timm

Experiments

Please see the instructions on each task page.

• Semantic Segmentation
• Object Detection
• Monocular Depth Estimation

License

The majority of this project is licensed under BSD 3-clause License. However, some code (psee_evaluator.py, coco_eval.py, det_head_yolox.py) is available under the Apache 2.0 license. The pre-trained weights are released under the Creative Commons BY-SA 4.0 License.

Acknowledgments

This work is based on results obtained from a project commissioned by the New Energy and Industrial Technology Development Organization (NEDO).