Skip to content

kfirgoldberg/FUN

Repository files navigation

Rethinking FUN: Frequency-Domain Utilization Network

Open In Colab

The search for efficient neural network architectures has gained much focus in recent years, where modern architectures focus not only on accuracy but also on inference time and model size. Here, we present FUN, a family of novel Frequency-domain Utilization Networks. These networks utilize the inherent efficiency of the frequency-domain by working directly in that domain, represented with the Discrete Cosine Transform. Using modern techniques and building blocks such as compound-scaling and inverted-residual layers we generate a set of such networks allowing one to balance between size, latency and accuracy while outperforming competing RGB-based models. Extensive evaluations verifies that our networks present strong alternatives to previous approaches. Moreover, we show that working in frequency domain allows for dynamic compression of the input at inference time without any explicit change to the architecture.

Description

Official PyTorch implementation of "Rethinking FUN: Frequency-domain Utilization Networks".

Based on the implementation by Ross Wightman

Recent Updates

Getting Started

Installation

  • Clone this repo:
git clone https://github.com/kfir99/FUN.git
cd FUN
  • Install a conda environment
conda create -n torch-env
conda activate torch-env
conda install -c pytorch pytorch torchvision cudatoolkit=10.2 jpeg2dct
conda install pyyaml

Pretrained Models

Pretrained models are available for download

Path Accuracy # Parameters (M) FPS (V-100, batch size = 1)
eFUN 77 4.2 124
eFUN-L 78.8 6.2 101
eFUN-S 75.6 3.4 132
eFUN-S+ 73.3 2.5 145

Training

Preparing your data

Training and validation data should be organized in the following structure:

* data_dir
    * train
        * class_name_a
            * images
        * class_name_b
            * images
            
            .
        
            .
        
            .
    * validation
        * class_name_a
            * images
        * class_name_b
            * images
        
            .
        
            .
        
            .

Command line

./distributed_train.sh \
 <number of abailable GPUs>\
 <data_dir>\
 --output <desired output path>\
 --dct\ 
 --model <efun/efun_l/efun_s/efun_s_plus>\
 --drop-path <0.2/0.2/0.2/0.3>\
 --no-prefetcher\
 -b 128\
 --sched step\
 --epochs 450\
 --decay-epochs 2.4\
 --decay-rate .97\
 --opt rmsproptf\
 --opt-eps .001\
 -j 8\
 --warmup-lr 1e-6\
 --weight-decay 1e-5\
 --drop 0.2\ 
 --model-ema\
 --model-ema-decay 0.9999\ 
 --remode pixel\
 --reprob 0.2\
 --lr .048

Testing

Preparing your data

Training and validation data should be organized in the following structure:

* data_dir
    * class_name_a
        * images
    * class_name_b
        * images
        
        .
    
        .
    
        .

Command line

python validate.py\
 <path to test data>\
 --model <efun/efun_l/efun_s/efun_s_plus>\
 --checkpoint <path to trained model>\
 --dct\
 --no-prefetcher\
 -j 32\
 -b 256

Repository structure

Files added specifically for eFUN are marked in bold

Path Description
FUN Repository root folder
├  convert Folder containing code for conversion from MXNET weights
├  docs Folder containing documentation of the results and graphs use in the repository
├  notebooks Folder with runnable notebooks for experimenting with the dataset
│  └  eFUN_inference_playground.ipynb A basic notebook to run inference on some dataset using a pretrained eFUN model
├  results Folder with reported results for original TIMM repository models
│  tests Folder with some tests for original TIMM repository models
│  timm Folder with actual TIMM pip package modules
│  ├  data Folder with implementation of all data-related functions
│  └  rgb2dct.py implementation of conversion from RGB image to its DCT representation
│  ├  loss Folder with implementations of different loss functions
│  ├  models Folder with implementation of all models provided in this repository
│  └  eFUN.py Implementation of all eFUN models
│  ├  optim Folder with implementations of different optimizers
│  ├  scheduler Folder with implementations of different training schedulers
│  ├  utils Folder with implementations of utility functions

TODOs

  • sota-bench integration

Citation

About

Official implementation of the FUN models

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published