InFi is a library for building input filters for trained models to be deployed, so as to improve the resource efficiency in the inference phase.

Installation

InFi has been tested on Python 3.7, TensorFlow 2.4, and MindSpore 1.5.

You need to install at least one of TensorFlow or MindSpore.

If you'd like to play with jupyter notebook examples, you need to install Jupyter-Lab.

Quick Start

InFi-Skip

We use the speech recognition workload on Hollywood2 dataset as an example to show how to train and evaluate InFi-Skip.

You can download the raw data (infi_dataset_hollywood2) from google drive.

The full code can be found in examples/.

First, we need to convert the extracted .wav audio data into spectrugram:

from datasets.hollywood2 import wav2spectrogram

def save_hollywood2_audio_spec(list_path, out_dir, root_path="infi_dataset_hollywood2/wav_audios/", postfix=".wav"):
    with open(list_path, "r") as fin:
        for line in fin.readlines():
            tmp = line.strip()
            wav_path = os.path.join(root_path, tmp+postfix)
            spec = wav2spectrogram(wav_path, FRAME_LENGTH=255, FRAME_STEP=128, SPEC_HEIGHT=500, SPEC_WIDTH=129)
            save_path = os.path.join(out_dir, f"{tmp}.npy")
            np.save(save_path, spec)

save_hollywood2_audio_spec("infi_dataset_hollywood2/list_train_audioclean.txt", "infi_dataset_hollywood2/spectrograms/")
save_hollywood2_audio_spec("infi_dataset_hollywood2/list_test.txt", "infi_dataset_hollywood2/spectrograms/")

Then we set the minimal number, say 2, of recognized words that a necessary inference requires:

from datasets.hollywood2 import redundancy_speech

MIN_LEN = 2
for split in ["train", "test"]:
    speech_data =  np.load(f"infi_dataset_hollywood2/inference_outputs/speech_{split}.npy", allow_pickle=True)
    res = list()
    for d in speech_data:
        p = redundancy_speech(d)
        if p <= MIN_LEN:
            res.append([0])
        else:
            res.append([1])
    res = np.array(res)
    redundancy_ratio = 1.0 - res.sum() / len(res)
    print(split, redundancy_ratio)
    np.save(f"infi_dataset_hollywood2/redundancy_labels/speech_wnum{MIN_LEN}_{split}.npy", res)

Now we have prepared the data for training and testing InFi-Skip. We can easily build and train InFi-Skip models via InFi APIs:

from datasets.hollywood2 import build_hollywood2_wavspec_dataset
from infi_tf import build_infi_skip, train_infi_skip

d_train = build_hollywood2_wavspec_dataset(list_path="infi_dataset_hollywood2/list_train_audioclean.txt",
                                           root_path="infi_dataset_hollywood2/spectrograms/",
                                           label_path="infi_dataset_hollywood2/redundancy_labels/speech_wnum2_train.npy")
d_test = build_hollywood2_wavspec_dataset(list_path="infi_dataset_hollywood2/list_test.txt",
                                         root_path="infi_dataset_hollywood2/spectrograms/",
                                         label_path="infi_dataset_hollywood2/redundancy_labels/speech_wnum2_test.npy")
infi = build_infi_skip(modality="audio", input_shape=(500,129,1), n_dense=200, n_layers=2, n_filters=32)
res = train_infi_skip(infi, d_train,
                      learning_rate=0.001, batch_size=32, epochs=20, val_data=d_test,
                      log_dir="logs/hollywood2_speech_wnum2/",
                      weight_dir="weights/hollywood2_speech_wnum2/")

Then we can evaluate the inference accuracy and filtering rate of trained InFi-Skip models:

infi = load_infi_skip("weights/hollywood2_speech_wnum2/20211207-145039-Epoch20.h5")
x_test = build_hollywood2_wavspec_dataset(list_path="../datasets/Hollywood2/list_test.txt",
                                          root_path="../datasets/Hollywood2/spectrograms/").batch(10)
y_pred = infi.predict(x_test)
y_true = np.load("../datasets/Hollywood2/redundancy_labels/speech_wnum2_test.npy")

def eval_skip(y_true, y_pred, threshold=0.5):
    wrong_count = 0
    filtered_count = 0
    for y1, y2 in zip(y_true, y_pred):
        if y1[0] == 1 and y2[0] < threshold:
            wrong_count += 1
        if y2[0] < threshold:
            filtered_count += 1
    
    total_num = len(y_true)
    filtered_rate = filtered_count / total_num
    label_acc = 1. - wrong_count / total_num
    
    return label_acc, filtered_rate

TN = 1.0 - y_true.sum() / len(y_true)
acc_list = []
r_list = []
for threshold in np.arange(0., 1.01, 0.01):
    acc, r = eval_skip(y_true, y_pred, threshold)
    acc_list.append(acc)
    r_list.append(r)

We can plot the results:

Another example using YOLOv3 object detector for vehicle counting is also in examples/video_yolov3* and the data for running the notebooks can be downloaded from google drive.

InFi-Reuse

An example of using InFi-Reuse on gender classification task can be found in examples/hollywood2_imgpair_preprocess.ipynb (preprocessing) and examples/hollywood2_gender_infi_reuse*. Data can be downloaded from google drive.

Citation

@inproceedings{yuan-infi,
  author = {Yuan, Mu and Zhang, Lan and He, Fengxiang and Tong, Xueting and Li, Xiang-Yang},
  title = {InFi: End-to-End Learnable Input Filter for Resource-Efficient Mobile-Centric Inference},
  year = {2022},
  isbn = {9781450391818},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3495243.3517016},
  doi = {10.1145/3495243.3517016},
  booktitle = {Proceedings of the 28th Annual International Conference on Mobile Computing And Networking},
  pages = {228–241},
  numpages = {14},
  location = {Sydney, NSW, Australia},
  series = {MobiCom '22}
}