TinyAgent: Easily applying neural networks inside systems.

TinyAgent is a tool to help experiment with machine learning for systems. TinyAgent generates C-based neural networks for inference in systems (Linux kernel, database kernel, middleware, etc. ), and trains the neural network with powerful userspace tools (TinyAgent is designed to read in parameters generated from PyTorch).

See motivation behind it.

---

As of now, TinyAgent has the following features:

1. Supports Gated Recurrent Unit (GRU, a variant of RNN) for sequence analysis. Supports Multi-layer Perceptron for fixed-length feature analysis. Supports ReLU/Tanh/Sigmoid activation function.
2. C-based implementation (compatible with compilation into Linux kernel).
3. Contains only four source files, ~ 500 lines of code.
4. Designed to read the model generated by PyTorch training and make the same inference.
5. Supports fully integer-based inference and floating-point-based inference.
6. No computational dependencies on other libraries. Floating-point-based inference requires math.h provided in userspace C library.

---

Usage

Computing

The computing part is defined in:

• For floating-point-based: tiny_agent_f.h and tiny_agent_f.c.
• For integer-based: tiny_agent.h and tiny_agent.c.

Create a GRU (Gated Recurrent Unit, an RNN model) unit and load the parameters:

struct gru_cell cell;
init_gru_cell(&cell, GRU_HIDDEN_SIZE, GRU_INPUT_SIZE);
cell.read(&cell, GRU_LOAD_PATH);

Clear the information of the hidden vector of the GRU cell:

cell.clear(&cell);

Fusing data into the hidden vector of GRU cell:

cell.inference(&cell, &DATA);

Delete the GRU cell:

cell.free(&cell);

Define a multi-layer perceptron with one hidden layer and sigmoid activation function, and load the parameters:

struct fc_layer layer1, layer2;
init_fc_layer(&layer1, FC1_INPUT_SIZE, FC2_INPUT_SIZE, SIGMOID);
init_fc_layer(&layer2, FC2_INPUT_SIZE, PREDICTION_SIZE, NONE);

struct mlp nn;
init_mlp(&nn);
nn.add_layer(&nn, &layer1);
nn.add_layer(&nn, &layer2);
nn.read(&nn, MLP_LOAD_PATH);

Inference data with the multi-layer perceptron:

nn.inference(&nn, cell.hidden);

Delete the multi-layer perceptron:

nn.free(&nn);

I/O

The main I/O involved in TinyAgent includes local file system read/write and heap memory request/release. Depending on the platform on which they are compiled, the implementation of these interfaces may be different. It's definded in tiny_agent_io.c and tiny_agent_io.h.

We provide implementations for three compilation environments:

1. Userspace C.
2. Linux kernel with version 4.9. You might need it if you want to experiment on Nvidia Jetson Board. Theoritically it works for kernel version <= 5.4.
3. Linux kernel with version 5.15. You might need it if you want to experiment on Raspberry Pi. Theoritically it works for kernel version > 5.4.

Integer-based computing

The integer-based inference is done through quantization technique. See how it is designed and implemented in TinyAgent.

Demo

We provide a demo to show how to train a neural network using PyTorch and use TinyAgent to read the trained network and make inference based on floating-point/integer.

Citation

A paper describing a deep reinforcement learning-driven CPU frequency controller built on TinyAgent running under the Linux CPUFreq framework will be put up soon, please read/cite it if you find it interesting : )