Element-wise Modulation of Random Matrices for Efficient Neural Layers

This repository contains the code, experiments, and supplementary materials for the paper "Element-wise Modulation of Random Matrices for Efficient Neural Layers."

Author: Maksymilian Szorc

Minimal PyTorch reference implementation of the PRP layer from the accompanying paper.

y = ((x * alpha) @ P) * weight + bias

P is a fixed random projection; only alpha, weight, and bias are trainable.

Install

git clone https://github.com/UniversalComputingResearch/prp.git
cd prp
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install .

The package itself requires PyTorch. torchvision is needed by the two dataset examples and is included in requirements.txt.

Examples

python -m PRP.fmnist_classification --epochs 20
python -m PRP.mnist_autoencoder --epochs 20

Both scripts download their data into data/, set a default seed, and expose all experiment settings through --help.

For example, to omit bias, keep projections out of checkpoints, and use different reproducible projection seeds per PRP layer:

python -m PRP.fmnist_classification --no-bias --no-persistent --projection-seed 123

The scripts also expose --dtype float64|float32|float16|bfloat16. With no --projection-seed, the global --seed deterministically allocates distinct seeds to successive layers.

Use

from PRP import ParametrizedRandomProjection

layer = ParametrizedRandomProjection(784, 512, projection_type="orthogonal")
output = layer(inputs)

The fixed projection is a buffer: it moves with the model, but is never optimized. Pass bias=False to omit the bias parameter. Pass persistent=False, seed=123 to leave the (possibly large) projection out of checkpoints while regenerating it deterministically when the module state is loaded. Use layer.to(dtype=torch.float16) or dtype=torch.bfloat16 to use reduced precision consistently with a dense layer.