Official repository for the paper "PonderBayes: Rationally Pondering Neural Networks" (Not submitted to peer review, but available for a browse in this repository)
Details such as python and package versions can be found in the generated pyproject.toml and poetry.lock files.
We recommend using an environment manager such as conda. After setting up your environment with the correct python version, please proceed with the installation of the required packages
For poetry users, getting setup is as easy as running
poetry install
We also provide a requirements.txt file for pip users who do not wish to use poetry. In this case, simply run
pip install -r requirements.txt
This requirements.txt file is generated by running the following
sh gen_pip_reqs.sh
Some additional requirements are necessary for running notebooks/accuracies.ipynb, which is the notebook we use for generating figures and tables:
- A complete TeXLive/MacTeX installation is required, since we make use of a TeX backend for our figure plots.
- Our pretrained and pretested checkpoints and logs, available on
The Internet Archive at
this link
- please unzip the checkpoints and place them as the
models/directory under the root of the project
- please unzip the checkpoints and place them as the
├── LICENSE
├── README.md <- The top-level README for developers using this project.
├── data/
│ ├── interim/ <- Intermediate data that has been transformed.
│ ├── processed/ <- The final, canonical data sets for modeling.
│ └── raw/ <- The original, immutable data dump.
├── models/ <- Trained and serialized models and logs
├── notebooks/ <- Jupyter notebooks.
│ └── accuracies.ipynb <- Notebook for generating figures and tables
├── reports/ <- Generated analysis as HTML, PDF, LaTeX, etc.
├── lisa/ <- LISA (slurm compute) scripts, jobs, config.
├── pyproject.toml <- project metadata, handled by poetry.
├── poetry.lock <- resolving and locking dependencies, handled by poetry.
├── requirements.txt <- for non-poetry users.
├── gen_pip_reqs.sh <- for generating the pip requirements.txt file
└── ponderbayes/ <- Source code for use in this project.
├── __init__.py <- Makes src a Python module
├── data/ <- Scripts to download or generate data
├── models/ <- Model definitions
├── run/ <- scripts to train, evaluate and use models
├── utils.py <- miscellaneous utils
└── visualization/ <- Scripts for visualization
The project structure is largely based on the cookiecutter data-science template. This is purposely opinionated so that paths align over collaborators without having to edit config files. Users may find the cookiecutter data-science opinions page, of relevance
The top level data/ and models/ directory are in version control only to
show structure. Their contents will not be committed and are ignored via
.gitignore.
For training, refer to ponderbayes/run/train.py:
usage: train.py [-h] [-s SEED] [--disable-logging]
[--model {pondernet,groupthink,RGT,lambdaGT,aRGT}]
[-c CHECKPOINT] [--n-elems N_ELEMS] [--n-hidden N_HIDDEN]
[--max-steps MAX_STEPS] [--lambda-p LAMBDA_P] [--beta BETA]
[--progress-bar] [--n-train-samples N_TRAIN_SAMPLES]
[--n-eval-samples N_EVAL_SAMPLES] [--mode MODE]
[--batch-size BATCH_SIZE] [--num-workers NUM_WORKERS]
[--early_stopping] [--val-check-interval VAL_CHECK_INTERVAL]
[--n-iter N_ITER] [--ensemble-size ENSEMBLE_SIZE]
Train a model
optional arguments:
-h, --help show this help message and exit
-s SEED, --seed SEED The seed to use for random number generation
--disable-logging Disable logging
--model {pondernet,groupthink,RGT,lambdaGT,aRGT}
What model variant to use
-c CHECKPOINT, --checkpoint CHECKPOINT
path to a checkpoint from which to resume training
from
--n-elems N_ELEMS Number of elements in the parity vectors
--n-hidden N_HIDDEN Number of hidden elements in the reccurent cell
--max-steps MAX_STEPS
Maximum number of pondering steps
--lambda-p LAMBDA_P Geometric prior distribution hyperparameter
--beta BETA Regularization loss coefficient
--progress-bar whether to show the progress bar
--n-train-samples N_TRAIN_SAMPLES
The number of training samples to comprising the
dataset
--n-eval-samples N_EVAL_SAMPLES
The number of training samples to comprising the
dataset
--mode MODE Whether to perform 'interpolation' or 'extrapolation'
--batch-size BATCH_SIZE
Batch size
--num-workers NUM_WORKERS
The number of workers
--early_stopping Whether to use early stopping
--val-check-interval VAL_CHECK_INTERVAL
Evaluate every x amount of steps, as opposed to every
epoch
--n-iter N_ITER Number of training steps to use
--ensemble-size ENSEMBLE_SIZE
Number of models to ensemble
For testing, refer to ponderbayes/run/test.py:
usage: test.py [-h] [-s SEED] -c CHECKPOINT [--progress-bar]
[--n-test-samples N_TEST_SAMPLES] [--batch-size BATCH_SIZE]
[--num-workers NUM_WORKERS]
Test a pondernet checkpoint
optional arguments:
-h, --help show this help message and exit
-s SEED, --seed SEED The seed to use for random number generation
-c CHECKPOINT, --checkpoint CHECKPOINT
path (relative to root) to a checkpoint to evaluate
--progress-bar whether to show the progress bar
--n-test-samples N_TEST_SAMPLES
The number of testing samples to comprise the dataset
--batch-size BATCH_SIZE
Batch size
--num-workers NUM_WORKERS
The number of workers