intrinsic-dimension-projections

Jonathan Lam & Richard Lee

See the paper and the presentation associated with this research.

Based on "Measuring the intrinsic dimension of objective landscapes" and its associated codebase.

Repository contents

• keras_ext: main TF2/keras low-level architectural files (projection implementation)
• models: model architectures
• plots: sample plots used in report (scripts to generate these plots are not included)
• runs: sample pickled runs (not a complete list of models used in the report)
• scripts: scripts to automate model building and training

Usage

The primary model for training was MNIST, and the training loop is in scripts/mnist.py. There, you can adjust the model hyperparameters:

• epochs: maximum number of epochs before termination (ignoring early stopping)
• intrinsic_dims: iterable of intrinsic dimension sizes to train
• initializers: initializers for the initial weight matrices theta_0
• lrs: set of learning rates to train with
• model_types: list of model types to train; can choose from "linear", "power", and "rff"
• normalize_p: whether to normalize the output basis vectors in the projection matrix

General project goals

• Apply other projection types (other than random linear mappings); in particular, try random fourier features (RFF) and power terms
• Rewrite/upgrade/modernize to Python3/TF2 (e.g., type hinting for better IDE support, do more linting)
• Include better documentation/explanations of terms than original code

Style guide

Placing a little bit of extra emphasis here because the codebase from the previous project was a little hard to decipher:

• Attempt to adhere to PEP8 as best as possible (use IntelliJ linting/code refactor when possible)
• Type hints in parameter lists/other initializations when possible to allow for maximum automated inferencing
• Comment at beginning of each file to describe what the file is and defining relevant terms
• Clear variable names (clarity at the cost of longer variable names)
• Comments describing every function (except obvious cases and some overloaded methods) for maximum clarity