Skip to content

jlam55555/intrinsic-dimension-projections

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 

intrinsic-dimension-projections

Jonathan Lam & Richard Lee

See the paper and the presentation associated with this research.

Based on "Measuring the intrinsic dimension of objective landscapes" and its associated codebase.

Repository contents

  • keras_ext: main TF2/keras low-level architectural files (projection implementation)
  • models: model architectures
  • plots: sample plots used in report (scripts to generate these plots are not included)
  • runs: sample pickled runs (not a complete list of models used in the report)
  • scripts: scripts to automate model building and training

Usage

The primary model for training was MNIST, and the training loop is in scripts/mnist.py. There, you can adjust the model hyperparameters:

  • epochs: maximum number of epochs before termination (ignoring early stopping)
  • intrinsic_dims: iterable of intrinsic dimension sizes to train
  • initializers: initializers for the initial weight matrices theta_0
  • lrs: set of learning rates to train with
  • model_types: list of model types to train; can choose from "linear", "power", and "rff"
  • normalize_p: whether to normalize the output basis vectors in the projection matrix

General project goals

  • Apply other projection types (other than random linear mappings); in particular, try random fourier features (RFF) and power terms
  • Rewrite/upgrade/modernize to Python3/TF2 (e.g., type hinting for better IDE support, do more linting)
  • Include better documentation/explanations of terms than original code

Style guide

Placing a little bit of extra emphasis here because the codebase from the previous project was a little hard to decipher:

  • Attempt to adhere to PEP8 as best as possible (use IntelliJ linting/code refactor when possible)
  • Type hints in parameter lists/other initializations when possible to allow for maximum automated inferencing
  • Comment at beginning of each file to describe what the file is and defining relevant terms
  • Clear variable names (clarity at the cost of longer variable names)
  • Comments describing every function (except obvious cases and some overloaded methods) for maximum clarity

About

Experimenting with projection types in intrinsic-dimension paper

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages