Skip to content

jlam55555/intrinsic-dimension-projections

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

54 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

intrinsic-dimension-projections

Jonathan Lam & Richard Lee

See the paper and the presentation associated with this research.

Based on "Measuring the intrinsic dimension of objective landscapes" and its associated codebase.

Repository contents

  • keras_ext: main TF2/keras low-level architectural files (projection implementation)
  • models: model architectures
  • plots: sample plots used in report (scripts to generate these plots are not included)
  • runs: sample pickled runs (not a complete list of models used in the report)
  • scripts: scripts to automate model building and training

Usage

The primary model for training was MNIST, and the training loop is in scripts/mnist.py. There, you can adjust the model hyperparameters:

  • epochs: maximum number of epochs before termination (ignoring early stopping)
  • intrinsic_dims: iterable of intrinsic dimension sizes to train
  • initializers: initializers for the initial weight matrices theta_0
  • lrs: set of learning rates to train with
  • model_types: list of model types to train; can choose from "linear", "power", and "rff"
  • normalize_p: whether to normalize the output basis vectors in the projection matrix

General project goals

  • Apply other projection types (other than random linear mappings); in particular, try random fourier features (RFF) and power terms
  • Rewrite/upgrade/modernize to Python3/TF2 (e.g., type hinting for better IDE support, do more linting)
  • Include better documentation/explanations of terms than original code

Style guide

Placing a little bit of extra emphasis here because the codebase from the previous project was a little hard to decipher:

  • Attempt to adhere to PEP8 as best as possible (use IntelliJ linting/code refactor when possible)
  • Type hints in parameter lists/other initializations when possible to allow for maximum automated inferencing
  • Comment at beginning of each file to describe what the file is and defining relevant terms
  • Clear variable names (clarity at the cost of longer variable names)
  • Comments describing every function (except obvious cases and some overloaded methods) for maximum clarity

About

Experimenting with projection types in intrinsic-dimension paper

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 2

  •  
  •  

Languages