A manifold optimization library for deep learning
Switch branches/tags
BMmctorch Jorghi12-PytorchROCmRemove_Deps Jorghi12-patch-2 Jorghi12-patch-3 Jorghi12-patch-4 Jorghi12-patch-5-1 Jorghi12-patch-5 Jorghi12-patch-6 Jorghi12-patch-7 Jorghi12-transpiler PyRocm_removing_deps a5d7abe always_scriptmodule anderspapitto-patch-1-1 anderspapitto-patch-1 bamdevm-patch-1 bamdevm-patch-2 batch_mm_t cache_vars caffe2-docker-image-rebuild circleci_all_commits circleci_disable_all_jobs circleci_flaky_tests circleci cpp cudastaticfixes docker/rocm-1.8.2 docker/rocm-update export-D9526248 export-D9526650 export-D9526737 export-D9539945 export-D9540025 export-D9545248 export-D9545704 export-D9557315 export-D9561478 export-D9561802 export-D9562197 export-D9562312 export-D9562467 export-D9563464 export-D9563753 export-D9564206 export-D9564516 export-D9578397 export-D9578398 export-D9578399 export-D9578734 export-D9579371 export-D9581560 export-D9583630 export-D9583699 export-D9585627 export-D9613800 export-D9613897 export-D9614321 export-D9617237 export-D9623916 export-D9631619 export-D9634750 export-D9634904 export-D9635105 export-D9635292 export-D9644700 export-D9644899 export-D9646190 export-D9648570 export-D9648830 export-D9652088 export-D9652089 export-D9654871 export-D9656548 export-D9657449 export-D9663476 export-D9666612 export-D9670493 export-D9676205 export-D9694097 export-D9694326 export-D9694327 export-D9694918 export-D9697878 export-D9724805 export-D9727134 export-D9727532 export-D9728631 export-D9731326 export-D9756666 export-D9757935 export-D9763422 export-D9763423 export-D9763424 export-D9771708 export-D9775191 export-D9778042 export-D9778043 export-D9779821 export-D9790187 export-D9806425 export-D9810823 export-D9811028 export-D9813544 export-D9813742 export-D9814536 export-D9823457 export-D9826209 export-D9830460 export-D9831230 export-D9831384 export-D9833361 export-D9841660 export-D9847835 export-D9847859 export-D9882726 export-D9884177 export-D9884563 export-D9924348 export-D9968041 export-D9968320 export-D9977058 export-D9977505 export-D9977654 export-D9979976 export-D9980641 ext_test_fix ezyang-patch-2 ezyang-patch-3 ezyang-patch-4 ezyang/retry-type-id-core ezyang/rocm-docker-update fast_dp fb-config fbsync gloo_dedup halfconv jerryzh168-patch-1-1 jerryzh168-patch-1 jit_frontend known-good master mctorch merge_variable_tensor nn_c_port random_device remove_time scalar_type simple_engine soumith-patch-1 ssnl-9348 suo/mutable suo/mutable2 suo/tensor_empty tensor-merge test10 testing/full-caffe2 tmp_enable_scalars v0.4.1 weak_tensor weak_tracing win_py2.7
Nothing to show
Clone or download
Pull request Compare This branch is 21 commits ahead, 2683 commits behind pytorch:master.
Latest commit 53ffe61 Nov 22, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci Dummy CircleCI config. (pytorch#9537) Jul 18, 2018
.github updated the environment collection script URL to the raw version on G… Apr 25, 2018
.jenkins ROCm contributions week 29 (pytorch#9653) Aug 2, 2018
aten weak_intrusive_ptr (pytorch#10038) Aug 3, 2018
binaries Remove template parameter from Tensor (pytorch#9939) Jul 27, 2018
caffe Fix public protobuf interface (pytorch#1961) Feb 21, 2018
caffe2 Fix IDEEP Copys (pytorch#10104) Aug 3, 2018
cmake ROCm contributions week 29 (pytorch#9653) Aug 2, 2018
conda Fixing missing PyCObject_Type bug (pytorch#8467) Jun 14, 2018
docker ROCm contributions week 29 (pytorch#9653) Aug 2, 2018
docs Adding katex rendering of equations, and required edits to equations. ( Aug 2, 2018
examples adding PCA example Sep 22, 2018
modules Remove template parameter from Tensor (pytorch#9939) Jul 27, 2018
scripts Fix Caffe2 with ATen conda build failure (pytorch#10020) Jul 30, 2018
test Make list literals construct ListType (pytorch#10193) Aug 3, 2018
third_party Update onnx to onnx/onnx@2a3a226 (pytorch#10167) Aug 2, 2018
tools Remove capture specifiers in register_aten_ops when they're not neede… Aug 2, 2018
torch fixing conditions in optimizers to pass all existing tests Sep 22, 2018
.clang-format Updates to .clang-format (pytorch#7683) May 18, 2018
.clang-tidy Match parameter names and = default (pytorch#9737) Jul 30, 2018
.dockerignore Add .dockerignore. (pytorch#3333) Oct 28, 2017
.gitattributes add .gitattributes for EOL conversion. (pytorch#9813) Aug 1, 2018
.gitignore Remove CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS and fix CUDA 8 build on Windo… Jul 23, 2018
.gitmodules Temporarily remove TBB (pytorch#8255) Jun 18, 2018
.travis.aten.yml [build] Have PyTorch depend on minimal libcaffe2.so instead of libATe… May 24, 2018
.travis.yml Travis CI: Run flake on Python 2.7 and 3.7 (pytorch#9953) Jul 27, 2018
CMakeLists.txt Prepare THC for first class scalars (0-dimensional tensors). Aug 1, 2018
LICENSE renaming existing pytorch files and adding new README, updating LICEN… Sep 22, 2018
Makefile Fix python support problems caused by building script errors. Apr 25, 2017
README.md Update README.md Nov 22, 2018
mypy-README.md Change output_declarations in function_wrapper.py to be a NamedTuple (p… Feb 23, 2018
mypy-files.txt Change output_declarations in function_wrapper.py to be a NamedTuple (p… Feb 23, 2018
mypy.ini Change output_declarations in function_wrapper.py to be a NamedTuple (p… Feb 23, 2018
pytorch-CITATION renaming existing pytorch files and adding new README, updating LICEN… Sep 22, 2018
pytorch-CODEOWNERS renaming existing pytorch files and adding new README, updating LICEN… Sep 22, 2018
pytorch-CONTRIBUTING.md renaming existing pytorch files and adding new README, updating LICEN… Sep 22, 2018
pytorch-NOTICE renaming existing pytorch files and adding new README, updating LICEN… Sep 22, 2018
pytorch-README.md renaming existing pytorch files and adding new README, updating LICEN… Sep 22, 2018
requirements.txt Change output_declarations in function_wrapper.py to be a NamedTuple (p… Feb 23, 2018
setup.py Strings lexing, parsing, implementation in print (pytorch#9324) Aug 2, 2018
setup_caffe2.py Re-Enable ATen in C2 in integration builds to test ONNX ATen conversions Jul 31, 2018
tox.ini Add pyHIPIFY scripts needed for ROCm transpilation to PyTorch (pytorc… Jul 11, 2018
ubsan.supp Suppress the vptr warning in ubsan (pytorch#9909) Jul 27, 2018

README.md

McTorch, a manifold optimization library for deep learning

McTorch is a Python library that adds manifold optimization functionality to PyTorch.

McTorch:

  • Leverages tensor computation and GPU acceleration from PyTorch.
  • Enables optimization on manifold constrained tensors to address nonlinear optimization problems.
  • Facilitates constrained weight tensors in deep learning layers.

Sections:

More about McTorch

McTorch builds on top of PyTorch and supports all PyTorch functions in addition to Manifold optimization. This is done to ensure researchers and developers using PyTorch can easily experiment with McTorch functions. McTorch's manifold implementations and optimization methods are derived from the Matlab toolbox Manopt and the Python toolbox Pymanopt.

Using McTorch for Optimization

  1. Initialize Parameter - McTorch manifold parameters are same as PyTorch parameters (torch.nn.Parameter) and requires just addition of one property to parameter initialization to constrain the parameter values.
  2. Define Cost - Cost function can be any PyTorch function using the above parameter mixed with non constrained parameters.
  3. Optimize - Any optimizer from torch.optim can be used to optimize the cost function using same functionality as any PyTorch code.

PCA Example

import torch
import torch.nn as nn

# Random data with high variance in first two dimension
X = torch.diag(torch.FloatTensor([3,2,1])).matmul(torch.randn(3,200))

# 1. Initialize Parameter
manifold_param = nn.Parameter(manifold=nn.Stiefel(3,2))

# 2. Define Cost - squared reconstruction error
def cost(X, w):
    wTX = torch.matmul(w.transpose(1,0), X)
    wwTX = torch.matmul(w, wTX)
    return torch.sum((X - wwTX)**2)

# 3. Optimize
optimizer = torch.optim.Adagrad(params = [manifold_param], lr=1e-2)

for epoch in range(30):
    cost_step = cost(X, manifold_param)
    print(cost_step)
    cost_step.backward()
    optimizer.step()
    optimizer.zero_grad()

Using McTorch for Deep Learning

Multi Layer Perceptron Example

import torch
import torch.nn as nn
import torch.nn.functional as F

# a torch module using constrained linear layers
class ManifoldMLP(nn.Module):
    def __init__(self):
        super(ManifoldMLP, self).__init__()
        self.layer1 = nn.Linear(in_features=28*28, out_features=100, weight_manifold=nn.Stiefel)
        self.layer2 = nn.Linear(in_features=100, out_features=100, weight_manifold=nn.PositiveDefinite)
        self.output = nn.Linear(in_features=100, out_features=10, weight_manifold=nn.Stiefel)

    def forward(self, x):
        x = F.relu(self.layer1(x))
        x = F.relu(self.layer2(x))
        x = F.log_softmax(self.output(x), dim=0)
        return x

# create module object and compute cost by applying module on inputs
mlp_module = ManifoldMLP()
cost = mlp_module(inputs)

Functionality Supported

This would be an ever increasing list of features. McTorch currently supports:

Manifolds

  • Stiefel
  • Positive Definite

All manifolds support k multiplier as well.

Optimizers

  • SGD
  • Adagrad
  • ConjugateGradient

Layers

  • Linear
  • Conv1d, Conv2d, Conv3d
  • Conv1d_transpose, Conv2d_transpose, Conv3d_transpose

Installation

This is same as PyTorch installation from source. I have tried the installation with Python 3.5.6 and it will be preferred version for installation.

If you are installing from source, we highly recommend installing an Anaconda environment.

  • Download anaconda installer from here - Anaconda and follow GUI steps to install
  • Create an environment, from anaconda console prompt: conda create -n myenv python=3.5.6
  • Activate the environment: source activate myenv

Linux

source activate myenv
git clone --recursive https://github.com/mctorch/mctorch
cd mctorch
python setup.py install

Mac OS

source activate myenv
git clone --recursive https://github.com/mctorch/mctorch
cd mctorch
MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py install

Installation FAQs

  1. ModuleNotFoundError: No module named 'torch._C': After installation when using McTorch make sure you have activate the conda environment and not in the same folder as McTorch as import torch tries to refer to the module in torch folder and throws an error.

For other os and optional dependencies go through Installation.

Release and Contribution

McTorch is currently under development and any contributions, suggestions and feature requests are welcome. We'd closely follow PyTorch stable versions to keep the base updated and will have our own versions for other additions.

McTorch is released under the open source 3-clause BSD License.

Team

Reference

Please cite 1 if you found this code useful.

McTorch, a manifold optimization library for deep learning

[1] M. Meghawanshi, P. Jawanpuria, A. Kunchukuttan, H. Kasai, and B. Mishra, McTorch, a manifold optimization library for deep learning

@techreport{meghwanshi2018mctorch,
  title={McTorch, a manifold optimization library for deep learning},
  author={Meghwanshi, Mayank and Jawanpuria, Pratik and Kunchukuttan, Anoop and Kasai, Hiroyuki and Mishra, Bamdev},
  institution={arXiv preprint arXiv:1810.01811},
  year={2018}
}