New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce network size after training #184

Open
ibmua opened this Issue Mar 6, 2015 · 10 comments

Comments

Projects
None yet
5 participants
@ibmua
Contributor

ibmua commented Mar 6, 2015

After I train a net, it grows in size and that is especially easy to notice with large batches. I would like to reduce the size to original to only include weights before saving, or at least to non-batch size. How can I do it?

Also, it would be a great idea, to make a standart function for preparing a net for saving that would include at least that thing and removal of descriptors.

@nicholas-leonard

This comment has been minimized.

Show comment
Hide comment
@nicholas-leonard

nicholas-leonard Mar 7, 2015

Member

Check out https://github.com/Atcold/net-toolkit/tree/master . I agree that it would be nice to include this into the core nn repo.

Member

nicholas-leonard commented Mar 7, 2015

Check out https://github.com/Atcold/net-toolkit/tree/master . I agree that it would be nice to include this into the core nn repo.

@ibmua

This comment has been minimized.

Show comment
Hide comment
@ibmua

ibmua Mar 8, 2015

Contributor

Great, thanks! This one should definitely get included into standart NN package - almost everybody who uses NN needs those more than half of the actual NN package functions anyway.

Contributor

ibmua commented Mar 8, 2015

Great, thanks! This one should definitely get included into standart NN package - almost everybody who uses NN needs those more than half of the actual NN package functions anyway.

@Atcold

This comment has been minimized.

Show comment
Hide comment
@Atcold

Atcold Mar 25, 2015

Contributor

There is an easier way.
Define model, clone out cleanModel, train model, getParameters from model and copy onto cleanModel's parameters.

Contributor

Atcold commented Mar 25, 2015

There is an easier way.
Define model, clone out cleanModel, train model, getParameters from model and copy onto cleanModel's parameters.

@nicholas-leonard

This comment has been minimized.

Show comment
Hide comment
@nicholas-leonard

nicholas-leonard Mar 26, 2015

Member

@Atcold so true! Hadn't thought about it that way. #200 would actually make that quite easy.

Member

nicholas-leonard commented Mar 26, 2015

@Atcold so true! Hadn't thought about it that way. #200 would actually make that quite easy.

@Atcold

This comment has been minimized.

Show comment
Hide comment
@Atcold

Atcold Mar 27, 2015

Contributor

Alright @nicholas-leonard, and this

clone = mlp:clone() -- this copies all the parameters and gradParameters
clone:share(mlp,"weight","gradWeight","bias","gradBias") -- this deletes and replaces them

should go into the manual (since I never used shared stuff before and it looks like it would make things easier).

Contributor

Atcold commented Mar 27, 2015

Alright @nicholas-leonard, and this

clone = mlp:clone() -- this copies all the parameters and gradParameters
clone:share(mlp,"weight","gradWeight","bias","gradBias") -- this deletes and replaces them

should go into the manual (since I never used shared stuff before and it looks like it would make things easier).

@nicholas-leonard

This comment has been minimized.

Show comment
Hide comment
@nicholas-leonard
Member

nicholas-leonard commented Apr 1, 2015

Noted.

@Atcold

This comment has been minimized.

Show comment
Hide comment
@Atcold

Atcold Apr 2, 2015

Contributor

Great.

Contributor

Atcold commented Apr 2, 2015

Great.

@mys007

This comment has been minimized.

Show comment
Hide comment
@mys007

mys007 Dec 11, 2015

Contributor

I would like to revive this issue with the following suggestion: I propose each module have a special function sanitize() which would clear all temporary variables, including outputs and gradInputs. All other ways of "sanitizing" the model are just hacks which don't handle everything or clean too much; only modules itself can know what's appropriate (encapsulation).
One could even have this in write() method, i.e. serialization would do this automatically, although this might introduce side effects (when serializing to clone or if one wants to really save everything for some reason); nevertheless, in cudnn this is used.
@soumith Could it even get to the Roadmap:)?

Contributor

mys007 commented Dec 11, 2015

I would like to revive this issue with the following suggestion: I propose each module have a special function sanitize() which would clear all temporary variables, including outputs and gradInputs. All other ways of "sanitizing" the model are just hacks which don't handle everything or clean too much; only modules itself can know what's appropriate (encapsulation).
One could even have this in write() method, i.e. serialization would do this automatically, although this might introduce side effects (when serializing to clone or if one wants to really save everything for some reason); nevertheless, in cudnn this is used.
@soumith Could it even get to the Roadmap:)?

@soumith

This comment has been minimized.

Show comment
Hide comment
@soumith

soumith Dec 11, 2015

Member

@mys007 Sergey will open a PR for this shortly, he wrote this feature and wrote unit tests when he was interning at Facebook.

Member

soumith commented Dec 11, 2015

@mys007 Sergey will open a PR for this shortly, he wrote this feature and wrote unit tests when he was interning at Facebook.

@mys007

This comment has been minimized.

Show comment
Hide comment
@mys007

mys007 Dec 11, 2015

Contributor

I call this programming by thought:).

Contributor

mys007 commented Dec 11, 2015

I call this programming by thought:).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment