Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Shrink Pt. 1: Sparsify #4

Open
3 tasks done
stephenjfox opened this issue Feb 15, 2019 · 1 comment
Open
3 tasks done

Shrink Pt. 1: Sparsify #4

stephenjfox opened this issue Feb 15, 2019 · 1 comment
Assignees
@stephenjfox
Labels
cleaning Use when cleaning up code, not necessarily changing function enhancement New feature or request
Projects
Official Alpha Launch

Comments

@stephenjfox
Copy link
Owner

stephenjfox commented Feb 15, 2019 •
edited

  • Prove that sparsify(tensor, threshold) performs as intended
  • Prove shrink on a given layer produces the correct count of neurons for a given layer
    • ... for all layers of a given nn.Module

I recommend an internal encapsulation to handle this relaying of information.

  • Simple: a named tuple for carrying the data
  • Complex: a small control class that makes focused decisions
@stephenjfox stephenjfox self-assigned this Feb 15, 2019
@stephenjfox stephenjfox added this to To do in Official Alpha Launch via automation Feb 15, 2019
This was referenced Feb 15, 2019
Closed
Open
@stephenjfox stephenjfox changed the title Unit test shrink() hierarchy Shrink Pt. 1: Sparsify Feb 17, 2019
@stephenjfox stephenjfox mentioned this issue Feb 17, 2019
Open
@stephenjfox stephenjfox added enhancement New feature or request cleaning Use when cleaning up code, not necessarily changing function labels Feb 17, 2019
stephenjfox added a commit that referenced this issue Feb 18, 2019
@stephenjfox
Attempt to test for #4
76e101d 
PyTorch's boolean comparison crap isn't useful and makes
  it a pain to test exact tensor values.
* Will resume later
@stephenjfox stephenjfox moved this from To do to In progress in Official Alpha Launch Feb 18, 2019
@stephenjfox
Copy link
Owner Author

Definition of "correct count of neurons":

  • The smallest, rounded-down value that can be computed
  • Example: given a Linear(5, 4)
    1. We have a 4x5 matrix (PyTorch represents the transpose internally for ease of computation)
    2. We compute the sparsity with morph.sparse.sparsify. This give us a float-as-percentage
    • In this example, 40%
    1. We can safely multiply the last dimension of our matrix by .6 -> 4x3 matrix.
    • No rounding necessary
    • BUT THIS CHANGES OUR INPUT SIZE. What do we do?
    1. Consider making 4x5 -> 2x5, as the alternative (3x5) has more parameters and we want to trim down as quickly as possible

Is this comment still relevant?

stephenjfox added a commit that referenced this issue Feb 19, 2019
@stephenjfox
Lay out skeleton for the true shrinking algo #4
ace72af
@stephenjfox stephenjfox mentioned this issue Feb 19, 2019
Merged
3 tasks
stephenjfox added a commit that referenced this issue Feb 19, 2019
@stephenjfox
Release 0.1.0/sparsify 2019 02 18 (#11)
57d0433 
* Attempt to test for #4

PyTorch's boolean comparison crap isn't useful and makes
  it a pain to test exact tensor values.
* Will resume later

* Skipping sparsify test

It's a painfully simple function that has worked every time
  I've used it.
- No it doesn't handle every edge case
+ Yes, it gets the job done and can be packaged for the general case

* Use instance `.nonzero()` instead of `torch.nonzero()`

* Fix "type-check" in layer inspectors

* WIP: Implement shrink() in terms of resize_layers()

It was as easy as I wanted it to be.
* The complexity is how to handle a given nested layer
  + Those will get implemented with a given feature
  - Need to program feature detection

TODO:
+ Implement the resizing on a layer-by-layer case, to
  make the shrinking a bit different
  + Instead of applying the data transformation uniformly,
    each layer gets
  + Those factors will be computed as 1 - percent_waste(layer)

* Lay out skeleton for the true shrinking algo #4

* shrink_layer() is simple

* Justification for giving Shrinkage a 'input_dimensions' property:

> The thought is that channel depth doesn't change the output dimensions for CNNs, and that's
  attribute we're concerned with in the convulotional case...
  * Linear layers only have two dimensions, so it's a huge deal there.
  * RNNs do linear things over 'timesteps', so it's a big deal there.
  * Residual/identity/skip-connections in CNNs need this.

> __It's decided__. The attribute stays
@stephenjfox stephenjfox moved this from In progress to Done in Official Alpha Launch Feb 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stephenjfox stephenjfox

Labels
cleaning Use when cleaning up code, not necessarily changing function enhancement New feature or request
Projects
Official Alpha Launch
  
Done
Milestone
No milestone
Development

No branches or pull requests
1 participant
@stephenjfox