Updating Stability Analysis + Migration to New Module #56
Conversation
| ## Reproducible Analysis | ||
|
|
||
| To reproduce the results of this analysis perform the following: | ||
|
|
There was a problem hiding this comment.
what will this reproducible analysis output? This is the reconstruction error?
There was a problem hiding this comment.
This is specifically for the stability analysis. I will make that clear in the next commit, thanks!
| xlab("Z Dimension") + | ||
| ylab("SVCCA Mean Similarity") + | ||
| scale_fill_manual(name = "Algorithm", | ||
| values = c("#e41a1c", |
There was a problem hiding this comment.
Is it worth creating a lookup table with colors -- HEX code as you've done before?
There was a problem hiding this comment.
Yeah... This is probably worth doing. It would help to ensure consistency. I will add to an issue
| Following SVD, the algorithm performs canonical correlation analysis (CCA) on the SVD components to align matching components and extract correlations between them. | ||
| SVCCA is used to compare two different representations, but can be applied to representations of different dimensions. | ||
| We use SVCCA to extract a single value representing how similar two representations are to each other. | ||
| This value is the mean of the SVCCA correlation estimates across all dimensions compared. |
There was a problem hiding this comment.
Correct me if I'm wrong - my understanding of CCA it's a dimensionality reduction method that creates a mapping to a low dimensional space that maximizes the correlation between X and Y. Not exactly sure how you're using this latent space projection to assess the correlation between features.
Also is there a reason you're using SVCCA? I'm not as familiar with alternatives, but was curious what other people have done
There was a problem hiding this comment.
Not exactly sure how you're using this latent space projection to assess the correlation between features.
We're using CCA to identify common features between neural network representations (in this case, weight matrices from different compression algorithms). So, how correlated are the learned features to one another.
SVCCA was shown to have some nice properties here. I am not aware of any alternatives to this problem (comparing stability of differently sized networks quickly)
closes #45
I add a README and a master analysis script. There are also updates to some plots. The corresponding files in module 4 are removed and replaced in the new module 5