Cleanup experiments new (#192)

* Setting up evaluation notebook for weight norm experiment * Put minimal model from experiments a level up * adding missing file * reorganising experiment folder and creation of a new anaylses folder
neuroinfo-os · Nov 28, 2018 · f5eb6bf · f5eb6bf
1 parent 7557330
commit f5eb6bf
Show file tree

Hide file tree

Showing 14 changed files with 882 additions and 755 deletions.
diff --git a/docs/analyses/1.minimal_model.ipynb b/docs/analyses/1.minimal_model.ipynb
diff --git a/...ents/6.weighted_binning_exploration.ipynb → ...yses/2.weighted_binning_exploration.ipynb b/...ents/6.weighted_binning_exploration.ipynb → ...yses/2.weighted_binning_exploration.ipynb
@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Standard vs. Weighted Binning"
+    "# Standard vs. Weighted Binning"
    ]
   },
   {
@@ -27,8 +27,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### In this notebook we try to show that the normal binning approach is not useful and try to derive a new one. \n",
+    "## 1. Hypothesis\n",
     "\n",
+    "In this notebook we try to show that the normal binning approach is not useful and try to derive a new one. \n",
+    "\n",
+    "## 2. Experiments\n",
     "As before we simplify the calculation of the mutual information between the input and a representation, by just calculating the entropy of the representation (as the representation is determined by the input).\n",
     "\n",
     "We use a very simplistic neural network model of 3 input, 3 hidden and 3 output neurons. The first weights matrix is an orthogonal matrix, such that the transposed matrix (after scaling) is the inverse matrix. We use linear activation function."
@@ -522,27 +525,6 @@
     "\n",
     "Next it would be interesting to find a way to implement this into the model and see what we find there."
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
@@ -561,7 +543,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.4"
+   "version": "3.6.5"
   }
  },
  "nbformat": 4,

diff --git a/docs/experiments/8.data_set.ipynb → docs/analyses/3.data_set.ipynb b/docs/experiments/8.data_set.ipynb → docs/analyses/3.data_set.ipynb
@@ -4,13 +4,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# The data set provided by Tishby"
+    "# Tishby's harmonics dataset"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "## Load Tishby's dataset\n",
     "First, we load the data set provided by Tishby."
    ]
   },
@@ -96,7 +97,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Our attempt to generate the data set above"
+    "## Our attempt to generate the data set above"
    ]
   },
   {

diff --git a/docs/experiments/9.analyze_entropy.ipynb → docs/analyses/4.analyze_entropy.ipynb b/docs/experiments/9.analyze_entropy.ipynb → docs/analyses/4.analyze_entropy.ipynb
diff --git a/docs/analyses/index.rst b/docs/analyses/index.rst
@@ -0,0 +1,22 @@
+
+
+Analyses
+========
+
+
+.. toctree::
+   :maxdepth: 2
+   :glob:
+
+   *
+
+
+
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
diff --git a/.../experiments/1.explanation_of_cohorts.rst → .../experiments/0.explanation_of_cohorts.rst b/.../experiments/1.explanation_of_cohorts.rst → .../experiments/0.explanation_of_cohorts.rst
@@ -4,7 +4,7 @@ Description of cohorts
 
 The experiments are structured in different cohort, containing one specific variation of parameters.
 To show the aim of the cohorts and to simplify the access of the saved artifacts using the artifact-viewer
-the following table offers a simple description for each cohort. 
+the following table offers a simple description for each cohort.
 
 
 +------------+-----------------------------------------------------------------------------------+
@@ -21,7 +21,7 @@ the following table offers a simple description for each cohort.
 |            | Additionally the hyperparameter of the estimators are varied.                     |
 |            | All experiments are done for relu and tanh using adam as optimizer.               |
 +------------+-----------------------------------------------------------------------------------+
-| cohort_4   | Comparison of training-, test- and full-dataset as base for the mi-computation.   |
+| cohort_4   | Comparison of training-, test- and full-dataset as base for the MI-computation.   |
 |            | All experiments are done for relu and tanh using Adam as optimizer.               |
 +------------+-----------------------------------------------------------------------------------+
 | cohort_5   | Comparison of different standard activation functions.                            |
@@ -36,3 +36,21 @@ the following table offers a simple description for each cohort.
 | cohort_8   | Comparison of architecture with batchnorm and without batchnorm.                  |
 |            | All experiments are done for relu and tanh using adam as optimizer.               |
 +------------+-----------------------------------------------------------------------------------+
+| cohort_9   | Comparison of architecture with batchnorm and without batchnorm.                  |
+|            | All experiments are done for relu and tanh using adam as optimizer.               |
++------------+-----------------------------------------------------------------------------------+
+| cohort_10  | Comparing weight norm for max_norm_weights = 0.9 and max_norm_weights = 0.6.      |
+|            |                                                                                   |
++------------+-----------------------------------------------------------------------------------+
+| cohort_11  |                                                                                   |
+|            |                                                                                   |
++------------+-----------------------------------------------------------------------------------+
+| cohort_12  |                                                                                   |
+|            |                                                                                   |
++------------+-----------------------------------------------------------------------------------+
+| cohort_13  | Effect of weight renormalization on activity patterns.                            |
+|            | Experiments for relu and tanh using adam as optimizer.                            |
++------------+-----------------------------------------------------------------------------------+
+| cohort_14  |                                                                                   |
+|            |                                                                                   |
++------------+-----------------------------------------------------------------------------------+
diff --git a/docs/experiments/10.cohort10_weight_norm.ipynb b/docs/experiments/10.cohort10_weight_norm.ipynb
diff --git a/...xperiments/7.weight_renormalization.ipynb → ...13.cohort_13_weight_renormalization.ipynb b/...xperiments/7.weight_renormalization.ipynb → ...13.cohort_13_weight_renormalization.ipynb
@@ -4,7 +4,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Effect of weight renormalization on activity patterns\n",
+    "# Cohort 13"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Effect of weight renormalization on activity patterns\n",
     "\n",
     "In this experiment we show the influence of weight renormalization on the structure of activations in different layers. \n",
     "\n",
@@ -17,7 +24,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Experiments with max_weight_norm=0.8"
+    "### Experiments with max_weight_norm=0.8"
    ]
   },
   {
@@ -200,7 +207,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Experiment with max_weight_norm=0.4\n",
+    "### Experiment with max_weight_norm=0.4\n",
     "\n",
     "In the following we present an example with `relu` and the norm of the weight vector for each layer restricted to 0.4\n",
     "This is a significantly stronger regularization which this time will also have an effect on the performance of the network. "
@@ -342,7 +349,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Supplementary material\n",
+    "### Supplementary material\n",
     "\n",
     "Below we find plots indicating the development of means and standard deviation of the gradient, its signal to noise ratio as well as the norm of the weight vector for all layers over the course of training. Comparing plots for unconstrained vs. constrained weight vector, we can reassure ourselves that rescaling the weights worked as we expected."
    ]
@@ -444,7 +451,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.3"
+   "version": "3.6.5"
   }
  },
  "nbformat": 4,