Skip to content

Commit

Permalink
Merge c49bb12 into 423d664
Browse files Browse the repository at this point in the history
  • Loading branch information
@trax-robot
trax-robot committed Jul 22, 2020
2 parents 423d664 + c49bb12 commit f01ac88
Showing 1 changed file with 57 additions and 28 deletions.
85 changes: 57 additions & 28 deletions docs/source/notebooks/layers_intro.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@
"This notebook introduces the core concepts of the Trax library through a series of code samples and explanations. The topics covered in following sections are:\n",
"\n",
" 1. **Layers**: the basic building blocks and how to combine them into networks\n",
" 1. **Data Streams**: how individual layers manage inputs and outputs\n",
" 1. **Data Stack**: how the Trax runtime manages data streams for the layers\n",
" 1. **Defining New Layer Classes**: how to define and test your own layer classes\n",
" 1. **Inputs and Outputs**: how data streams flow through the layers\n",
" 1. **Defining New Layer Classes**\n",
" 1. **Testing and Debugging Layer Classes**\n",
"\n",
"\n"
]
Expand All @@ -26,7 +26,7 @@
"id": "BIl27504La0G"
},
"source": [
"## General Setup\n",
"### General Setup\n",
"Execute the following few cells (once) before running any of the code samples in this notebook."
]
},
Expand Down Expand Up @@ -138,7 +138,7 @@
"id": "-LQ89rFFsEdk"
},
"source": [
"# 1. Layers\n",
"## 1. Layers\n",
"\n",
"The Layer class represents Trax's basic building blocks:\n",
"```\n",
Expand Down Expand Up @@ -168,7 +168,7 @@
"id": "LyLVtdxorDPO"
},
"source": [
"## Layers compute functions."
"### Layers compute functions."
]
},
{
Expand Down Expand Up @@ -204,7 +204,7 @@
"id": "hCoapc5le8B7"
},
"source": [
"### Example 1. tl.Relu $[n_{in} = 1, n_{out} = 1]$"
"**Example 1.** tl.Relu $[n_{in} = 1, n_{out} = 1]$"
]
},
{
Expand Down Expand Up @@ -268,7 +268,7 @@
"id": "7sYxIT8crFVE"
},
"source": [
"### Example 2. tl.Concatenate $[n_{in} = 2, n_{out} = 1]$"
"**Example 2.** tl.Concatenate $[n_{in} = 2, n_{out} = 1]$"
]
},
{
Expand Down Expand Up @@ -338,7 +338,7 @@
"id": "z7N1qe91eYyM"
},
"source": [
"## Layers are configurable.\n",
"### Layers are configurable.\n",
"\n",
"Many layer types have creation-time parameters for flexibility. The \n",
"`Concatenate` layer type, for instance, has two optional parameters:\n",
Expand All @@ -357,7 +357,7 @@
"id": "l53Jw23pZ4s6"
},
"source": [
"### Example 3. tl.Concatenate(n_items=3, axis=0)"
"**Example 3.** tl.Concatenate(n_items=3, axis=0)"
]
},
{
Expand Down Expand Up @@ -434,7 +434,7 @@
"id": "1oZv3R8bRMvF"
},
"source": [
"## Layers are trainable.\n",
"### Layers are trainable.\n",
"\n",
"Many layer types include weights that affect the computation of outputs from\n",
"inputs, and they use back-progagated gradients to update those weights.\n",
Expand All @@ -451,7 +451,7 @@
"id": "3d64M7wLryji"
},
"source": [
"### Initialization\n",
"**Initialization**\n",
"\n",
"Trainable layers must be initialized before use. Trax can take care of this\n",
"as part of the overall training process. In other settings (e.g., in tests or\n",
Expand Down Expand Up @@ -516,7 +516,7 @@
"id": "yL8HAj6GEAp1"
},
"source": [
"### Example 4. tl.LayerNorm $[n_{in} = 1, n_{out} = 1]$"
"**Example 4.** tl.LayerNorm $[n_{in} = 1, n_{out} = 1]$"
]
},
{
Expand Down Expand Up @@ -583,7 +583,7 @@
"id": "d47gVdGV1vWw"
},
"source": [
"## Layers combine into layers.\n",
"### Layers combine into layers.\n",
"\n",
"The Trax library authors encourage users to build new layers as combinations of\n",
"existing layers. Hence, the library provides a small set of _combinator_\n",
Expand All @@ -602,7 +602,7 @@
"id": "vC1ymG2j0iyp"
},
"source": [
"### Combine with `Serial`\n",
"**Combine with `Serial`**\n",
"\n",
"The most common way to combine layers is with the `Serial` class:\n",
"```\n",
Expand Down Expand Up @@ -655,7 +655,7 @@
"id": "uPOnrDa9ViPi"
},
"source": [
"### Example 5. y = layer_norm(relu(x)) $[n_{in} = 1, n_{out} = 1]$"
"**Example 5.** y = layer_norm(relu(x)) $[n_{in} = 1, n_{out} = 1]$"
]
},
{
Expand Down Expand Up @@ -719,7 +719,7 @@
"source": [
"And we can inspect the block as a whole, as if it were just another layer:\n",
"\n",
"### Example 5'. Inspecting a Serial layer."
"**Example 5'.** Inspecting a `Serial` layer."
]
},
{
Expand Down Expand Up @@ -770,7 +770,8 @@
"id": "kJ8bpYZtE66x"
},
"source": [
"### Combine with `Branch`\n",
"**Combine with `Branch`**\n",
"\n",
"The `Branch` combinator arranges layers into parallel computational channels:\n",
"```\n",
"def Branch(*layers, name='Branch'):\n",
Expand Down Expand Up @@ -852,7 +853,7 @@
"id": "JGGnKjg4ESIg"
},
"source": [
"### Example 6. Branch"
"**Example 6.** `Branch`"
]
},
{
Expand Down Expand Up @@ -919,7 +920,7 @@
"id": "zr2ZZ1vO8T8V"
},
"source": [
"# 2. Data Streams\n",
"## 2. Inputs and Outputs\n",
"\n",
"The Trax runtime supports the concept of multiple data streams, which gives\n",
"individual layers flexibility to:\n",
Expand All @@ -939,7 +940,7 @@
"```\n",
"In other words, layer by layer:\n",
"\n",
" - `Branch(shortcut, layers)`: makes two copies of the single incoming data stream, passes one copy via the shortcut (typically a no-op), and processes the other copy via the given layers, applied in series. [$n_{in} = 1$, $n_{out} = 2$]\n",
" - `Branch(shortcut, layers)`: makes two copies of the single incoming data stream, passes one copy via the shortcut (typically a no-op), and processes the other copy via the given layers (applied in series). [$n_{in} = 1$, $n_{out} = 2$]\n",
" - `Add()`: combines the two streams back into one by adding two tensors elementwise. [$n_{in} = 2$, $n_{out} = 1$]"
]
},
Expand All @@ -950,7 +951,9 @@
"id": "QQVo6vhPgO9x"
},
"source": [
"# 3. Data Stack"
"### Data Stack\n",
"\n",
"*TBD*"
]
},
{
Expand All @@ -960,7 +963,7 @@
"id": "65ite-671cTT"
},
"source": [
"# 4. Defining New Layer Classes"
"## 3. Defining New Layer Classes"
]
},
{
Expand All @@ -970,7 +973,7 @@
"id": "hHSaD9H6hDTf"
},
"source": [
"## Simpler layers, with the `Fn` layer-creating function.\n",
"### With the `Fn` layer-creating function.\n",
"\n",
"Many layer types needed in deep learning compute pure functions from inputs to\n",
"outputs, using neither weights nor randomness. You can use Trax's `Fn` function\n",
Expand Down Expand Up @@ -1011,7 +1014,7 @@
"id": "TX30lGLXcjB1"
},
"source": [
"### Example 7. Use `Fn` to define a new layer type:"
"**Example 7.** Use `Fn` to define a new layer type:"
]
},
{
Expand Down Expand Up @@ -1091,7 +1094,7 @@
"id": "2lCjml7SCR-u"
},
"source": [
"### Example 8. `Fn` with multiple outputs:"
"**Example 8.** `Fn` with multiple outputs:"
]
},
{
Expand Down Expand Up @@ -1164,7 +1167,7 @@
"id": "GrXQUSbKDs41"
},
"source": [
"### Example 9. Use `Fn` to define a configurable layer:"
"**Example 9.** Use `Fn` to define a configurable layer:"
]
},
{
Expand Down Expand Up @@ -1267,7 +1270,33 @@
"id": "cqM6WJwNhoHI"
},
"source": [
"## Full subclass definitions, where necessary"
"### By defining a `Layer` subclass\n",
"\n",
"*TBD*"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "D77mYZZD41QO"
},
"source": [
"### By defining a `Combinator` subclass\n",
"\n",
"*TBD*"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "PgdQvZ5G6Aei"
},
"source": [
"## Testing and Debugging Layer Classes\n",
"\n",
"*TBD*"
]
}
],
Expand Down

0 comments on commit f01ac88

Please sign in to comment.