Skip to content

Commit

Permalink
update docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@vpj
vpj committed Nov 7, 2023
1 parent ba58ad9 commit f26c603
Show file tree
Hide file tree
Showing 4 changed files with 10 additions and 10 deletions.
2 changes: 1 addition & 1 deletion docs/diffusion/ddpm/unet.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1017,7 +1017,7 @@ <h2>U-Net</h2>
<div class='code'>
<div class="highlight"><pre><span class="lineno">313</span> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">image_channels</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">3</span><span class="p">,</span> <span class="n">n_channels</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">64</span><span class="p">,</span>
<span class="lineno">314</span> <span class="n">ch_mults</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">int</span><span class="p">,</span> <span class="o">...</span><span class="p">],</span> <span class="n">List</span><span class="p">[</span><span class="nb">int</span><span class="p">]]</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span>
<span class="lineno">315</span> <span class="n">is_attn</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">bool</span><span class="p">,</span> <span class="o">...</span><span class="p">],</span> <span class="n">List</span><span class="p">[</span><span class="nb">int</span><span class="p">]]</span> <span class="o">=</span> <span class="p">(</span><span class="kc">False</span><span class="p">,</span> <span class="kc">False</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="kc">True</span><span class="p">),</span>
<span class="lineno">315</span> <span class="n">is_attn</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">bool</span><span class="p">,</span> <span class="o">...</span><span class="p">],</span> <span class="n">List</span><span class="p">[</span><span class="nb">bool</span><span class="p">]]</span> <span class="o">=</span> <span class="p">(</span><span class="kc">False</span><span class="p">,</span> <span class="kc">False</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="kc">True</span><span class="p">),</span>
<span class="lineno">316</span> <span class="n">n_blocks</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">2</span><span class="p">):</span></pre></div>
</div>
</div>
Expand Down
2 changes: 1 addition & 1 deletion docs/normalization/group_norm/readme.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ <h1><a href="https://nn.labml.ai/normalization/group_norm/index.html">Group Norm
<p>This is a <a href="https://pytorch.org">PyTorch</a> implementation of the <a href="https://arxiv.org/abs/1803.08494">Group Normalization</a> paper.</p>
<p><a href="https://nn.labml.ai/normalization/batch_norm/index.html">Batch Normalization</a> works well for large enough batch sizes but not well for small batch sizes, because it normalizes over the batch. Training large models with large batch sizes is not possible due to the memory capacity of the devices.</p>
<p>This paper introduces Group Normalization, which normalizes a set of features together as a group. This is based on the observation that classical features such as <a href="https://en.wikipedia.org/wiki/Scale-invariant_feature_transform">SIFT</a> and <a href="https://en.wikipedia.org/wiki/Histogram_of_oriented_gradients">HOG</a> are group-wise features. The paper proposes dividing feature channels into groups and then separately normalizing all channels within each group.</p>
<p>Here&#x27;s a <a href="https://nn.labml.ai/normalization/group_norm/experiment.html">CIFAR 10 classification model</a> that uses instance normalization.</p>
<p>Here&#x27;s a <a href="https://nn.labml.ai/normalization/group_norm/experiment.html">CIFAR 10 classification model</a> that uses group normalization.</p>
<p><a href="https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/group_norm/experiment.ipynb"><img alt="Open In Colab" src="https://colab.research.google.com/assets/colab-badge.svg"></a> </p>

</div>
Expand Down
4 changes: 2 additions & 2 deletions docs/sitemap.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -645,7 +645,7 @@

<url>
<loc>https://nn.labml.ai/diffusion/ddpm/unet.html</loc>
<lastmod>2023-02-17T16:30:00+00:00</lastmod>
<lastmod>2023-11-06T16:30:00+00:00</lastmod>
<priority>1.00</priority>
</url>

Expand Down Expand Up @@ -1072,7 +1072,7 @@

<url>
<loc>https://nn.labml.ai/transformers/vit/index.html</loc>
<lastmod>2023-10-24T16:30:00+00:00</lastmod>
<lastmod>2023-11-06T16:30:00+00:00</lastmod>
<priority>1.00</priority>
</url>

Expand Down
12 changes: 6 additions & 6 deletions docs/transformers/vit/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -570,25 +570,25 @@ <h2>Vision Transformer</h2>
<div class='section-link'>
<a href='#section-37'>#</a>
</div>
<p>Add positional embeddings </p>
<p>Concatenate the <code class="highlight"><span></span><span class="p">[</span><span class="n">CLS</span><span class="p">]</span></code>
token embeddings before feeding the transformer </p>

</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">195</span> <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">pos_emb</span><span class="p">(</span><span class="n">x</span><span class="p">)</span></pre></div>
<div class="highlight"><pre><span class="lineno">195</span> <span class="n">cls_token_emb</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">cls_token_emb</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">x</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="lineno">196</span> <span class="n">x</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">cat</span><span class="p">([</span><span class="n">cls_token_emb</span><span class="p">,</span> <span class="n">x</span><span class="p">])</span></pre></div>
</div>
</div>
<div class='section' id='section-38'>
<div class='docs'>
<div class='section-link'>
<a href='#section-38'>#</a>
</div>
<p>Concatenate the <code class="highlight"><span></span><span class="p">[</span><span class="n">CLS</span><span class="p">]</span></code>
token embeddings before feeding the transformer </p>
<p>Add positional embeddings </p>

</div>
<div class='code'>
<div class="highlight"><pre><span class="lineno">197</span> <span class="n">cls_token_emb</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">cls_token_emb</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">x</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="lineno">198</span> <span class="n">x</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">cat</span><span class="p">([</span><span class="n">cls_token_emb</span><span class="p">,</span> <span class="n">x</span><span class="p">])</span></pre></div>
<div class="highlight"><pre><span class="lineno">198</span> <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">pos_emb</span><span class="p">(</span><span class="n">x</span><span class="p">)</span></pre></div>
</div>
</div>
<div class='section' id='section-39'>
Expand Down

0 comments on commit f26c603

Please sign in to comment.