address more feedback

kaizhangNV · kaizhangNV · commit 5cec1eab3906 · 2025-08-07T13:11:20.000-07:00
diff --git a/docs/auto-diff-tutorial-2.md b/docs/auto-diff-tutorial-2.md
@@ -9,9 +9,9 @@ intro_image_hide_on_mobile: false
 
 ## What Will We Learn
 
-In the [last tutorial](auto-diff-tutorial-1.md), we explain basic knowledge about Slang's autodiff feature and introduce some advanced techniques about custom derivative implementation and custom differential type.
+In the [last tutorial](auto-diff-tutorial-1.md), we explained basic knowledge about Slang's autodiff feature and introduced some advanced techniques about custom derivative implementation and custom differential type.
 
-In this tutorial, we will walk through a real-world example using what we've learnt from the last tutorial.
+In this tutorial, we will walk through a real-world example using what we've learned from the last tutorial.
 
 We will learn how to implement a tiny Multi-Layer Perceptron (MLP) to approximate a set of polynomial functions using Slang's automatic differentiation capabilities.
 
@@ -29,8 +29,9 @@ f_{4}(x,\, y) =&x + 0.5y^{2}
 \end{bmatrix}$$
 
 - **Network Architecture**: 4 inputs → 16 hidden units → 4 outputs
+  - We will encode the 2 inputs $x$ and $y$ into $x$, $y$, $x^2$ and $y^2$.
 
-- **Learning Method**: Adam Optimizer
+- **Learning Method**: Simple gradient descent
 
 ## 2. Mathematical Formulation
 
@@ -42,11 +43,11 @@ $$L(\theta) = \left| \left| MLP(x,y;\theta) - f(x,y) \right| \right|^{2}$$
 
 Where:
 
-- $MLP(x,y;\theta)$ is our neural network\'s output
+- $MLP(x,y;\theta)$ is our neural network's output
 - $f(x,y)$ is the ground truth polynomial set
 - $L(\theta)$ is the [mean squared error](https://en.wikipedia.org/wiki/Mean_squared_error)
 
-### The Workflow will be
+### The workflow will be
 
 - **Forward Pass**: Compute $MLP(x,y;\theta)$ and $L(\theta)$
 - **Backward Pass**: Compute $\frac{\partial L}{\partial\theta}$ (gradients w.r.t. parameters)
@@ -71,19 +72,19 @@ public struct MyNetwork
 
     public half4 eval(no_diff half x, no_diff half y)
     {
-        // construct MLVec<4> from x, y
+        // construct MLVec<4> from x, y, x^2, y^2
         // call internal _eval
         // convert MLVec<4> to half4
     }
 }
 ```
 
-In our interface methods design, we introduce an internal representation of vector type `MLVec<4>` in the network evaluation. And this design choice will help us hide the details about how we are going to perform the arithmetic/logic operations on the vector type, as they are irrelevant to how the network should be designed. For now, you can treat this type as an opaque type, and we will talk about the detail in later section. Therefore, in the public method, we will just convert the input to `MLVec`, call the internal `_eval` method, and convert `MLVec` back to a normal vector. And in the internal `_eval` method, we will just chain the evaluations of each layer together.
-> Note that we are using `half` type, which is [16-bit floating-point type](external/slang/docs/user-guide/02-conventional-features.md#scalar-types) for better thoughput and reducing memory usage.
+In our interface method design, we introduce an internal representation of vector type `MLVec<4>` in the network evaluation. And this design choice will help us hide the details about how we are going to perform the arithmetic/logic operations on the vector type, as they are irrelevant to how the network should be designed. For now, you can treat this type as an opaque type, and we will talk about the details in later section. Therefore, in the public method, we will just convert the input to `MLVec`, call the internal `_eval` method, and convert `MLVec` back to a normal vector. And in the internal `_eval` method, we will just chain the evaluations of each layer together.
+> Note that we are using `half` type, which is [16-bit floating-point type](external/slang/docs/user-guide/02-conventional-features.md#scalar-types) for better throughput and to reduce memory usage.
 
 ### 3.2 Feed-Forward Layer Definition
 
-Each layer performs: $LeakyRelu(Wx+x, b)$, so we can create a struct to abstract this as follow:
+Each layer performs: $LeakyRelu(Wx+b)$, so we can create a struct to abstract this as follows:
 
 ```hlsl
 public struct FeedForwardLayer<int InputSize, int OutputSize>
@@ -101,8 +102,7 @@ public struct FeedForwardLayer<int InputSize, int OutputSize>
 }
 ```
 
-The evaluation is very simple, just doing a matrix vector multiplication plus a bias vector, and then performing a LeakyRelu function on it. But note that we use pointers in this struct to reference the parameters (e.g. weight and bias) instead of declaring arrays or even global storage buffers. Because our MLP will be run on the GPU and each evaluation function will be executed per-thread, if we used arrays for each layer, an array would be allocated for each thread and that would explode the GPU memory. For the same reason, we cannot declare a global storage buffer  because each thread will hold a storage buffer that stores the exact same data.
-Therefore, the most efficient approach is to use pointers or references to the global storage buffer, so every thread which executes the layer's method can access it.
+The evaluation is very simple, just doing a matrix vector multiplication plus a bias vector, and then performing a LeakyRelu function on it. But note that we use pointers in this struct to reference the parameters (e.g. weight and bias) instead of declaring arrays or even global storage buffers. Because our MLP will be run on the GPU and each evaluation function will be executed per-thread, if we used arrays for each layer, an array would be allocated for each thread and that would explode the GPU memory. For the same reason, we cannot declare a storage buffer directly within the layer struct because each thread would create its own copy of the storage buffer containing identical data, leading to massive memory waste. Instead, we use pointers to reference a single shared global storage buffer.
 
 ### 3.3 Vector Type Definition
 
@@ -136,14 +136,14 @@ MLVec<OutputSize> matMulTransposed<int OutputSize, int InputSize>(MLVec<InputSiz
 ```
 - Outer product of two vectors
 ```hlsl
-void outerProductAccumulate<int M, int N>(MLVec<M> v0, MLVec<N>v1, NFloat* matrix);
+void outerProductAccumulate<int M, int N>(MLVec<M> v0, MLVec<N> v1, NFloat* matrix);
 ```
 
-The First two operations are straightforwad, which is just matrix vector multiplication and its transpose version. The last operation defines an outer product of two vectors, the result will be a matrix, such that $\text{x} \otimes \text{y} = \text{x} \cdot \text{y}^{T}$, where $\text{x}$ and $\text{y}$ are column vectors with same length.
+The first two operations are straightforward, which is just matrix vector multiplication and its transpose version. The last operation defines an outer product of two vectors, the result will be a matrix, such that $\text{x} \otimes \text{y} = \text{x} \cdot \text{y}^{T}$, where $\text{x}$ and $\text{y}$ are column vectors with the same length.
 
 ### 3.4 Loss Function Definition
 
-The loss function measures how far our network output is from the target, given that we have already defined the interfaces for the MLP network, we can simply implement the loss function:
+The loss function measures how far our network output is from the target, since we have already defined the interfaces for the MLP network, we can simply implement the loss function:
 
 ```hlsl
 public half loss(MyNetwork* network, half x, half y)
@@ -172,15 +172,15 @@ public half4 groundtruth(half x, half y)
 
 ## 4. Backward Pass Design
 
-After implementing the forward pass of the network evaluation, we then need to implement the backward pass. You will see how effortless it is to implement the backward pass with Slang's autodiff. We're going to start the implementation from the end of the workflow to the beginning, because that's how the direction of the gradients flow.
+After implementing the forward pass of the network evaluation, we then need to implement the backward pass. You will see how effortless it is to implement the backward pass with Slang's autodiff. We will start the implementation from the end of the workflow to the beginning, because that's how the direction in which the gradients flow.
 
 ### 4.1 Backward Pass of Loss
 
-To implement the backward derivative of loss function, we only need to mark the function is `[Differentiable]` as we learnt from [last tutorial](auto-diff-tutorial-1.md#forward-mode-differentiation)
+To implement the backward derivative of loss function, we only need to mark the function as `[Differentiable]` as we learned from [last tutorial](auto-diff-tutorial-1.md#forward-mode-differentiation)
 
 ```hlsl
 [Differentiable]
-public half loss(MyNetwork* network, no_diff half x, no_diff halfy)
+public half loss(MyNetwork* network, no_diff half x, no_diff half y)
 {
     let networkResult = network->eval(x, y);    // MLP(x,y; θ)
     let gt = no_diff groundtruth(x, y);         // target(x,y)
@@ -194,13 +194,13 @@ And from the Slang kernel function, we just need to call the backward mode of th
 bwd_diff(loss)(network, input.x, input.y, 1.0h);
 ```
 
-One important thing to notice is that we are using the [`no_diff` attribute](external/slang/docs/user-guide/07-autodiff.html#excluding-parameters-from-differentiation) to decorate the inputs `x` and `y`, as well as `groudtruth` calculation, because in the backward pass, we only care about the result of $\frac{\partial loss}{\partial\theta}$. The `no_diff` attribute just tells Slang to treat the variables or instructions as non-differentiable, so there will be no backward mode instructions generated for those variables or instructions. In this case, since we don't care about the derivative of loss function with respective of input, therefore we can safely mark them as non-differentiable.
+One important thing to notice is that we are using the [`no_diff` attribute](external/slang/docs/user-guide/07-autodiff.html#excluding-parameters-from-differentiation) to decorate the inputs `x` and `y`, as well as `groundtruth` calculation, because in the backward pass, we only care about the result of $\frac{\partial loss}{\partial\theta}$. The `no_diff` attribute just tells Slang to treat the variables or instructions as non-differentiable, so there will be no backward mode instructions generated for those variables or instructions. In this case, since we don't care about the derivative of loss function with respect to the input, therefore we can safely mark them as non-differentiable.
 
 Since `loss` function is differentiable now, every instruction inside this function has to be differentiable except those marked as `no_diff`. Therefore, `network->eval(x, y)` must be differentiable, so next we are going to implement the backward pass for this method.
 
 ### 4.2 Automatic Propagation to MLP
 
-Just like the `loss` function, the only thing we need to do for `MyNetwork::eval` in order to use them with autodiff is to mark them as differentiable:
+Just like the `loss` function, the only thing we need to do for `MyNetwork::eval` in order to use it with autodiff is to mark it as differentiable:
 ```hlsl
 public struct MyNetwork
 {
@@ -225,19 +225,21 @@ public struct MyNetwork
 
 ## 4.3 Custom Backward Pass for Layers
 
-Following the propagation direction of the gradients, we will next implement the backward derivative of FeedForwardLayer. But here we're going to do something different. Instead of asking Slang to automatically synthesize the backward autodiff for us, we will provide a custom derivative implementation. Because the network parameters and gradients are a buffer storage declared in the layer, we will have to provide a custom derivative to write the gradient back to the global buffer storage. You can reference [progagate derivative to storage buffer](auto-diff-tutorial-1.md#how-to-propagate-derivatives-to-global-buffer-storage) in last tutorial to refresh your memory. Another benefit of providing a custom derivative here is that our layer is just matrix multiplication with bias, and its derivative is quite simple, so there are lots of options to accelerate it with specific hardware (e.g. Nvidia tensor cores). Therefore, it's good practice to implement the custom derivative.
+Following the propagation direction of the gradients, we will next implement the backward derivative of FeedForwardLayer. But here we're going to do something different. Instead of asking Slang to automatically synthesize the backward autodiff for us, we will provide a custom derivative implementation. Because the network parameters and gradients are a buffer storage declared in the layer, we will have to provide a custom derivative to write the gradient back to the global buffer storage. You can reference [propagate derivative to storage buffer](auto-diff-tutorial-1.md#how-to-propagate-derivatives-to-global-buffer-storage) in last tutorial to refresh your memory. Another benefit of providing a custom derivative here is that our layer is just matrix multiplication with bias, and its derivative is quite simple, so there are lots of options to accelerate it with specific hardware (e.g. Nvidia tensor cores). Therefore, it's good practice to implement the custom derivative.
 
 First, let's revisit the mathematical formula, given:
+
 $$Z=W\cdot x+b$$
 $$Y=LeakyRelu(Z)$$
 
 Where $W \in R^{m \times n}$, $x \in R^{n}$ and $b \in R^{m}$, the
 gradient of $W$, $x$ and $b$ will be:
-$$Z = W\cdot x + b$$
-$$dZ=dY\cdot(z > 0?1:\alpha)$$
+
+$$Z=W\cdot x+b$$
+$$dZ=dY\cdot(z > 0 ? 1:\alpha)$$
 $$dW=dZ\cdot x^{T}$$
-$$\text{d}b= \text{d}Z$$
-$$dx = W^{T} \cdot dZ$$
+$$\text{d}b=\text{d}Z$$
+$$dx=W^{T}\cdot dZ$$
 
 Therefore, the implementation should be:
 ```hlsl
@@ -267,8 +269,8 @@ public void evalBwd(inout DifferentialPair<MLVec<InputSize>> x, MLVec<OutputSize
 
     // Step 4: Compute input gradients (for chaining to previous layer)
     // dx = W^T * dZ
-    let dx= **matMulTransposed<InputSize>(dResult, weights);
-    x= {x.p, dx}; // Update differential pair
+    let dx = matMulTransposed<InputSize>(dResult, weights);
+    x = {x.p, dx}; // Update differential pair
 }
 ```
 
@@ -277,13 +279,13 @@ The key point in this implementation is that we use atomic add when writing the
 InterlockedAddF16Emulated(biasesGrad + i, dResult.data[i], originalValue);
 ```
 
-In the forward pass we already know that the parameters stored in a global storage buffer is shared by all threads, and so are gradients. Therefore, during backward pass, each thread will accumulate its gradient data to the shared storage buffer, we must use atomic add to accumulate all the gradients without race conditions.
+In the forward pass we already know that the parameters stored in a global storage buffer are shared by all threads, and so are the gradients. Therefore, during the backward pass, each thread will accumulate its gradient data to the shared storage buffer, so we must use atomic add operation to accumulate all the gradients without race conditions.
 
 The implementation of `outerProductAccumulate` and `matMulTransposed` are trivial for-loop multiplication, so we will not show the details in this tutorial. The complete code can be found at [here](https://github.com/shader-slang/neural-shading-s25/tree/main/hardware-acceleration/mlp-training).
 
-### 4.3 Make the vector differentiable
+### 4.4 Make the vector differentiable
 
-If we just compile what we have right now, we would generate a compile error because `MLVec` is not a differentiable type, so Slang doesn't expect the signature of backward of layer's eval method to be
+If we just compiled what we have right now, we would get a compile error because `MLVec` is not a differentiable type, so Slang doesn't expect the signature of backward of layer's eval method to be:
 ```hlsl
 public void evalBwd(inout DifferentialPair<MLVec<InputSize>> x, MLVec<OutputSize> dResult)
 ```
@@ -302,11 +304,11 @@ public struct MLVec<int N> : IDifferentiable
 }
 ```
 
-### 4.4 Parameter Update
+### 4.5 Parameter Update
 
-After back propagation, the last step is to update the parameters by the gradients we just computed
+After backpropagation, the last step is to update the parameters by the gradients we just computed:
 ```hlsl
-public struct Optimzer
+public struct Optimizer
 {
     public static const NFloat learningRate = 0.01h; // Step size
     public static void step(inout NFloat param, inout NFloat grad)
@@ -319,8 +321,7 @@ public struct Optimzer
 
 ## 5. Putting It All Together
 
-We will create two kernel functions, one for the training, and another
-one for parameter updating.
+We will create two kernel functions: one for training and another for parameter updating.
 
 The training kernel will be:
 ```hlsl
@@ -368,6 +369,6 @@ void adjustParameters(
 }
 ```
 
-The training process will be a loop that alternatively invokes the slang compute kernel `learnGradient` and `adjustParameters`, until the loss converges to an acceptable threshold value.
+The training process will be a loop that alternately invokes the slang compute kernel `learnGradient` and `adjustParameters`, until the loss converges to an acceptable threshold value.
 
-The host side implementation for this example is not shown, it will be boilerplate code to setup the graphics pipeline and allocate buffers for parameters. You can access the [github repo](https://github.com/shader-slang/neural-shading-s25/tree/main/hardware-acceleration/mlp-training) for complete code of this example, which includes the host side implementation. Alternatively, you can use the more powerful tool [SlangPy](https://github.com/shader-slang/slangpy) to run this MLP example without writing any graphics boilerplate code.
+The host side implementation for this example is not shown, as it is simply boilerplate code to setup the graphics pipeline and allocate buffers for parameters. You can access the [github repo](https://github.com/shader-slang/neural-shading-s25/tree/main/hardware-acceleration/mlp-training) for this example's complete code, which includes the host side implementation. Alternatively, you can use the more powerful tool [SlangPy](https://github.com/shader-slang/slangpy) to run this MLP example without writing any graphics boilerplate code.
