Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Confusion in discretization process #1

Closed
TokisakiKurumi2001 opened this issue Feb 3, 2024 · 1 comment
Closed

Confusion in discretization process #1

TokisakiKurumi2001 opened this issue Feb 3, 2024 · 1 comment

Comments

@TokisakiKurumi2001
Copy link

Hi there. This note is really helpful for me to understand about the Mamba. Thank you for such an amazing job you've done.

However, when looking at the notes, I stumble into a problem trying to the discrete formula.

From you notes, using the Euler Method, we will arrive the final formula as below.

$h_t = \mathbf{\bar{A}}h_{t-1} + \mathbf{\bar{B}}x_{t-1}$.

From that equation, I interpret that the new hidden state depends on the previous hidden state and the previous input x.

However, in the Mamba paper, I notice that the equation is slightly different.

$h_t = \mathbf{\bar{A}}h_{t-1} + \mathbf{\bar{B}}x_t$.

The term that changed here is $x_{t-1}$ become $x_t$.

Does it imply that using different algorithm (ZOH vs Euler) will result in different final equation? In addition, I found this Wikipedia link that states the equation different from the paper.

I'm a newbie here so if there is any mistake, please tell me know. Thank you.

@TokisakiKurumi2001
Copy link
Author

From this blog, it seems that under small $\Delta$, $x_t \approx x_{t+1}$. I think the authors make the substitution here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant