How and Why is the industry quietly switching away from Transformers?

[ayeshaaa0134]

🏷️ Discussion Type

Question

💬 Feature/Topic Area

Models

Body

Everyone talks about how great Transformers are, but I'm noticing major labs and companies are actively moving to State Space Models, Mamba, and other architectures.

But I don't get it how were SSMs and Mamba actually made at first?

[Kristofferrrr]

SSMs are actually much older than Transformers. They come from control theory, where a system keeps a hidden "state" that is updated over time.

Mamba builds on modern SSMs and makes that state update depend on the input. This lets the model decide what information to keep or forget.

The main idea is:

Transformers: compare tokens with each other using attention.

Mamba/SSMs: continuously update a compact internal state.

This can make Mamba much more efficient for very long sequences, while still learning long-range dependencies.

[ayeshaaa0134]

hmmmm , but my concern is how do researchers even come up with something like this in the first place? how did someone come up with this to turn a continuous-time differential equation into a deep learning layer that replaces attention? and especially how do they even cmoe up with these equations and formulas in the first place , how do they even know that if this maths thing is causing problems we should repalce it with this maths how do they get that

[Kristofferrrr]

They usually don’t invent the math from nothing. They start with math that already exists and ask, “Can this help solve our problem?”

For SSMs, researchers took ideas from control theory, where equations describe how a system changes over time. Then they changed those equations so they could work with things like text.

The process is basically:

existing math → try it in a neural network → see what does not work well → change it → test again

Mamba was made in a similar way. Researchers saw that older SSMs were not good enough at deciding what information was important. So they changed the model so the input itself helps decide what to remember and what to forget.

So researchers usually do not know the perfect formula beforehand. They build on older ideas, experiment, measure the results, and improve the parts that cause problems.

[Knyazovich]

don’t invent the math from nothing