support training on near-infinite sequence lengths?

[luke-mcdermott-mi]

infinite sequence length inference is currently supported by stepping through each token; however, I do not see any support for training on this. The parallel scan applied per batch does not have support to save the final hidden state for the following batch, so are there any plans to train mamba on context lengths > 4096 or whatever is the length cap from memory?

[JulienSiems]

There has been a recent paper on this: https://arxiv.org/pdf/2406.02080