Add TiRex-2 results#165
Conversation
| from tirex2 import ForecastModel, load_model | ||
|
|
||
|
|
||
| class TiRex2Model(fev.ForecastingModel): |
There was a problem hiding this comment.
@podestplatz Thanks for the contribution!
Unrelated to the PR: I noticed that the TiRex-2 technical report seems to use the arithmetic mean of MASE/SQL scores instead of the recommended aggregation with (normalization by snaive score + gmean), same as in GIFT-Eval. Is there a specific reason for this choice? MASE/SQL scores can have very different scales for some tasks, and arithmetic mean aggregation gives them a disproportionately high weight - that's why we recommend using gmean aggregation.
There was a problem hiding this comment.
For consistency reasons we went with
the raw values for the fev-bench results.
Thanks for pointing that out though, we'll address this in a future revision of the paper.
| @@ -0,0 +1,4 @@ | |||
| tirex-2 | |||
There was a problem hiding this comment.
Let's pin the package version here for reproducibility.
| @@ -0,0 +1,58 @@ | |||
| import contextlib | |||
There was a problem hiding this comment.
Can you please add a comment here or a README.md under models/tirex-2 describing which hardware was used to run the experiments? This would be helpful to put the runtime into context.
|
I addressed your comments, thanks for including TiRex-2 in the benchmark that fast! |
|
Thanks @podestplatz! |
2 participants
Hi,
we would like to submit TiRex-2 to fev-bench. TiRex-2 is a causal multivariate TSFM using xLSTM capable of leveraging future information, if available. As TiRex it is based on the recurrent xLSTM architecture and provides fast inference speeds.
models/tirex-2/model.pyandmodels/tirex-2/requirements.txtcontain the code and dependencies to reproduce theresults included in
benchmarks/fev_bench/results/tirex-2.csv.Please note that access to the checkpoints on Hugging Face is gated, meaning users must provide their email and username, and approval is granted automatically.
If you run into issues regarding compilation of the sLSTM kernel, please make sure that the system-level dependencies
are available in your evaluation environment.
Thanks!