[ENH] Amazon chronos forecaster interface

[fkiraly]

More foundation models coming up.

As chronos is not on huggingface, it will not be covered by the generic huggingface wrapper.
Usage is fairly easy, so I would consider this a good first issue.

https://github.com/amazon-science/chronos-forecasting

This might be interesting design-wise.
The model comes with different sets of pretrained weights, I would make that choice an __init__ param, rather than a class constructor method, like in chronos proper - although I wonder whether there is a bottleneck here. Delalyed init (e.g., in fit) might be another choice.

[RigvedManoj]

@fkiraly I am new to open source, is this a "good first issue" i can start with?

[fkiraly]

yes, absolutely! I'd say it is medium level for a first issue.

Here's the relevant guide:
https://www.sktime.net/en/latest/developer_guide/add_estimators.html

[RigvedManoj]

For Chronos forecasting , I see that they have a list of arguments and keyword arguments. The Chronos utility takes these as optional variable arguments (*args, **kwargs) but I see that this is not allowed in sktime.

how should this be handled, should each argument be taken as an optional parameter and passed to Chronos?

[fkiraly]

I can see two options we commonly use:

• spell all the params out, if there are a fixed set, and pass them as explicit kwargs
• you can have dict (or list) valued arguments, which you then unwrapp with ** or *

[yashkabra19]

@fkiraly is someone working on this right now? I am looking at good first issues to solve and I think I could start here

[fkiraly]

@RigvedManoj might already be working on this, although I don't know the exact status.

If you're interested in foundation models, here's a collection of issues:
#6177
related to interfacing 3rd party, and sktime native design questions.

[yashkabra19]

ok thanks will check it out

[TripleShahar]

@fkiraly chronos is now on hugging face
https://huggingface.co/amazon/chronos-t5-tiny
@yashkabra19 / @RigvedManoj - what's the status?

[fkiraly]

hm, how does this relate to #5796 then?
The huggingface forecaster adapter

[RigvedManoj]

@fkiraly chronos is now on hugging face https://huggingface.co/amazon/chronos-t5-tiny @yashkabra19 / @RigvedManoj - what's the status?

I have created a draft pull request. I am able to run chronos via sktime. But the checkEstimator is not running.

[fkiraly]

@RigvedManoj, could you paste code here and a description of the error you get?

[RigvedManoj]

@RigvedManoj, could you paste code here and a description of the error you get?

from sktime.utils.estimator_checks import check_estimator
from sktime.forecasting.chronos import Chronos
check_estimator(Chronos)
Process finished with exit code -1073741819 (0xC0000005)

do you want me to paste the Chronos forecast estimator code as well?

[fkiraly]

do you want me to paste the Chronos forecast estimator code as well?

Not needed, I assume this happens if you run on your branch?

FYI, you can make code blocks by three backticks and then python, in markdown

[RigvedManoj]

do you want me to paste the Chronos forecast estimator code as well?

Not needed, I assume this happens if you run on your branch?

FYI, you can make code blocks by three backticks and then python, in markdown

Changed.

yes, I am getting this error when i run on my branch

[fkiraly]

FYI @benHeid, anything familiar from the hugging face work?

[benHeid]

hm, how does this relate to #5796 then? The huggingface forecaster adapter

Mhm I tried to load it via the forecaster adapter. But this fails since the entries in the config are named differently... The interface of the different configs is not really unified. Consequently, it is difficult to implement everything using only one adapter..

[benHeid]

FYI @benHeid, anything familiar from the hugging face work?

Nope, I haven't seen that error before. Which model are you loading? Might it be possible that you are not having enough memory for storing the model or you are not allowed to store the model at the specific location? https://answers.microsoft.com/en-us/windows/forum/all/application-error-0xc0000005/6224ae45-a251-4f21-b076-74524618d00a

[benHeid]

hm, how does this relate to #5796 then? The huggingface forecaster adapter

Mhm I tried to load it via the forecaster adapter. But this fails since the entries in the config are named differently... The interface of the different configs is not really unified. Consequently, it is difficult to implement everything using only one adapter..

However, it would be possible by adding several case distinctions. Anyway we should pay attention on how to reduce potential code duplication when adding LoRA methods to multiple of these forecasters.

[RigvedManoj]

FYI @benHeid, anything familiar from the hugging face work?

Nope, I haven't seen that error before. Which model are you loading? Might it be possible that you are not having enough memory for storing the model or you are not allowed to store the model at the specific location? https://answers.microsoft.com/en-us/windows/forum/all/application-error-0xc0000005/6224ae45-a251-4f21-b076-74524618d00a

def get_test_params(cls, parameter_set="default"):
        params = {"modelName": "amazon/chronos-t5-small"}
        return params

This is the parameter I am setting in get_test_params.
I am able to run this model separately when I am doing manual testing like below:

y = load_airline()
y_train, y_test = temporal_train_test_split(y)
fh = ForecastingHorizon(y_test.index, is_relative=False)
forecaster = Chronos("amazon/chronos-t5-small", kwargs_dict={"torch_dtype": torch.bfloat16})
forecaster.fit(y_train)
y_pred = forecaster.predict(fh)
print(mean_absolute_percentage_error(y_test, y_pred))

So I am not sure if it's a memory issue. It could be performance related though because Chronos takes a long time to run without gpu.

[fkiraly]

hm, why don't you open a PR, and we see what happens on the VM?

[RigvedManoj]

I have a draft PR, do i need to convert it into a PR for it to run on the VM?

[RigvedManoj]

If readthedocs build is succeeded, does it mean the check estimator has run successfully?

[fkiraly]

no, that's only the documentation. The other tests need to be started by a core developer. Let me start them.

[RigvedManoj]

no, that's only the documentation. The other tests need to be started by a core developer. Let me start them.

#6205

Of some 200 tests, around 35 have failed, I am not able to figure out what the failed tests are though?

[benHeid]

no, that's only the documentation. The other tests need to be started by a core developer. Let me start them.

#6205

Of some 200 tests, around 35 have failed, I am not able to figure out what the failed tests are though?

See comment in the PR

[pranavvp16]

I can would like to work on this

[fkiraly]

@pranavvp16, yes, you're tentatively assigned.

I think we need to check with @Z-Fran whether he is still working on this - are you, @Z-Fran?
(there is no message since 3 weeks on the PR, and you have not been in the planning session today, so my default assumption is "no")

[Z-Fran]

@pranavvp16, yes, you're tentatively assigned.

I think we need to check with @Z-Fran whether he is still working on this - are you, @Z-Fran? (there is no message since 3 weeks on the PR, and you have not been in the planning session today, so my default assumption is "no")

I'm working on this