Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a new gift_eval module to TimesCopilot that enables reproducible evaluation of time series forecasters using the GIFT-Eval benchmark framework. The module integrates with GluonTS to provide standardized evaluation metrics and comparison against established baselines like seasonal naive models.
- Adds core evaluation infrastructure with
GIFTEvalclass andGluonTSPredictoradapter - Implements dataset loading and preprocessing utilities for GIFT-Eval compatible datasets
- Includes comprehensive CI tests that validate results match official seasonal naive benchmarks
Reviewed Changes
Copilot reviewed 10 out of 11 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
timecopilot/gift_eval/utils.py |
Defines dataset lists and configurations for short/medium/long term evaluations |
timecopilot/gift_eval/gluonts_predictor.py |
Adapter class to wrap TimeCopilot forecasters as GluonTS predictors |
timecopilot/gift_eval/eval.py |
Main evaluation class with predictor evaluation and CSV output functionality |
timecopilot/gift_eval/data.py |
Dataset loading and preprocessing utilities adapted from GIFT-Eval |
tests/gift_eval/test_evaluation.py |
Test suite validating evaluation results against official benchmarks |
tests/gift_eval/conftest.py |
Test fixtures for dataset downloading and caching |
pyproject.toml |
Adds gift_eval test marker configuration |
mkdocs.yml |
Adds gift_eval module to documentation navigation |
docs/api/gift_eval/gift_eval.md |
Documentation page for the gift_eval module |
.github/workflows/ci.yaml |
Adds CI workflow for gift_eval tests |
timecopilot/gift_eval/eval.py
Outdated
| csv_file_path.parent.mkdir(parents=True, exist_ok=True) | ||
| results_df.to_csv(csv_file_path, index=False) | ||
|
|
||
| print( |
There was a problem hiding this comment.
[nitpick] The print statement should use a logger instead of print() for better control over output in production environments.
| print( | |
| logging.info( |
There was a problem hiding this comment.
thanks, fixed in AzulGarza/timecopilot@944fd8a.
timecopilot/gift_eval/eval.py
Outdated
| """ | ||
| # fmt: on | ||
| res_dataset_properties = requests.get( | ||
| "https://raw.githubusercontent.com/SalesforceAIResearch/gift-eval/refs/heads/main/notebooks/dataset_properties.json" |
There was a problem hiding this comment.
The hardcoded URL should be extracted to a module-level constant or configuration variable for better maintainability.
| "https://raw.githubusercontent.com/SalesforceAIResearch/gift-eval/refs/heads/main/notebooks/dataset_properties.json" | |
| DATASET_PROPERTIES_URL |
There was a problem hiding this comment.
thanks, fixed in AzulGarza/timecopilot@944fd8a.
this pr adds the
gift_evalmodule. it allows to evaluate time copilot forecasters in a reproducible and easy way using the gift eval tools. it includes ci tests showing that the results obtained by the seasonal naive model are the exact ones as the provided by the mantainers in hugging face