Skip to content

feat: add gift eval module#140

Merged
AzulGarza merged 10 commits intomainfrom
feat/gift-eval-utils
Jul 29, 2025
Merged

feat: add gift eval module#140
AzulGarza merged 10 commits intomainfrom
feat/gift-eval-utils

Conversation

@AzulGarza
Copy link
Member

this pr adds the gift_eval module. it allows to evaluate time copilot forecasters in a reproducible and easy way using the gift eval tools. it includes ci tests showing that the results obtained by the seasonal naive model are the exact ones as the provided by the mantainers in hugging face

@AzulGarza AzulGarza requested a review from Copilot July 26, 2025 20:47
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces a new gift_eval module to TimesCopilot that enables reproducible evaluation of time series forecasters using the GIFT-Eval benchmark framework. The module integrates with GluonTS to provide standardized evaluation metrics and comparison against established baselines like seasonal naive models.

  • Adds core evaluation infrastructure with GIFTEval class and GluonTSPredictor adapter
  • Implements dataset loading and preprocessing utilities for GIFT-Eval compatible datasets
  • Includes comprehensive CI tests that validate results match official seasonal naive benchmarks

Reviewed Changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
timecopilot/gift_eval/utils.py Defines dataset lists and configurations for short/medium/long term evaluations
timecopilot/gift_eval/gluonts_predictor.py Adapter class to wrap TimeCopilot forecasters as GluonTS predictors
timecopilot/gift_eval/eval.py Main evaluation class with predictor evaluation and CSV output functionality
timecopilot/gift_eval/data.py Dataset loading and preprocessing utilities adapted from GIFT-Eval
tests/gift_eval/test_evaluation.py Test suite validating evaluation results against official benchmarks
tests/gift_eval/conftest.py Test fixtures for dataset downloading and caching
pyproject.toml Adds gift_eval test marker configuration
mkdocs.yml Adds gift_eval module to documentation navigation
docs/api/gift_eval/gift_eval.md Documentation page for the gift_eval module
.github/workflows/ci.yaml Adds CI workflow for gift_eval tests

csv_file_path.parent.mkdir(parents=True, exist_ok=True)
results_df.to_csv(csv_file_path, index=False)

print(
Copy link

Copilot AI Jul 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] The print statement should use a logger instead of print() for better control over output in production environments.

Suggested change
print(
logging.info(

Copilot uses AI. Check for mistakes.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, fixed in AzulGarza/timecopilot@944fd8a.

"""
# fmt: on
res_dataset_properties = requests.get(
"https://raw.githubusercontent.com/SalesforceAIResearch/gift-eval/refs/heads/main/notebooks/dataset_properties.json"
Copy link

Copilot AI Jul 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The hardcoded URL should be extracted to a module-level constant or configuration variable for better maintainability.

Suggested change
"https://raw.githubusercontent.com/SalesforceAIResearch/gift-eval/refs/heads/main/notebooks/dataset_properties.json"
DATASET_PROPERTIES_URL

Copilot uses AI. Check for mistakes.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, fixed in AzulGarza/timecopilot@944fd8a.

@AzulGarza AzulGarza merged commit 1b068e4 into main Jul 29, 2025
8 checks passed
@AzulGarza AzulGarza deleted the feat/gift-eval-utils branch July 29, 2025 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants