Skip to content

Conversation

@noahho
Copy link
Collaborator

@noahho noahho commented Aug 13, 2025

Motivation and Context

This patch expands our test coverage to run against all available TabPFN models while keeping CI time reasonable.

  • Introduce ModelSource in tests to programmatically fetch model filenames.
  • Restructure parameter grids so we fully exercise one “primary” model (broad grid) and run a fast “smoke” sweep across the remaining models.
  • Reduce reliance on hardcoded "auto" model selection inside tests to make failures attributable to specific model artifacts.

This addresses gaps identified in “Run tests with all models” and aligns with feedback that initialization and interface tests should validate each shipped model.


Public API Changes

  • No Public API changes
    (Only test code is modified.)

How Has This Been Tested?

  • Confirmed the new grids keep test duration practical on CPU-only environments.

Checklist

  • The changes have been tested locally.
  • Documentation updated: N/A (tests only).
  • CHANGELOG.md entry: N/A (tests only).
  • Code follows project style (ruff format & ruff check).
  • Considered impact on public API: None (tests only).

Implementation Notes

  • Built:
    • _full_grid: exhaustive combos only for the first model path.
    • _smoke_grid: a single, fast combo for each remaining model path.
  • Combined as all_combinations = list(_full_grid) + list(_smoke_grid).

Prior Work / Acknowledgements

Big thanks to the work in PR #437 “Run tests with all models” (by @martino-vic) which motivated this direction and highlighted the need to validate every shipped model. That PR is good prior art but currently not mergeable (pending checks/CLA and minor integration issues). This patch folds in the workable pieces (programmatic model discovery + targeted grid split) and aligns them with the current test suite and style rules.

Copilot AI review requested due to automatic review settings August 13, 2025 11:30
gemini-code-assist[bot]

This comment was marked as outdated.

This comment was marked as outdated.

gemini-code-assist[bot]

This comment was marked as outdated.

@noahho noahho requested a review from Copilot August 13, 2025 11:38
gemini-code-assist[bot]

This comment was marked as outdated.

This comment was marked as outdated.

@noahho noahho mentioned this pull request Aug 13, 2025
7 tasks
noahho and others added 3 commits August 13, 2025 12:41
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@noahho
Copy link
Collaborator Author

noahho commented Aug 13, 2025

/gemini review

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request significantly improves test coverage by parameterizing tests to run against all available TabPFN models, which is a great enhancement. The approach of using a full grid for a primary model and a smoke test for others is clever and keeps CI times reasonable. I've identified a minor robustness issue where the tests could crash if no models are found. Adding a check to gracefully skip the tests in this scenario would make the test suite more resilient. Overall, this is a solid contribution to improving the project's test quality.

@noahho noahho requested a review from LeoGrin August 13, 2025 11:46
@LeoGrin
Copy link
Collaborator

LeoGrin commented Aug 13, 2025

bugbot run

Copy link
Collaborator

@LeoGrin LeoGrin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just two comments

@LeoGrin
Copy link
Collaborator

LeoGrin commented Aug 13, 2025

not sure about the failing test, probably just easier to go over the precision limit by random change if we test more models right?

@noahho noahho merged commit 86bad3f into main Aug 19, 2025
10 checks passed
oscarkey added a commit that referenced this pull request Nov 12, 2025
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------

Co-authored-by: noahho <Noah.homa@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants