feat: improved benchmarks and do-no-harm feature selection

[thinkall]

Summary

Improves FeatCopilot benchmarks (rigorous statistical methodology, broader dataset coverage, source-tracked reporting) and the feature engineering pipeline so it no longer regresses on real-world datasets.

Benchmark Infrastructure

• 5-fold stratified cross-validation with mean ± std reporting (was single 80/20 split)
• Wilcoxon signed-rank test for statistical significance (p < 0.05)
• Dataset source tracking: 31 real-world (30 INRIA + 1 HuggingFace) vs 32 synthetic, reported separately
• Win/Tie/Loss counts with significance markers
• Source provenance: loader-level synthetic-fallback detection, so a Kaggle/HF loader that falls back to a synthesized dataset is correctly bucketed as synthetic
• New CLI options: --real-world, --n-folds, --n-seeds, --fast

FeatCopilot Feature Selection Improvements

• Fixed redundancy eliminator: never removes original features (originals are categorically protected; was dropping them, e.g., diamonds 6→4 features)
• Stricter L1 refinement: uses mean importance threshold (was 0.5× mean — too lenient)
• Do-no-harm gate: held-out validation (5-split shuffle) to verify derived features help before keeping them. Falls back to original features automatically when features don't add value, with a RuntimeWarning so the fallback is never silent.
• Repaired Copilot LLM SDK integration: the --with-llm benchmark path was previously falling back to mock LLM responses on github-copilot-sdk >= 0.1.32; updated featcopilot/llm/copilot_client.py to the new SDK API (mandatory on_permission_request, send_and_wait, disconnect) so LLM-generated features are actually evaluated.

Results (5-fold CV, FeatCopilot tabular + LLM engines, 63 datasets)

Dataset	Before	After
diabetes	-4.40%	+0.00% ✅
covertype	-3.01%	+0.00% ✅
electricity	-2.73%	+1.42% ✅
higgs	-1.77%	+0.73% ✅
bank_marketing	-2.12%	+0.00% ✅
diamonds	-0.56%	+0.00% ✅
eye_movements	+0.31%	+2.16% ✅
fake_news (HF text)	n/a	+76.31% ✅
nyc_taxi	n/a	+8.62% ✅

Real-world classification (15 INRIA + 1 HuggingFace text, 5-fold CV):

• Win: 4 (25%), Tie: 11 (69%), Loss: 1 (6%)
• Max regression: -1.17% (covertype_cat; not statistically significant at p<0.05)
• No statistically significant regression on any real-world classification dataset

Real-world regression (15 INRIA, 5-fold CV):

• Win: 1 (7%, nyc_taxi +8.62%), Tie: 13 (87%), Loss: 1 (7%, delays_zurich)
• Note: delays_zurich shows -129.52% only because its baseline R² is ≈0.009 (essentially noise prediction); absolute Δ in R² is small and the result is not statistically significant (p=1.0)

All real-world datasets combined (31, 5-fold CV):

• Win: 5 (16%), Tie: 24 (77%), Loss: 2 (6%)
• Mean improvement: -1.29% (skewed entirely by the delays_zurich near-zero-baseline edge case; median improvement is +0.00%)

Synthetic datasets (32, supplementary, 5-fold CV):

• Win: 20 (63%), Tie: 11 (34%), Loss: 1 (3%)
• Mean improvement: +13.70%; Best: +104.34% (triple_interaction_regression), +103.41% (xor_regression)

Testing

• All 768 tests pass (was 625 at PR open; +143 regression tests added across review iterations)
• Pre-commit (black, ruff) passes
• Python 3.10 / 3.11 / 3.12 / 3.13 all green in CI

[codecov-commenter]

Codecov Report

❌ Patch coverage is 96.24060% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.90%. Comparing base (fd30793) to head (1e82335).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
featcopilot/transformers/sklearn_compat.py	96.06%	5 Missing ⚠️
featcopilot/engines/text.py	60.00%	2 Missing ⚠️
featcopilot/llm/copilot_client.py	88.88%	2 Missing ⚠️
featcopilot/selection/unified.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main       #3      +/-   ##
==========================================
+ Coverage   88.18%   88.90%   +0.71%     
==========================================
  Files          36       36              
  Lines        3808     3937     +129     
==========================================
+ Hits         3358     3500     +142     
+ Misses        450      437      -13     

Flag	Coverage Δ
unittests	88.90% <96.24%> (+0.71%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4d9d5df5fe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

This PR upgrades the benchmarking suite to support more rigorous, publication-style evaluation (CV, significance testing, dataset source stratification) and hardens feature selection to prevent regressions by preserving original features and adding a “do-no-harm” validation gate.

Changes:

• Add a do-no-harm gate to AutoFeatureEngineer.fit_transform() that can drop derived features when held-out validation doesn’t show benefit.
• Fix feature selection behavior to preserve original features during redundancy elimination and tighten L1 refinement thresholds.
• Rework the simple-model benchmark runner/reporting to use k-fold CV with Wilcoxon signed-rank tests and to separate real-world vs synthetic datasets.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
featcopilot/transformers/sklearn_compat.py	Adds do-no-harm validation gate after selection.
featcopilot/selection/unified.py	Preserves original features in redundancy step; tightens L1 refinement thresholds/fallback.
featcopilot/selection/redundancy.py	Ensures redundancy eliminator never removes original features when paired with derived ones.
benchmarks/simple_models/run_simple_models_benchmark.py	Switches to CV + Wilcoxon; adds real-world filtering and new CLI args; updates report format.
benchmarks/simple_models/SIMPLE_MODELS_BENCHMARK.md	Updates published benchmark report snapshot to new format.
benchmarks/datasets.py	Introduces dataset source tagging + helpers (is_real_world, list_real_world_datasets, etc.).
benchmarks/init.py	Re-exports new dataset source helpers/constants.
benchmarks/README.md	Updates benchmark documentation/results summary to reflect new methodology and source split.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Updates FeatCopilot’s benchmarking and feature-selection pipeline to (a) report more statistically rigorous benchmark results and (b) prevent engineered features from causing regressions by adding a “do-no-harm” validation gate.

Changes:

• Added a held-out validation “do-no-harm” gate to AutoFeatureEngineer.fit_transform() plus new unit tests covering keep/fallback behavior and task-type inference.
• Adjusted feature selection to preserve original features through redundancy elimination and tightened derived-feature refinement thresholds.
• Refactored the simple-models benchmark to use k-fold CV with Wilcoxon signed-rank significance testing and to report real-world vs synthetic datasets separately; updated benchmark docs accordingly.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/test_sklearn_compat.py	Adds tests for the new do-no-harm gate and its integration points.
featcopilot/transformers/sklearn_compat.py	Introduces _do_no_harm_gate() and invokes it after selection in fit_transform().
featcopilot/selection/unified.py	Ensures original features aren’t dropped when filtering by non-redundancy; tightens L1 refine.
featcopilot/selection/redundancy.py	Updates redundancy removal rules to avoid dropping original features when correlated.
benchmarks/simple_models/run_simple_models_benchmark.py	Moves to CV + Wilcoxon reporting and adds --real-world/--n-folds/--n-seeds/--fast options.
benchmarks/simple_models/SIMPLE_MODELS_BENCHMARK.md	Updates the published report snapshot to match the new methodology and source split.
benchmarks/datasets.py	Adds dataset source tracking + helper APIs for listing/flagging real-world vs synthetic datasets.
benchmarks/init.py	Re-exports new dataset source helpers/constants.
benchmarks/README.md	Updates benchmark methodology and summarizes results split by source.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[thinkall]

@copilot review

[Copilot]

@copilot review

All 6 round-1 and 3 round-2 review threads have been addressed in commits c1c2b35 and 813a7d4. Here's a summary:

sklearn_compat.py (do-no-harm gate):

• Task-type inference now uses sklearn.utils.multiclass.type_of_target() instead of the integer-dtype heuristic — correctly handles bool, nullable Int64, and {0.0, 1.0} float-encoded binary targets.
• Gate only runs when self._selector is not None (i.e., selection actually ran), so apply_selection=True with max_features=None remains a no-op.
• X_original is now used to build the baseline feature matrix, not a column-slice of X_engineered, making the engineered-vs-original comparison correct even if selection renames/drops original columns.
• Selector is updated via the public set_selected_features() on BaseSelector instead of mutating _selected_features directly.
• 7 new unit tests cover keep/fallback decisions, float-encoded target detection, gate bypass when no selector, and transform() consistency after fallback.

Benchmark runner:

• with_llm is now passed through to apply_featcopilot, so --with-llm works correctly.
• use_feature_cache dead parameter removed from run_single_benchmark() and CLI.
• Time-series/forecast tasks now use TimeSeriesSplit (chronological ordering preserved); --n-seeds is a no-op for those tasks with a first-iteration guard.

690 tests pass.

[Copilot]

Pull request overview

This PR upgrades FeatCopilot’s benchmarking workflow to be more publication-oriented (cross-validation + significance testing + real-world vs synthetic separation) and tightens the feature engineering/selection pipeline with a “do-no-harm” gate and stronger protections for original features.

Changes:

• Add a held-out “do-no-harm” validation gate to AutoFeatureEngineer.fit_transform() and a public selector hook to enforce original-only fallback consistently.
• Fix/adjust feature selection to preserve original features during redundancy elimination and apply stricter refinement thresholds.
• Rework the simple-models benchmark to use k-fold CV + Wilcoxon testing and to report real-world vs synthetic results separately, with new CLI flags.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_sklearn_compat.py	Adds unit tests for the do-no-harm gate and post-fallback transform consistency.
tests/test_selection.py	Adds coverage for set_selected_features() overriding downstream transform().
featcopilot/transformers/sklearn_compat.py	Implements the do-no-harm gate and invokes it after feature selection in fit_transform().
featcopilot/selection/unified.py	Ensures original features are preserved through redundancy filtering; tightens L1 refinement selection threshold.
featcopilot/selection/redundancy.py	Updates redundancy removal logic to never drop original features when correlated with derived features.
featcopilot/core/base.py	Adds BaseSelector.set_selected_features() public API for post-fit selection overrides.
benchmarks/simple_models/run_simple_models_benchmark.py	Switches to k-fold CV + Wilcoxon test; adds --real-world, --n-folds, --n-seeds, --fast; improves report structure.
benchmarks/simple_models/SIMPLE_MODELS_BENCHMARK.md	Updates the benchmark snapshot/report format to match the new methodology and data-source split.
benchmarks/datasets.py	Introduces dataset source tagging and helper APIs (is_real_world, list_real_world_datasets, list_synthetic_datasets).
benchmarks/init.py	Re-exports dataset source constants/helpers via the benchmarks package API.
benchmarks/README.md	Updates benchmark documentation to reflect CV + significance testing + source-separated reporting.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

This PR upgrades FeatCopilot’s benchmarking workflow to cross-validated, statistically tested reporting (with real-world vs synthetic separation), and strengthens the feature-selection pipeline with a “do-no-harm” validation gate plus fixes to preserve original features.

Changes:

• Add a held-out validation “do-no-harm” gate to AutoFeatureEngineer.fit_transform() (with encoding support for categorical/bool/datetime) and a public set_selected_features() hook for selectors.
• Fix feature-selection behavior to never drop original features during redundancy elimination and tighten L1 refinement thresholds.
• Rework the simple-model benchmark to use k-fold CV + Wilcoxon signed-rank significance testing, add dataset source tracking, and extend CLI/reporting.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
featcopilot/transformers/sklearn_compat.py	Adds do-no-harm gate + gate encoding helper and wires gate into fit_transform().
featcopilot/core/base.py	Exposes BaseSelector.set_selected_features() for post-fit selection overrides.
featcopilot/selection/unified.py	Ensures original features are preserved through redundancy filtering; tightens L1 refinement threshold.
featcopilot/selection/redundancy.py	Changes redundancy removal rules to never remove originals when correlated with derived features.
tests/test_sklearn_compat.py	Adds comprehensive unit tests for do-no-harm gate behavior and encoding edge cases.
tests/test_selection.py	Adds coverage for selection overrides, safety branch, and redundancy eliminator branches.
benchmarks/simple_models/run_simple_models_benchmark.py	Switches benchmark to CV + seeds + Wilcoxon; adds real-world filtering and richer report output.
benchmarks/datasets.py	Introduces dataset source tagging and helper APIs to list/filter real-world vs synthetic datasets.
benchmarks/__init__.py	Re-exports new dataset source helpers/constants.
benchmarks/simple_models/SIMPLE_MODELS_BENCHMARK.md	Updates report snapshot and reproduction instructions.
benchmarks/README.md	Updates benchmark methodology and summarizes results with real-world vs synthetic separation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

This PR makes the benchmark suite statistically rigorous (k-fold CV with paired significance testing and source-aware reporting) and strengthens FeatCopilot’s feature selection pipeline with a fail-closed “do-no-harm” gate plus safer selection/refinement behaviors, while also modernizing the codebase to Python 3.10+ typing and tooling.

Changes:

• Reworks simple_models benchmarks to use k-fold CV (with seeds), fold-local preprocessing (leakage-safe), Wilcoxon signed-rank testing, and real-world vs synthetic reporting.
• Adds/extends feature selection safeguards: categorical protection in redundancy elimination, stricter L1 refinement, a do-no-harm validation gate (with gate_n_jobs), and selector pinning via BaseSelector.set_selected_features.
• Updates project metadata/tooling for Python 3.10+ and expands test coverage for benchmark preprocessing/encoding and selection/gate behavior.

Reviewed changes

Copilot reviewed 43 out of 43 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_sklearn_compat.py	Adds extensive end-to-end tests for the do-no-harm gate and _encode_for_gate, plus gate_n_jobs plumbing.
tests/test_selection.py	Adds tests for selector override behavior, categorical baseline scoring defaults, redundancy behavior, and deprecation warning coverage.
tests/test_benchmark_encoding.py	New tests for leakage-safe fold-local encoding/imputation and report bucketing/CLI validation.
pyproject.toml	Raises required Python to 3.10+, updates dev tool versions, and adjusts Black/Ruff/Mypy targets.
featcopilot/utils/validation.py	Modernizes typing (`X
featcopilot/utils/parallel.py	Modernizes typing and imports (collections.abc.Callable).
featcopilot/utils/models.py	Modernizes typing for cached model metadata helpers.
featcopilot/utils/cache.py	Modernizes typing for cache API and internal helpers.
featcopilot/transformers/sklearn_compat.py	Adds do-no-harm gate + _encode_for_gate, introduces gate_n_jobs, and updates typing/get_params.
featcopilot/stores/rule_store.py	Modernizes typing for rule store API.
featcopilot/stores/feast_store.py	Modernizes typing for Feast store integration APIs.
featcopilot/stores/base.py	Modernizes typing for base feature store interfaces.
featcopilot/selection/unified.py	Ensures original features aren’t dropped post-redundancy elimination; tightens L1 refinement thresholds.
featcopilot/selection/statistical.py	Modernizes typing and uses zip(..., strict=True) for safety.
featcopilot/selection/redundancy.py	Enforces “never drop originals” rules, deprecates original_preference with FutureWarning, and updates typing/docs.
featcopilot/selection/importance.py	Modernizes typing and minor internal dict initialization cleanup.
featcopilot/llm/transform_rule_generator.py	Modernizes typing across generator APIs.
featcopilot/llm/semantic_engine.py	Modernizes typing across engine config and methods.
featcopilot/llm/openai_client.py	Modernizes typing across OpenAI client config and APIs.
featcopilot/llm/litellm_client.py	Modernizes typing across LiteLLM client config and APIs.
featcopilot/llm/explainer.py	Modernizes typing across explainer APIs.
featcopilot/llm/copilot_client.py	Modernizes typing across Copilot client APIs.
featcopilot/llm/code_generator.py	Modernizes typing across code generator APIs.
featcopilot/engines/timeseries.py	Modernizes typing in time series engine APIs.
featcopilot/engines/text.py	Modernizes typing and minor dict initialization cleanup.
featcopilot/engines/tabular.py	Modernizes typing in tabular engine APIs.
featcopilot/engines/relational.py	Modernizes typing in relational engine APIs.
featcopilot/core/transform_rule.py	Modernizes typing in transform rule model/methods.
featcopilot/core/registry.py	Modernizes typing in registry getter APIs.
featcopilot/core/feature.py	Modernizes typing in feature and feature set structures.
featcopilot/core/base.py	Adds BaseSelector.set_selected_features hook and modernizes typing across base interfaces.
docs/getting-started/installation.md	Updates documented minimum Python version to 3.10+.
benchmarks/simple_models/run_simple_models_benchmark.py	Implements leakage-safe CV benchmarking + Wilcoxon testing + source-aware report generation + new CLI flags.
benchmarks/simple_models/SIMPLE_MODELS_BENCHMARK.md	Updates the benchmark report snapshot to new format/content.
benchmarks/datasets.py	Adds dataset source tracking and helpers for real-world vs synthetic subsets.
benchmarks/compare_tools/run_fe_tools_comparison.py	Modernizes typing in FE tools comparison benchmark utilities.
benchmarks/automl/run_automl_benchmark.py	Modernizes typing and minor strict zip safety for column sanitization.
benchmarks/init.py	Re-exports new dataset source helpers/constants.
benchmarks/README.md	Updates benchmark methodology and summary reporting to match new scientific workflow.
README.md	Updates documented minimum Python version to 3.10+.
AGENTS.md	Updates repo contributor guidance to Python 3.10+ conventions.
.pre-commit-config.yaml	Updates hook revisions and Ruff hook configuration.
.github/workflows/tests.yml	Updates CI test matrix to Python 3.10–3.13.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[thinkall]

@copilot please review the entire PR (not just the latest commit). The PR has accumulated many changes across multiple iterations and the review should consider the full diff against main.

[Copilot]

Pull request overview

Copilot reviewed 45 out of 46 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[thinkall]

@copilot please review the entire PR (not just the latest commit).

[thinkall]

@copilot please review the entire PR (head commit 085371e). Earlier review request did not trigger the workflow.

[Copilot]

Pull request overview

Copilot reviewed 45 out of 46 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[thinkall]

@copilot please review the entire PR (not just the latest commit).

[Copilot]

Pull request overview

Copilot reviewed 45 out of 46 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 45 out of 46 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[thinkall]

@copilot please review the entire PR (not just the latest commit).

[thinkall]

@copilot please review the entire PR (head 1e82335). Earlier review request did not trigger the workflow.

[Copilot]

Pull request overview

Copilot reviewed 45 out of 46 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[thinkall]

@copilot please review the entire PR (not just the latest commit).

[Copilot]

Pull request overview

Copilot reviewed 45 out of 46 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.