Add numeric stack constraints and telemetry preflight

README.md

@@ -1,40 +1,68 @@
 # porjectx
 
-## Dependency compatibility
+## Reproducible environment bootstrap
 
-The `toptek/requirements-lite.txt` file pins the scientific stack to keep it
-compatible with the bundled scikit-learn release:
+The numeric stack is pinned via [`constraints.txt`](constraints.txt) to avoid the
+ABI/runtime mismatches that previously caused NumPy/SciPy import errors. The
+root [`requirements.txt`](requirements.txt) includes that constraint file and
+pulls in the toolkit's dependencies from `toptek/requirements-lite.txt`.
 
-- `scikit-learn==1.3.2`
-- `numpy>=1.21.6,<1.28`
-- `scipy>=1.7.3,<1.12`
+On Windows, run the helper script to recreate a clean environment and verify the
+stack:
 
-These ranges follow the support window published by scikit-learn 1.3.x and are
-also consumed transitively by `toptek/requirements-streaming.txt` through its
-`-r requirements-lite.txt` include. Installing within these bounds avoids the
-ABI mismatches that occur with the NumPy/SciPy wheels when using newer major
-releases. In particular, upgrading NumPy beyond `<1.28` causes SciPy to raise
-its "compiled against NumPy 1.x" `ImportError`, mirroring the guidance already
-documented in `toptek/README.md`.
+```powershell
+.\scripts\setup_env.ps1
+```
 
-## Verifying the environment
+The script rebuilds `.venv`, installs from `requirements.txt`, then prints
+`STACK_OK` followed by the resolved versions in JSON form. The check ensures the
+runtime matches `numpy==1.26.4`, `scipy==1.10.1`, and `scikit-learn==1.3.2`
+exactly alongside compatible `pandas`, `joblib`, and `threadpoolctl` wheels.
 
-Use Python **3.10 or 3.11**—matching the guidance in `toptek/README.md`'s
-quickstart—to stay within the wheel support window for SciPy and
-scikit-learn 1.3.x. Python 3.12 is currently unsupported because prebuilt
-SciPy/scikit-learn wheels for that interpreter depend on NumPy ≥1.28 and
-SciPy ≥1.12, which exceed this project's pinned ranges. Create and activate a
-compatible virtual environment, then install and check for dependency issues:
+For POSIX shells the equivalent manual steps are:
 
 ```bash
 python -m venv .venv
 source .venv/bin/activate
 pip install --upgrade pip
-pip install -r toptek/requirements-lite.txt
-pip check
+pip install -r requirements.txt
 ```
 
-The final `pip check` call should report "No broken requirements found",
-confirming that the pinned dependency set resolves without conflicts. Users on
-Python 3.12 should downgrade to Python 3.10/3.11 or wait for a dependency
-refresh that supports NumPy ≥1.28 and SciPy ≥1.12 before proceeding.
+## Runtime telemetry and guardrails
+
+The entry point now executes `toptek.core.utils.assert_numeric_stack()` and
+`toptek.core.utils.set_seeds(42)` during startup. Version validation writes a
+structured report to `reports/run_stack.json` so crash reports include the exact
+Python and numeric-library versions. Structured logging is initialised via
+`logging.basicConfig` with a rotating file handler targeting
+`logs/toptek_YYYYMMDD.log` alongside console output, keeping telemetry for both
+CLI and GUI sessions.
+
+## UI configuration surface
+
+The manual trading shell and Tkinter dashboard read defaults from
+[`configs/ui.yml`](configs/ui.yml). The file ships with sensible demo values so
+the GUI renders without external data sources:
+
+- `appearance` — theme token (currently `dark`) and accent family used by the
+  style registry.
+- `shell` — defaults for the research symbol/timeframe, training lookback,
+  calibration flag, simulated backtest window, and preferred playbook.
+- `chart` — LiveChart refresh cadence (`fps`), point budget, and price
+  precision used by streaming widgets.
+- `status` — copy for the status banners shown in the Login, Train, Backtest,
+  and Guard tabs so product teams can retune messaging without touching code.
+
+Operators can override the YAML at runtime with environment variables or CLI
+flags:
+
+- Environment variables follow the `TOPTEK_UI_*` convention, e.g.
+  `TOPTEK_UI_SYMBOL`, `TOPTEK_UI_INTERVAL`, `TOPTEK_UI_LOOKBACK_BARS`,
+  `TOPTEK_UI_CALIBRATE`, `TOPTEK_UI_FPS`, and `TOPTEK_UI_THEME`.
+- CLI switches (`--symbol`, `--timeframe`, `--lookback`, `--model`, `--fps`)
+  apply the same overrides for one-off runs and are reflected back into the GUI
+  when it launches.
+
+These controls keep the default Topstep demo intact while making it easy to
+point the toolkit at alternative markets or stress-test higher frequency charts
+without editing source files.

configs/ui.yml

@@ -0,0 +1,31 @@
+appearance:
+  theme: dark
+  accent: violet
+shell:
+  symbol: ES=F
+  interval: 5m
+  research_bars: 240
+  lookback_bars: 480
+  calibrate: true
+  model: logistic
+  simulation_bars: 720
+  playbook: momentum
+chart:
+  fps: 12
+  max_points: 180
+  price_decimals: 2
+status:
+  login:
+    idle: "Awaiting verification"
+    saved: "Saved. Run verification to confirm access."
+    verified: "All keys present. Proceed to Research ▶"
+  training:
+    idle: "Awaiting training run"
+    success: "Model artefact refreshed. Continue to Backtest ▶"
+  backtest:
+    idle: "No simulations yet"
+    success: "Sim complete. If expectancy holds, draft a manual trade plan ▶"
+  guard:
+    pending: "Topstep Guard: pending review"
+    intro: "Manual execution only. Awaiting guard refresh..."
+    defensive_warning: "DEFENSIVE_MODE active. Stand down and review your journal before trading."

constraints.txt

@@ -0,0 +1,6 @@
+numpy==1.26.4
+scipy==1.10.1
+scikit-learn==1.3.2
+joblib>=1.3,<2
+threadpoolctl>=3,<4
+pandas>=1.5,<2.3

pyproject.toml

@@ -0,0 +1,3 @@
+[build-system]
+requires = ["setuptools>=61", "wheel"]
+build-backend = "setuptools.build_meta"

requirements.txt

@@ -0,0 +1,2 @@
+-c constraints.txt
+-r toptek/requirements-lite.txt

scripts/setup_env.ps1

@@ -0,0 +1,33 @@
+param(
+    [string]$Python = "py -3.11"
+)
+
+$ErrorActionPreference = "Stop"
+
+if (Test-Path ".venv") {
+    Remove-Item ".venv" -Recurse -Force
+}
+
+& $Python -m venv .venv
+
+$venvPath = Join-Path (Resolve-Path ".venv").Path "Scripts"
+$venvPython = Join-Path $venvPath "python.exe"
+
+& $venvPython -m pip install --upgrade pip
+& $venvPython -m pip install -r requirements.txt
+
+$stackCheck = @"
+import importlib
+import json
+import platform
+
+modules = ["numpy", "scipy", "sklearn", "pandas", "joblib", "threadpoolctl"]
+versions = {name: importlib.import_module(name).__version__ for name in modules}
+print("STACK_OK")
+print(json.dumps({
+    "python": platform.python_version(),
+    "versions": versions,
+}, indent=2))
+"@
+
+& $venvPython -c $stackCheck

tests/test_training_pipeline_integration.py

@@ -65,11 +65,26 @@ def fake_train_classifier(X, y, **kwargs):
 
     monkeypatch.setattr("toptek.main.build_features", fake_build_features)
     monkeypatch.setattr("toptek.main.model.train_classifier", fake_train_classifier)
-    monkeypatch.setattr("toptek.main.data.sample_dataframe", lambda: _sample_dataframe(140))
+    monkeypatch.setattr(
+        "toptek.main.data.sample_dataframe", lambda: _sample_dataframe(140)
+    )
 
-    args = argparse.Namespace(cli="train", model="logistic", symbol="ES", timeframe="5m", lookback="90d", start=None)
+    args = argparse.Namespace(
+        cli="train",
+        model="logistic",
+        symbol="ES",
+        timeframe="5m",
+        lookback="90d",
+        start=None,
+    )
     configs = {"risk": {}, "app": {}, "features": {}}
-    paths = utils.AppPaths(root=tmp_path, cache=tmp_path / "cache", models=tmp_path / "models")
+    paths = utils.AppPaths(
+        root=tmp_path,
+        cache=tmp_path / "cache",
+        models=tmp_path / "models",
+        logs=tmp_path / "logs",
+        reports=tmp_path / "reports",
+    )
 
     run_cli(args, configs, paths)
 
@@ -122,16 +137,26 @@ def fake_train_classifier(X, y, **kwargs):
         )
 
     monkeypatch.setattr("toptek.gui.widgets.build_features", fake_build_features)
-    monkeypatch.setattr("toptek.gui.widgets.sample_dataframe", lambda rows: _sample_dataframe(rows))
-    monkeypatch.setattr("toptek.gui.widgets.model.train_classifier", fake_train_classifier)
+    monkeypatch.setattr(
+        "toptek.gui.widgets.sample_dataframe", lambda rows: _sample_dataframe(rows)
+    )
+    monkeypatch.setattr(
+        "toptek.gui.widgets.model.train_classifier", fake_train_classifier
+    )
     monkeypatch.setattr("tkinter.messagebox.showwarning", lambda *args, **kwargs: None)
     monkeypatch.setattr("tkinter.messagebox.showinfo", lambda *args, **kwargs: None)
 
     notebook = ttk.Notebook(root)
     notebook.pack()
 
     configs: dict[str, dict[str, object]] = {}
-    paths = utils.AppPaths(root=tmp_path, cache=tmp_path / "cache", models=tmp_path / "models")
+    paths = utils.AppPaths(
+        root=tmp_path,
+        cache=tmp_path / "cache",
+        models=tmp_path / "models",
+        logs=tmp_path / "logs",
+        reports=tmp_path / "reports",
+    )
 
     tab = TrainTab(notebook, configs, paths)
     tab.calibrate_var.set(False)

tests/test_ui_config.py

@@ -0,0 +1,46 @@
+from __future__ import annotations
+from pathlib import Path
+
+import pytest
+
+from core import ui_config
+
+
+def test_load_ui_config_defaults(tmp_path: Path) -> None:
+    path = tmp_path / "ui.yml"
+    path.write_text("{}\n", encoding="utf-8")
+    cfg = ui_config.load_ui_config(path, env={})
+    assert cfg.shell.symbol == "ES=F"
+    assert cfg.shell.interval == "5m"
+    assert cfg.shell.research_bars == 240
+    assert cfg.chart.fps == 12
+    assert cfg.status.login.idle == "Awaiting verification"
+    assert cfg.appearance.theme == "dark"
+
+
+def test_load_ui_config_env_overrides(tmp_path: Path) -> None:
+    path = tmp_path / "ui.yml"
+    path.write_text(
+        "shell:\n  symbol: ES=F\n  calibrate: true\nchart:\n  fps: 8\n",
+        encoding="utf-8",
+    )
+    env = {
+        "TOPTEK_UI_SYMBOL": "NQ=F",
+        "TOPTEK_UI_CALIBRATE": "false",
+        "TOPTEK_UI_LOOKBACK_BARS": "960",
+        "TOPTEK_UI_FPS": "24",
+        "TOPTEK_UI_THEME": "dark",
+    }
+    cfg = ui_config.load_ui_config(path, env=env)
+    assert cfg.shell.symbol == "NQ=F"
+    assert cfg.shell.calibrate is False
+    assert cfg.shell.lookback_bars == 960
+    assert cfg.chart.fps == 24
+    assert cfg.appearance.theme == "dark"
+
+
+def test_load_ui_config_validation(tmp_path: Path) -> None:
+    path = tmp_path / "ui.yml"
+    path.write_text("chart:\n  fps: 0\n", encoding="utf-8")
+    with pytest.raises(ValueError):
+        ui_config.load_ui_config(path, env={})

tests/test_utils_stack.py

@@ -0,0 +1,68 @@
+"""Tests for numeric stack validation and logging utilities."""
+
+import json
+import logging
+from logging.handlers import RotatingFileHandler
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from toptek.core import utils
+
+
+def test_assert_numeric_stack_writes_report(tmp_path: Path) -> None:
+    reports_dir = tmp_path / "reports"
+    versions = utils.assert_numeric_stack(reports_dir=reports_dir)
+
+    report_path = reports_dir / "run_stack.json"
+    assert report_path.exists()
+
+    payload = json.loads(report_path.read_text(encoding="utf-8"))
+    assert payload["status"] == "ok"
+    assert payload["required"]["numpy"] == versions["numpy"]
+    assert payload["expected"]["scipy"] == utils.STACK_REQUIREMENTS["scipy"]
+
+
+def test_assert_numeric_stack_raises_on_mismatch(
+    monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+) -> None:
+    monkeypatch.setitem(utils.STACK_REQUIREMENTS, "numpy", "0.0.0")
+
+    with pytest.raises(RuntimeError) as excinfo:
+        utils.assert_numeric_stack(reports_dir=tmp_path)
+
+    assert "scripts/setup_env.ps1" in str(excinfo.value)
+
+    report = json.loads((tmp_path / "run_stack.json").read_text(encoding="utf-8"))
+    assert report["status"] == "error"
+
+
+def test_set_seeds_reproducible() -> None:
+    utils.set_seeds(123)
+    first = np.random.random(3)
+    utils.set_seeds(123)
+    second = np.random.random(3)
+
+    assert np.allclose(first, second)
+
+
+def test_configure_logging_installs_rotating_handler(tmp_path: Path) -> None:
+    root_logger = logging.getLogger()
+    original_handlers = list(root_logger.handlers)
+    for handler in original_handlers:
+        root_logger.removeHandler(handler)
+
+    try:
+        log_path = utils.configure_logging(tmp_path, level="INFO")
+        assert log_path.exists()
+        assert any(
+            isinstance(handler, RotatingFileHandler)
+            for handler in logging.getLogger().handlers
+        )
+    finally:
+        for handler in logging.getLogger().handlers:
+            handler.close()
+        logging.getLogger().handlers.clear()
+        for handler in original_handlers:
+            logging.getLogger().addHandler(handler)

toptek/README.md

@@ -10,14 +10,12 @@ Toptek is a Windows-friendly starter kit for working with the ProjectX Gateway (
 
 ```powershell
 # Windows, Python 3.11
-py -3.11 -m venv .venv
+.\scripts\setup_env.ps1
 .venv\Scripts\activate
-pip install --upgrade pip
-pip install -r requirements-lite.txt
 
-copy .env.example .env
+copy toptek\.env.example .env
 # edit PX_* in .env OR use GUI Settings
-python main.py
+python toptek\main.py
 ```
 
 ## CLI usage examples
@@ -62,7 +60,9 @@ Configuration defaults live under the `config/` folder and are merged with value
 
 ## Requirements profiles
 
-- `requirements-lite.txt`: minimal dependencies for polling workflows. NumPy is capped below 1.28 so the bundled SciPy wheels stay importable; installing NumPy 2.x triggers a SciPy `ImportError` about missing manylinux-compatible binaries.
+- `../constraints.txt`: pins NumPy 1.26.4, SciPy 1.10.1, scikit-learn 1.3.2 plus compatible `pandas`, `joblib`, and `threadpoolctl` wheels.
+- `../requirements.txt`: references the constraint file and pulls in the lite dependency set.
+- `requirements-lite.txt`: minimal dependencies for polling workflows (consumed via the root requirements).
 - `requirements-streaming.txt`: extends the lite profile with optional SignalR streaming support.
 
 ## Development notes

toptek/config/app.yml

@@ -2,3 +2,5 @@ polling_interval_seconds: 5
 cache_directory: data/cache
 models_directory: models
 log_level: INFO
+logs_directory: logs
+reports_directory: reports