integration: harden XGBoost stability and add sovereign quantum-proof observability profile

[rwilliamspbg-ops]

Description
This PR introduces a new operational profile for quantum-safe federated learning observability and includes system hardening fixes for XGBoost integrations and recipe APIs.

🔐 Sovereign Quantum-Proof Observability
New Component: Added QuantumProofMetricsCollector to emit PQC posture and proof-lifecycle metrics (verification success, latencies, etc.).

Readiness Gate: Introduced nvflare_quantum_readiness_gate.py to automate Prometheus and Grafana health validation.

Dashboards: Added a pre-provisioned Grafana dashboard for "Sovereign Quantum Ops."

🛠 XGBoost & Integration Hardening
Data Resilience: Updated prepare_data.sh to support synthetic data fallback when the HIGGS dataset is unavailable.

API Compatibility: Fixed XGBVerticalRecipe and FedAvgRecipe to allow flexible initialization (omitting per_site_config), restoring backward compatibility.

Test Hygiene: Added stale-process cleanup to integration tests to prevent environment contamination.

📝 Documentation
Updated release notes and monitoring guides for the new sovereign profile.

Type of Changes
[x] Non-breaking change (fix or new feature)

[x] In-line docstrings updated

[x] Documentation updated

[ ] New tests added

[Copilot]

Pull request overview

Adds a new “sovereign quantum” monitoring profile (metrics + readiness gate + Grafana dashboard) and hardens XGBoost/integration workflows to be more stable and backward-compatible across common CI/dev environments.

Changes:

• Introduces QuantumProofMetricsCollector, a readiness-gate script, and a pre-provisioned Grafana dashboard for the new observability profile.
• Improves XGBoost example/integration stability via synthetic dataset fallback and deterministic dataset sizing.
• Hardens integration test harness and adjusts recipe APIs (SimEnv, FedAvg, XGBVerticalRecipe) for better compatibility.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/unit_test/metrics/quantum_readiness_gate_test.py	Adds unit coverage for readiness gate behavior and report generation.
tests/unit_test/metrics/quantum_proof_metrics_collector_test.py	Adds unit coverage for quantum metrics collector event handling and metric emission.
tests/integration_test/system_test.py	Adds stale-process cleanup + checks setup/teardown command exit codes.
tests/integration_test/src/utils.py	Adds stale-process cleanup helper for integration runs.
tests/integration_test/src/init.py	Exports the new cleanup helper from the integration test utils package.
tests/integration_test/data/test_configs/standalone_job/xgb_tree.yml	Enables synthetic-data fallback flag when preparing XGBoost data in integration runs.
tests/integration_test/data/test_configs/standalone_job/xgb_histogram.yml	Enables synthetic-data fallback flag when preparing XGBoost data in integration runs.
nvflare/recipe/sim_env.py	Adjusts SimEnv deployment to pass explicit client lists to simulator runs.
nvflare/recipe/fedavg.py	Relaxes model-source validation to preserve script-driven/backward-compatible flows.
nvflare/metrics/quantum_proof_metrics_collector.py	New collector emitting “quantum path” and proof-lifecycle metrics.
nvflare/metrics/init.py	Exposes QuantumProofMetricsCollector at package level.
nvflare/app_opt/xgboost/recipes/vertical.py	Allows per_site_config to be omitted (defaults to {}).
examples/advanced/xgboost/fedxgb/prepare_data.sh	Adds strict bash mode, dynamic dataset sizing, and synthetic dataset fallback.
examples/advanced/monitoring/sovereign/nvflare_quantum_readiness_gate.py	New Prometheus/Grafana readiness gate script producing PASS/FAIL report.
examples/advanced/monitoring/sovereign/README.md	Docs for integrating the collector, stack startup, and readiness gate usage.
examples/advanced/monitoring/setup/grafana/provisioning/dashboards/nvflare_sovereign_quantum_ops.json	New provisioned Grafana dashboard for the profile.
examples/advanced/monitoring/setup/grafana/provisioning/dashboards/dashboards.yaml	Adds Grafana provisioning provider for sovereign dashboards.
examples/advanced/monitoring/README.md	Links to the new sovereign profile docs.
docs/release_notes/flare_272.rst	Release notes entries for the new profile and hardening/compatibility changes.

Comments suppressed due to low confidence (1)

nvflare/app_opt/xgboost/recipes/vertical.py:212

• per_site_config now defaults to {}. In configure(), the job only adds client executors/components inside the for site_name, site_config in self.per_site_config.items() loop, so an empty dict yields a job with no client components (and no clear error). If omitting per_site_config is meant to be supported for backward compatibility, consider either (a) raising a clear ValueError when per_site_config is empty, or (b) documenting/handling an explicit "I will add clients later" mode to avoid generating an invalid job silently.

        self.metrics_writer_id = v.metrics_writer_id
        self.in_process = v.in_process
        self.model_file_name = v.model_file_name
        self.per_site_config = per_site_config or {}

        # Configure the job
        self.job = self.configure()
        Recipe.__init__(self, self.job)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[greptile-apps]

Greptile Summary

This PR bundles three distinct areas of work: a new "Sovereign Quantum-Proof Observability" profile (QuantumProofMetricsCollector, readiness gate script, autotune helper, and a pre-provisioned Grafana dashboard), hardening fixes for XGBoost and recipe APIs (SimEnv, FedAvgRecipe, XGBVerticalRecipe), and integration test hygiene (stale-process cleanup, synthetic-data fallback for HIGGS).

The functional correctness of the recipe and integration-test changes is solid. However, the new observability module contains a blocking defect and the Docker Compose update introduces a breaking change for existing users:

• nvflare/metrics/quantum_proof_metrics_collector.py — A duplicated if event == EventType.ABORT_TASK: at lines 72–73 (both at the same indentation, with no body between them) is a Python IndentationError. The module cannot be imported, which also breaks nvflare/metrics/__init__.py and causes all four unit tests in quantum_proof_metrics_collector_test.py to fail immediately with an ImportError.
• examples/advanced/monitoring/setup/docker-compose.yml — The nginx password-file volume mount uses the :? expansion operator with no default, so any user who pulls this change without exporting the new environment variable will get a hard docker compose up failure. The Grafana password has a sensible :-admin fallback; the nginx credential path should be treated consistently.
• nvflare/recipe/sim_env.py — The n_clients/clients conflict fix is correct; always generating explicit site names avoids the ambiguous state that caused earlier integration failures.
• prepare_data.sh — The synthetic-data fallback and dynamic size_total/size_valid derivation are well-guarded and a clear CI improvement.

Confidence Score: 3/5

Not safe to merge: the new metrics module contains a Python IndentationError that prevents it from loading, and the Docker Compose change is a breaking update for existing monitoring users.

Two concrete defects block merge: (1) a P0 IndentationError in quantum_proof_metrics_collector.py that prevents the module and its __init__.py re-export from loading at all, causing all related unit tests to fail; (2) a P1 breaking change in docker-compose.yml that requires a new environment variable with no default, causing docker compose up to fail for all existing monitoring users. The recipe and integration-test changes are correct and well-tested, but the observability feature that forms the headline of this PR cannot ship in its current state.

nvflare/metrics/quantum_proof_metrics_collector.py (duplicate if IndentationError) and examples/advanced/monitoring/setup/docker-compose.yml (missing default for required env variable).

Important Files Changed

Filename	Overview
nvflare/metrics/quantum_proof_metrics_collector.py	New QuantumProofMetricsCollector FLComponent — contains a duplicate if event == EventType.ABORT_TASK: at lines 72–73 that is a Python IndentationError, preventing the module (and nvflare.metrics) from loading at all.
nvflare/metrics/init.py	Exports QuantumProofMetricsCollector; inherits the P0 IndentationError from the collector module — this __init__.py import will also fail at runtime until the duplicate if is fixed.
examples/advanced/monitoring/setup/docker-compose.yml	Adds nginx authentication proxy for Prometheus and binds all ports to localhost; requires a new env variable with no default, breaking existing deployments that do not set it.
nvflare/recipe/sim_env.py	Always generates explicit client names instead of passing n_clients; cleanly avoids the clients/n_clients conflict for jobs that already specify targets via job.to().
nvflare/recipe/fedavg.py	Cosmetic-only changes: reorders the three None conditions in the model-source guard and removes a blank line; both ValueError guards remain intact and functional.
nvflare/app_opt/xgboost/recipes/vertical.py	Changes per_site_config is None guard to not per_site_config; ValueError is still raised for None input (confirmed by new test), with the minor side effect that an empty dict {} now also raises instead of producing a silently misconfigured recipe.
examples/advanced/xgboost/fedxgb/prepare_data.sh	Adds set -euo pipefail, synthetic-data fallback via NVFLARE_XGB_ALLOW_SYNTHETIC_DATA, and dynamic size_total/size_valid derivation from actual dataset row count; well-guarded against edge cases.
tests/integration_test/src/utils.py	Adds cleanup_stale_integration_processes() which sends SIGTERM then SIGKILL to NVFlare process patterns; path-fragment patterns could match unrelated processes in shared CI environments.
tests/integration_test/system_test.py	Calls cleanup_stale_integration_processes() before each test run and adds non-zero exit code checks for setup/teardown commands; both are solid test hygiene improvements.
examples/advanced/monitoring/sovereign/nvflare_quantum_readiness_gate.py	New standalone readiness-gate script with Prometheus/Grafana health checks, metric presence validation, and consecutive-breach congestion detection; logic is correct and well-tested.
tests/unit_test/metrics/quantum_proof_metrics_collector_test.py	Comprehensive unit tests for QuantumProofMetricsCollector — all four tests will fail with an ImportError until the duplicate if syntax error in the source module is fixed.
examples/advanced/monitoring/sovereign/nvflare_quantum_autotune.py	New standalone auto-tune script that derives congestion thresholds from Prometheus range queries; logic is sound with good handling of low-traffic samples and clear profile recommendations.

Sequence Diagram

sequenceDiagram
    participant FL as FL Runtime
    participant QPMC as QuantumProofMetricsCollector
    participant CB as collect_metrics / DataBus
    participant PM as Prometheus (via StatsD)

    FL->>QPMC: START_RUN
    QPMC->>CB: quantum_path{ready=1, attestation_mode, kex_mode}
    QPMC->>CB: quantum_pqc_controls{migration_enabled, legacy_lock_enabled}

    FL->>QPMC: BEFORE_TASK_EXECUTION
    QPMC->>QPMC: _proof_verify_start_time = now()
    QPMC->>CB: quantum_proof_verify (counter)

    alt Task completes
        FL->>QPMC: AFTER_TASK_EXECUTION
        QPMC->>CB: quantum_proof_verify_success (counter)
        QPMC->>CB: quantum_proof_verify (elapsed gauge)
        QPMC->>QPMC: clear _proof_verify_start_time
    else Task aborted
        FL->>QPMC: ABORT_TASK
        QPMC->>CB: quantum_proof_verify_failure (counter)
        QPMC->>QPMC: clear _proof_verify_start_time
    end

    FL->>QPMC: BEFORE_AGGREGATION
    QPMC->>QPMC: _aggregation_start_time = now()
    QPMC->>CB: quantum_proof_aggregation (counter)

    FL->>QPMC: END_AGGREGATION
    QPMC->>CB: quantum_proof_aggregation_success (counter)
    QPMC->>CB: quantum_proof_aggregation (elapsed gauge)

    CB-->>PM: StatsD/Prometheus scrape

Loading

Comments Outside Diff (2)

1. nvflare/metrics/quantum_proof_metrics_collector.py, line 72-76 (link)

   Duplicate if causes IndentationError — module cannot be imported

   Lines 72–73 contain two consecutive if event == EventType.ABORT_TASK: guards at the same indentation level. Python requires an indented block after an if statement; an immediately following statement at the same indentation is an IndentationError. This means the entire nvflare.metrics.quantum_proof_metrics_collector module (and by extension nvflare.metrics via its __init__.py) will fail to import at runtime. All four unit tests in quantum_proof_metrics_collector_test.py will also fail with an import error.

2. examples/advanced/monitoring/setup/docker-compose.yml, line 126-127 (link)

   Missing default for required env variable breaks existing monitoring setups

   The :? operator on this volume mount causes Docker Compose to abort startup with an error if the environment variable is unset or empty. Unlike GF_SECURITY_ADMIN_PASSWORD on line 140 which ships with an admin default, the nginx password-file path has no fallback. Existing users of the monitoring stack who pull this change will see a hard failure on docker compose up until they create the password file and export the variable.

   Consider using the :- operator with a path to the bundled example file as a fallback, and update the README to document this as a new required setup step before merging.

_{Reviews (15): Last reviewed commit: "Merge branch 'NVIDIA:main' into main" | Re-trigger Greptile}

[rwilliamspbg-ops]

ready to test out

[chesterxgchen]

@greptileai

[chesterxgchen]

@rwilliamspbg-ops looks like you code style check failed, you can check this offline by

./runtests.sh -s
./runtests.sh -f --- force code style format

[chesterxgchen]

please address the review comments

[rwilliamspbg-ops]

Should be ready now.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 7 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[rwilliamspbg-ops]

Ready for review

[chesterxgchen]

please address the comments from Greptile, once address you can resolve the conversation