Adversarial feature lifecycle engineering for security and fraud ML
Signal decay · PSI drift · Replacement discovery · Time-aware validation · Champion / challenger\n\n
A fraud or security model can remain statistically “healthy” while the signals underneath it are quietly becoming obsolete. Attackers change devices, spread activity across identities, reduce velocity, rotate infrastructure, and learn which behaviors are easy to detect.
FeatureDecay Lab treats the feature itself as a production asset with a lifecycle:
Signal created → useful → monitored → adversary adapts → value decays
↑ ↓
└──── replacement discovery ← shadow validation ← investigate
The project is intentionally not another fraud classifier. It asks a narrower and more operational question:
Is the signal that drove yesterday's decision still earning trust today—and what should replace it if it is not?
| Layer | What FeatureDecay measures |
|---|---|
| Population | PSI, event-rate movement, weekly feature distribution shift |
| Signal | Single-feature ROC-AUC, normalized signal strength, strength delta, half-life |
| Model | PR-AUC, ROC-AUC, calibration, precision/recall at a fixed review budget |
| Challenger | Random-forest challenger lift vs logistic champion |
| Novel behavior | Isolation Forest anomaly-signal usefulness |
| Decision | Keep, investigate, shadow replacement, or validate before promotion |
The live Streamlit dashboard exposes 30+ model, signal, drift, and lifecycle KPIs rather than only a final model score.
The synthetic generator deliberately creates a controlled behavior change.
device_reuse → strong signal
behavior_graph → weak / emerging signal
geo_velocity → stable supporting signal
A model learns that repeated device reuse is strongly associated with the synthetic adverse outcome.
The adversary adapts:
device_reuse strength ↓↓↓
behavior_graph strength ↑↑↑
IP reputation slightly ↓
geo_velocity relatively stable
FeatureDecay then separates two different questions:
- Did the population move? → PSI
- Did the signal stop predicting the outcome? → time-varying signal strength
Those are not the same thing. A feature can drift without becoming useless, or lose predictive value without a dramatic marginal distribution change.
Feature: device_reuse
Initial strength: high
Current strength: materially lower
Half-life: observed
PSI: elevated
Recommendation:
Investigate → shadow replacement → validate before promotion
Candidate replacement:
behavior_graph
No rule or feature is automatically promoted. The project preserves a human-reviewed decision boundary.
flowchart LR
A[Synthetic security / fraud events] --> B[Feature lifecycle store]
B --> C[Weekly signal diagnostics]
C --> D[Predictive strength]
C --> E[PSI / population shift]
C --> F[Signal half-life]
B --> G[Time-aware model evaluation]
G --> H[Logistic champion]
G --> I[RF challenger]
G --> J[Isolation Forest]
D --> K[Lifecycle policy]
E --> K
F --> K
H --> K
I --> K
J --> K
K --> L[Keep / investigate / shadow replacement]
L -. feedback .-> B
The dashboard is designed as an operational signal-lifecycle surface, not a notebook with disconnected charts.
The top scorecard includes signals such as observations, weeks monitored, current and baseline event rate, event-rate shift, signals tracked/stable/decaying/emerging, current average signal strength, aggregate signal-strength change, strongest current signal, fastest decaying signal, PSI threshold counts, observed signal half-lives, champion PR-AUC/ROC-AUC, precision/recall at the top-10% review budget, Brier calibration score, challenger PR-AUC/lift, anomaly-signal AUC, train/test rows, replacement-candidate count, and synthetic-data boundary.
Signal lifecycle — how predictive strength evolves by week.
Drift & replacement — PSI heatmap plus replacement candidates.
Model health — champion, challenger, anomaly signal, calibration.
Decision view — lifecycle actions with explicit guardrails.
Synthetic data — inspect and export the generated replay population.
Weekly feature strength now includes approximate 95% ROC-AUC intervals and a three-week smoothed trajectory. A feature is classified as decaying or emerging only when the baseline-to-current change exceeds the materiality threshold and persists for at least two recent weeks. Change-point candidates and sustained half-life estimates support investigation; they do not authorize automatic replacement.
For each feature and week, the lab computes single-feature ROC-AUC and converts it into a symmetric diagnostic:
signal_strength = 2 × |AUC - 0.5|
PSI compares the current feature distribution to the baseline week. A signal's half-life is the first monitored week where predictive strength falls to ≤50% of its initial strength. The champion is trained on earlier weeks and evaluated on later weeks so the test set reflects a future population rather than a random shuffle.
The synthetic generator can be replaced with any event-level security, fraud, identity, abuse, or trust-and-safety dataset that has a timestamp, an outcome label, and candidate behavioral signals.
event_id string
observation_time timestamp
label 0/1 or reviewed outcome
feature_1 numeric / encoded
feature_2 numeric / encoded
...
The current engine uses week as the lifecycle window. In a real integration, derive that field from the event timestamp and choose an operational cadence such as daily, weekly, or release-based windows.
| Source | Example signals |
|---|---|
| Identity / IAM | failed-auth velocity, device reuse, impossible travel, MFA method, session novelty |
| Fraud / payments | account age, amount velocity, device-account fanout, beneficiary novelty, geo change |
| Endpoint / EDR | process rarity, signer reputation, parent-child novelty, network destination novelty |
| Email / messaging abuse | sender age, recipient fanout, URL reputation, template similarity, report velocity |
| Cloud / SaaS | permission changes, token use, API-call velocity, resource novelty, geo/device changes |
For enterprise use, the source table can live in Snowflake, BigQuery, Databricks, Spark, a SIEM export, or a feature store. The adapter only needs to normalize data into the lifecycle contract before calling signal_health() and model_health().
import pandas as pd
from engine import signal_health, signal_summary
raw = pd.read_parquet("security_events.parquet")
raw["week"] = pd.to_datetime(raw["event_time"]).dt.isocalendar().week.astype(int)
raw["label"] = raw["confirmed_incident"].astype(int)
# Rename or map real behavioral signals to the feature contract used by the lab.
health = signal_health(raw)
summary = signal_summary(health)For a productionized version, the recommended path is:
warehouse / lake / feature store
↓
validated feature view
↓
FeatureDecay diagnostics
↓
monitoring table + dashboard
↓
shadow replacement evaluation
↓
controlled feature-policy change
The important boundary is that the project does not require proprietary data to demonstrate the method, but the same lifecycle logic can sit on top of real enterprise telemetry once an approved schema and labels are available.
FeatureDecay matters because attackers and fraudsters respond to controls. A feature that was highly discriminative at launch can become weak after users or adversaries change behavior. If the team only watches aggregate model AUC, the deterioration may be hidden by other features until false negatives, review burden, or loss materially increase.
Operationally, this project can help answer:
- Which production signals are losing value before the whole model visibly fails?
- Which new behaviors are becoming more predictive and deserve shadow evaluation?
- Is a metric change caused by population drift, predictive decay, or both?
- Which feature should be investigated or retired first?
- Can a challenger signal replace a decaying feature without increasing false positives or customer friction?
For a fraud/security team, the practical impact is shorter time from adversarial behavior change → detection of signal degradation → validated logic update. That can reduce stale rules, unnecessary analyst review, missed abuse, and emergency retraining. It also gives business partners a concrete story beyond “the model drifted”: which signal changed, how much it changed, what the replacement candidate is, and what evidence is required before promotion.
| Component | Purpose |
|---|---|
| Logistic Regression | transparent champion baseline |
| Random Forest | nonlinear challenger |
| Isolation Forest | unsupervised anomaly signal |
| PSI | population drift diagnostic |
| Single-feature AUC | feature lifecycle diagnostic |
Stable signal
→ keep + monitor
Decaying signal
→ investigate + tighten monitoring
Emerging signal
→ validate as replacement candidate
High PSI
→ investigate population shift before promotion
The system does not automatically retire or promote a production feature.
CI runs the time-aware benchmark on Python 3.10–3.12 and uploads the generated events, signal-health table, lifecycle summary, model metrics, KPIs, and a manifest containing a deterministic dataset SHA-256.
See the benchmark protocol and GitHub Actions.
.
├── app.py
├── engine.py
├── tests/test_engine.py
├── reports/evaluation.md
├── assets/dashboard-preview.svg
├── .streamlit/config.toml
├── .github/workflows/ci.yml
├── Dockerfile
└── requirements.txt
pip install -e '.[dev]'
python engine.py --out artifacts
streamlit run app.pyThe CLI materializes events.csv, signal_health.csv, signal_summary.csv, model_metrics.json, and kpis.json under artifacts/.
FeatureDecay is designed to show end-to-end thinking across predictive modeling, anomaly detection, adversarial trend analysis, novel feature/signal engineering, monitoring production decision inputs, model calibration, champion/challenger evaluation, time-aware validation, controlled logic changes, and operational decision guardrails.
All events, labels, model metrics, and lifecycle outcomes are synthetic. Feature strength is diagnostic rather than causal. PSI identifies distribution movement, not its cause. Any lifecycle recommendation should be validated through shadow testing and human review before a hypothetical production change.