Personalization Feedback Engine

A recommendation engine that learns from every click while it is serving, and shows you it is learning.

---

The problem

Most recommenders are retrained on a schedule. A batch job runs overnight, reads yesterday's interactions, fits a model, and ships it in the morning.

So when a customer's taste changes at nine in the morning, the system keeps recommending yesterday's suggestions for the rest of the day. It saw every click that proved it was wrong and could not act on any of them, because acting means waiting for the next batch window. A new user is worse off still: they get generic popularity until they have accumulated enough history to appear in a training set that has not been built yet.

The interactions were never the bottleneck. The retraining schedule was.

What this does

The model updates inside the request that reports the interaction. Click an item, and the next recommendation for that user is computed from weights that already include the click. There is no batch boundary in between and no retrain button to press, because there is nothing to wait for.

• Per-interaction learning. Every click, skip, purchase and rating updates a per-user online model immediately.
• Adapts within a session. A user whose taste shifts mid-session gets different recommendations before they leave.
• Explains itself. Each item carries a reason a user could check, and the system says plainly when it is serving popularity because it does not know them yet.
• Measured live. Click-through is attributed back to the exact list that earned it, so per-strategy performance comes from real outcomes rather than offline scores alone.

---

The evidence

Everything below is measured against simulated users whose preference vectors the recommenders never see, so each number is checkable against an answer key. Run it yourself with python examples/demo.py — it takes about eight seconds.

Learning without retraining

Twenty users, a catalog of 300 items, and one online model each. The metric is the hit rate: of the ten items served, how many fall in that user's genuine top twenty. A system that has learned nothing scores 0.067.

	hit rate
before any feedback	0.045
after 300 interactions each	0.390
retraining runs in between	0

8.7x better, and 20 of 20 users improved. None got worse.

The curve, averaged across the cohort:

  interactions   hit rate    spread  curve
             0      0.045     0.074  #####
            25      0.070     0.123  #######
            75      0.225     0.189  #######################
           150      0.260     0.169  ###########################
           300      0.390     0.137  ########################################

Fifty interactions is not enough for this learner to converge, which is why the curve runs to a few hundred rather than stopping where it first looks flat.

Recovering from drift

Forty users have their hidden preferences shifted by 0.8, and the system keeps serving. The online model is updated by the traffic it serves. The batch model is left alone, exactly as it would be between nightly refits.

	online	batch
before the shift	0.200	0.460
immediately after the shift	0.068	0.145
after serving the cohort again	0.150	0.145
after a batch retrain	0.150	0.173

The online model recovered +0.082 from feedback alone. The batch model moved +0.000 over the same period, because nothing refitted it.

Refitting it recovered only +0.028 — less than the online model managed. A refit is fitted over all history, and most of that history predates the shift and still describes the taste these users have moved away from. Incremental updates carry no such weight: they only ever saw the last few hundred events, which are the ones still true.

Where each strategy wins

The whole population, every strategy scored on the same lists:

strategy	hit rate	ndcg@10	mrr	coverage	diversity	CTR
collaborative	0.427	0.868	0.784	0.620	0.381	0.328
content	0.525	0.892	0.923	0.917	0.268	0.332
online	0.226	0.756	0.423	0.750	0.311	0.297
hybrid	0.344	0.818	0.619	0.890	0.319	0.319
random	0.067

The online learner trails here, and that is not hidden. Each user supplied about eleven events in this comparison, and the curve above shows it needs a few hundred to converge. It wins where it was measured properly — within a session, and when taste changes.

---

Architecture

flowchart TB
    UI["Next.js UI :3001"]
    API["FastAPI :8000"]
    KAFKA["Kafka / Redpanda<br/><i>optional</i>"]
    PROC["feedback processor"]
    ONLINE["online learner<br/><b>updates per interaction</b>"]
    COLLAB["collaborative filter<br/><i>batch</i>"]
    CONTENT["content-based filter<br/><i>batch</i>"]
    HYBRID["hybrid blend<br/><i>normalised scores</i>"]
    RANK["ranker<br/>diversity, freshness"]
    EXPLAIN["explainer"]
    FS[("feature store<br/>:8001")]
    CANARY[("canary deploy<br/>:8002")]
    PG[("PostgreSQL<br/>served lists, clicks")]

    UI -->|"click, skip, purchase"| API
    API --> KAFKA
    KAFKA --> PROC
    API -->|"direct feedback"| PROC
    PROC ==>|"learn_one"| ONLINE

    ONLINE --> HYBRID
    COLLAB --> HYBRID
    CONTENT --> HYBRID
    FS -.->|"user features"| HYBRID
    HYBRID --> RANK --> EXPLAIN --> API
    API --> UI

    CANARY -.->|"traffic split"| API
    API -->|"served list"| PG
    PROC -->|"attribute click"| PG
    PG -.->|"batch refit"| COLLAB
    PG -.->|"batch refit"| CONTENT

    linkStyle 4 stroke:#0a7d32,stroke-width:3px

Loading

The thick edge is the whole argument. An interaction reaches learn_one inside the request that reported it, so the next list the API serves is computed from weights that already include it. The dotted edges into the batch models are the slow path everything else relies on: they refit on a schedule, and the online layer sits on top of them.

Screenshots

Cold start. A new user, no history. The system says so rather than pretending otherwise.

After feedback. The same user after eight clicks. The interaction count has moved, the cold-start tag has cleared, and the reasons are personalised.

Strategy comparison. Click-through by strategy, each with its standard error, and the experiment's verdict.

---

How this differs from a batch recommender

The comparison is worth stating directly rather than leaving implied. Both columns describe systems in this repository: the collaborative and content-based filters here are fitted in batch, exactly as a conventional recommender would be, and the online layer sits on top of them.

	a batch-only recommender	this engine
when the model changes	on a schedule	on every interaction
a new user's first session	generic until the next training run	personalised within a few clicks
taste changes mid-session	invisible until retrained	reflected in the next request
what a click does	waits in a table	updates the model, then is stored
explanation	usually none	a reason per item, checkable against history
how it is measured	offline metrics on a held-out split	offline metrics and click-through attributed to served lists

The online layer does not replace the batch models, and the numbers above are the reason. Collaborative and content-based filtering supply the base a new user is served from, and they beat the online learner outright when a user has only supplied a handful of events. What the online layer adds is the ability to move now, which is worth most exactly when the batch models are most wrong: for a user they have never seen, and for one whose taste has just changed.

The three strategies, and when each wins

Collaborative filtering — "people who chose this also chose that". Strong once a user has overlap with others; knows nothing about a brand new item, and nothing about what any item actually is. Similarity is shrunk toward zero when the overlap is thin, because a single coincidental co-click is not evidence of anything.

Content-based filtering — "similar to what you chose before". Handles new items immediately, since it reads their features rather than their history. Its weakness is that it only ever suggests more of the same, which is visible in its diversity score above, the lowest of the four.

Online learning — "your recent behaviour points this way". The only one that can react inside a session, and the only one that recovers from drift without being refitted. Needs a few hundred interactions per user to converge, and is honest about serving popularity below that threshold.

The hybrid blends all three on normalised scores, because their raw scores are not comparable: collaborative scores are unbounded sums of similarities, content scores are cosines in [-1, 1], and online scores are probabilities. Averaging them directly would let whichever model produced the largest numbers decide the ranking regardless of the weights.

Honest evaluation

Every offline number here is computed against simulated users with clean, stable preference vectors, over a catalog of 300 items where each user has a well-defined right answer.

None of that holds in production. Real preferences are noisier, real catalogs are orders of magnitude larger, and there is no answer key at all. These numbers establish that the mechanism works. They are not a forecast of live performance.

What does transfer is the comparison between strategies under identical conditions, and the shape of the learning curve. What does not transfer is the absolute value of any single metric.

This is why the system also measures itself online, from lists it genuinely served and clicks attributed back to them, and why every click-through figure is reported with its standard error. Two strategies differing by a point of CTR over two hundred impressions have not been distinguished from each other, and the experiment layer refuses to name a winner until each arm carries at least a thousand impressions.

The /metrics/learning/{user_id} route deliberately does not report accuracy against ground truth. Live traffic has no ground truth, and inventing one would be worse than omitting it.

---

Quick start

Requires the shared platform stack (PostgreSQL, Redis, Redpanda) to be running.

pip install -r requirements.txt
cp .env.example .env

# The evidence above, start to finish, in about eight seconds
python examples/demo.py

# The API
uvicorn serving.api:app --reload --port 8000

# The interface, in another terminal
cd frontend && npm install && npm run dev

Then open http://localhost:3001, type a user name, and start clicking. The interaction count moves with each click, and the cold-start tag clears once the model has enough to work with.

With Docker:

docker compose up --build

API reference

method	route	what it does
GET	/health	catalog size, users learned, which upstreams answered
GET	/recommend/{user_id}	a ranked list with scores, reasons, and the strategy that served it
POST	/interactions	record an interaction; the model is updated before this returns
GET	/users/{user_id}/profile	learned weights, interaction count, top categories
GET	/catalog	items available to recommend
GET	/metrics/online	click-through and conversion by strategy
GET	/metrics/learning/{user_id}	how much one user's model has absorbed
POST	/experiments	declare an experiment and its traffic split
GET	/experiments/{name}	arm results, significance test, plain-English verdict
POST	/retrain	refit the batch models, returning the reason either way

GET /recommend/{user_id} accepts k, strategy (auto, hybrid, online, collaborative, content, popularity) and experiment. With auto, the experiment layer assigns the arm.

Configuration

Every value in .env.example is the default the code already uses, so an empty .env runs against the local stack unchanged. The ones worth knowing:

variable	default	what it controls
ONLINE_LEARNING_RATE	0.05	how fast the model moves toward recent behaviour; higher adapts sooner and forgets sooner
DIVERSITY_PENALTY	0.3	how steeply a repeated category is discounted when ranking
RETRAIN_INTERACTION_THRESHOLD	500	new interactions that justify a batch refit
RETRAIN_CTR_DROP_TOLERANCE	0.2	relative fall in click-through that counts as degradation
FEATURE_STORE_URL	http://localhost:8001	optional; falls back to locally computed features
CANARY_URL	http://localhost:8002	optional; falls back to serving the baseline

Project layout

simulation/     users with hidden preferences, and the catalog they browse
models/         collaborative, content-based, online learner, hybrid blend
feedback/       turns an interaction into a model update; decides when to refit
serving/        ranker, explainer, FastAPI application
evaluation/     offline ranking metrics, online outcomes, the experiment layer
integrations/   feature store and canary deployment clients
store/          metadata store: interactions, served lists, snapshots, arms
events/         Kafka schemas and producer
examples/       the demo that produces the evidence above
scripts/        registry registration, browser verification
frontend/       the Next.js interface

Part of a ML platform

This is the sixth service in a platform built one project at a time. Each is independent and each is used by the ones after it:

1. ML Feature Store — features shared between training and serving
2. Pipeline Lineage Tracker — where data came from and what it went through
3. ML Canary Deploy — a model registry, and a slice of traffic to the new version first
4. Real-Time Anomaly Detection — anomalies caught in flight
5. Causal Inference Engine — whether an intervention actually caused the outcome

What this engine actually calls:

From the feature store, over HTTP: POST /registry/features to declare the four user features it owns, and GET /features/online to read them at serving time. When the store cannot answer, features are computed locally from interaction history and the response says "feature_source": "local" rather than pretending otherwise.

From canary deploy, over HTTP: GET /deployments/{name} to read the traffic split, applied per request locally so a recommendation does not pay a network round trip for a decision that is one comparison against a random number. Model registration uses that service's own registry through scripts/register_in_registry.py.

Both integrations are over HTTP rather than by importing the other repositories, because all three define top-level config, store and db modules that would shadow each other in a single interpreter. Both degrade rather than fail: a recommender that stops serving because a feature store is briefly unreachable is worse than one that serves from local features and says so.

License

MIT