Final project for the Ironhack Data Science bootcamp.
This project started as a pure computer-vision blackjack advisor running on a Raspberry Pi 5 and grew into a full data-science study. The CV layer is no longer the goal: it is the instrument we use to feed a live experiment that tests how a vigilante agent (a card-counting detector) classifies a real human while they play actual hands at a physical table.
The project is built on three independent layers that feed into each other:
Two YOLOv8 detectors observe a real blackjack table from a Raspberry Pi Camera. One reads the cards (13 classes: A, 2–10, J, Q, K), the other reads the chip stack (5 denominations: 1€, 5€, 25€, 50€, 100€). The README walks through how the dataset was captured, annotated, trained on Google Colab and shipped back to the Pi.
A Monte Carlo simulation engine plays millions of hands under configurable rules (decks, penetration, S17/H17, payout 3:2 vs 6:5). It produces the statistical ground truth for the study: how much basic strategy reduces the house edge, how much card counting reverses it, and how those quantities depend on rule variations. It also produces the labelled dataset used to train the detector of Pillar 3.
With Pillars 1 and 2 in place, the system enters session mode
(python main.py --session). Four agents watch the real table at once:
three advisors (basic strategy / naïve counter / expert counter) give live
recommendations on the monitor, and a vigilante — a RandomForest
classifier trained against the Monte Carlo simulator — silently scores how
likely it is that the human at the table is counting cards. Every hand is
logged into a fresh dataset: cards, recommended actions, actions actually
taken, bet pattern, the running probability the vigilante assigns, and the
EXPULSION event if that probability crosses the threshold. The experiment
is the project. The CV stack is what makes the vigilante observable in the
real world, not just in simulation.
blackjack-cv/
├── main.py # Live entry point (auto-mode + --session mode)
├── config.py # All system constants, zones, model paths
├── requirements.txt
├── NOTES.md # Why some columns/keys stay in Spanish (RF model)
│
├── docs/
│ ├── camera.md # IMX708 sensor reference
│ └── study_design.md # Study design — master reference
│
├── src/
│ ├── game/ # Pure game logic (no I/O)
│ │ ├── card.py # Card(rank), value, is_ace
│ │ ├── hand.py # Hand: total, soft, bust, blackjack, pair
│ │ ├── deck.py # Deck for simulation/tests
│ │ └── state.py # GameState, Phase, Action, Outcome
│ │
│ ├── decision/
│ │ ├── strategy.py # Basic strategy: recommend(), full_row()
│ │ ├── live_advisors.py # CardCountTracker + 3 advisors + LiveVigilante
│ │ └── session_manager.py # Orchestrates the 4 advisors, logs each hand
│ │
│ ├── perception/
│ │ ├── camera.py # picamera2 wrapper (Pi 5 + Bookworm)
│ │ ├── camera_ipc.py # picamera2 in a subprocess (see §10, GLib/Qt5 bug)
│ │ ├── detector.py # YOLOv8 card detector (with split-zone filter)
│ │ └── chip_detector.py # YOLOv8 chip detector (5 classes)
│ │
│ ├── simulation/ # Monte Carlo engine + adversarial agents
│ │ ├── rules.py # Rules dataclass (decks, S17/H17, payout, …)
│ │ ├── shoe.py # Shoe with configurable penetration
│ │ ├── engine.py # play_hand() — one full hand
│ │ ├── runner.py # Multi-hand orchestration
│ │ ├── sessions.py # Builds the session-level dataset for the detector
│ │ ├── players.py # BasicPlayer, CardCounter, ParametricCounter (genotype)
│ │ └── adversarial.py # GA counter ↔ RandomForest vigilante coevolution
│ │
│ ├── ui/display.py # Tkinter/OpenCV canvas (auto-mode + session dashboard)
│ ├── analysis/ # logger.py + stats.py (legacy auto-mode log)
│ └── core/motion.py # Motion detector for frame stabilization
│
├── scripts/
│ ├── simulate.py # Keyboard simulator (no camera)
│ ├── capture_dataset.py # Capture card photos
│ ├── capture_chips_dataset.py # Capture chip photos
│ ├── manual_annotate.py # Tkinter annotator: --mode cards|chips
│ ├── auto_annotate.py # Build YOLO dataset for cards
│ ├── auto_annotate_chips.py # Build YOLO dataset for chips
│ ├── preview_bboxes.py # Visual check of labelled bboxes
│ ├── test_camera.py # Show live feed + zones
│ ├── test_detector.py # Live card detector preview
│ ├── test_chip_detector.py # Live chip detector preview
│ ├── run_montecarlo.py # Simulate millions of hands
│ ├── run_experiment_1.py # H1: basic vs intuitive
│ ├── run_experiment_h2.py # H2: counting flips the edge
│ ├── run_experiment_detector.py # Train and benchmark the vigilante
│ ├── run_experiment_robustez.py # Vigilante robustness against disguised counters
│ └── run_experiment_adversarial.py # GA counter ↔ vigilante coevolution
│
├── notebooks/
│ ├── 00_estudio_completo.ipynb # Master notebook: full empirical study
│ ├── experimento_1_eda.ipynb # H1 EDA
│ ├── experimento_h2_conteo_eda.ipynb # H2 EDA
│ ├── detector_contadores.ipynb # Detector design + ROC + feature importance
│ └── detector_robustez.ipynb # Robustness under counter disguise
│
├── tests/ # pytest — 121 tests, all green
│ ├── test_card.py, test_hand.py, test_deck.py, test_state.py
│ ├── test_strategy.py
│ ├── test_detector_split.py
│ ├── test_engine.py, test_shoe.py
│ ├── test_counter.py, test_adversarial.py
│ ├── test_live_advisors.py
│ ├── test_sessions.py
│ └── test_split_flow.py
│
├── models/
│ ├── yolov8l_blackjack.pt # Card detector (13 classes)
│ └── yolov8s_chips.pt # Chip detector (5 classes)
│
└── data/ # Mostly excluded from git
├── raw_images/, raw_chips_images/ # Source photos (not committed)
├── labeled/, labeled_chips/ # YOLO datasets (not committed)
├── sim_*.csv # Monte Carlo outputs (regenerable)
├── sim_sessions.csv # Session dataset → trains the vigilante
├── sessions/<id>/snapshots/ # Real-table session snapshots (committed as evidence)
└── archive_*/ # Historical photo archives (not committed)
Note on language. Code surface, prints and UI are in English. Some DataFrame column names and dict keys remain in Spanish because they are baked into the trained RandomForest in
data/sim_sessions.csv; renaming them would break the model'spredict_probacalls. SeeNOTES.mdfor the full list and rationale.
| Element | Detail |
|---|---|
| Hardware | Raspberry Pi 5, Pi Camera Module 3 (IMX708, 12.3 MP) |
| OS | Raspberry Pi OS Bookworm (64-bit) |
| Language | Python 3.13 |
| Camera access | picamera2 (required on Pi 5 + Bookworm — cv2.VideoCapture does not work with libcamera) |
| Card detector | YOLOv8 large (87 MB, inference on Pi, training on Colab) |
| Chip detector | YOLOv8 small (22 MB) |
| Vision | OpenCV 4.x |
| ML | scikit-learn (RandomForest vigilante), Ultralytics (YOLO) |
| Analysis | NumPy, pandas, matplotlib, seaborn |
| Tests | pytest |
| Virtual env | python3 -m venv venv --system-site-packages (picamera2 lives in the system site-packages on Bookworm) |
Deep dive:
notebooks/modelos_explicados.ipynbPart A is the reference document for this pillar — chronology, scripts, parameter justifications, inference details, lessons learned. The sections below are a high-level summary.
The current dataset is the third iteration. The first two are archived (not deleted) on disk and excluded from git:
| Date | Archive | What it held | Why it was discarded |
|---|---|---|---|
| 2026-05-16 | archive_full_card_bbox_2026-05-16/ (944 MB) |
1 450 photos with bbox over the entire card + a 0.960 mAP50 model | A whole-card bbox was fragile across decks — the small rank in the corner generalises better |
| 2026-05-19 | archive_old_cards_corner_rank_2026-05-19/ (1.5 GB) |
4 660 photos with corner-rank bbox on an old deck + several intermediate models | The user bought new cards with much larger corner ranks → stronger signal → fewer photos needed |
For each detector the flow is the same. Cards first, chips second.
Step 1 — Capture photos.
python scripts/capture_dataset.py # cards (keys A, 2–9, 0=10, J, W=Q, K)
python scripts/capture_chips_dataset.py # chips (keys 1–5 = chip_1/5/25/50/100)Target: ≥100 photos/class for cards (13 × 100 = 1 300), ≥80 photos/class for chips (5 × 80 = 400). Capture on the real green felt (same lighting and reflections as production), with varied angle (0°–40°), rotation, distance and partial occlusion.
Step 2 — Manually annotate.
python scripts/manual_annotate.py # cards
python scripts/manual_annotate.py --mode chips # chipsA Tkinter window with a scroll-zoom canvas. For cards you draw a bbox
only around the rank in the upper-left corner (not the whole card); for
chips you draw a bbox around the whole token. The class is inferred from the
folder name (raw_images/A/A_0042.jpg → class A). YOLO labels are written
as .txt next to each .jpg.
Capture policy (one card per photo): if a frame contains more than one readable rank in the upper-left corner, the photo must be deleted. YOLO treats anything outside a bbox as background; an unlabelled visible rank trains the model to ignore that rank. The annotator only draws one bbox of the folder's class, so multi-card photos actively hurt recall.
Step 3 — Build the dataset.
python scripts/auto_annotate.py # → data/labeled/{images,labels}/{train,val}/
python scripts/auto_annotate_chips.py # → data/labeled_chips/...Stratified 80/20 split per class plus a generated dataset.yaml. Unlabelled
photos are reported but do not block the build — you can train on a subset.
Step 4 — Train on Colab.
cd data && zip -r labeled.zip labeled/ # upload to Google Drive, run YOLOv8 on a Colab GPUThe Pi 5 has no GPU. Training runs on Colab (L4 or T4); only inference runs
on the Pi. Final card model is models/yolov8l_blackjack.pt; chip model is
models/yolov8s_chips.pt.
Step 5 — Verify on the Pi.
python scripts/test_camera.py # frame + zones
python scripts/test_detector.py # card detector live
python scripts/test_chip_detector.py # chip detector liveThe frame is split into three zones (see config.py). The card detector
filters by (x, y) of the bbox centre; the chip detector runs YOLO on the
full frame and filters by the zone afterwards (it must not crop before
inference — see §6.2).
| Zone | Rectangle (normalised) | What lives there |
|---|---|---|
ZONE_DEALER |
x 0–1, y 0–0.50 | dealer cards |
ZONE_PLAYER_RIGHT (Hand 1) |
x 0–0.325, y 0.50–1 | main hand cards |
ZONE_PLAYER_LEFT (Hand 2) |
x 0.325–0.65, y 0.50–1 | split hand cards |
ZONE_BETTING |
x 0.65–1, y 0.55–1 | chips only |
The "Hand 1 / Hand 2" labels are semantic, not geometric: Hand 1 (the main hand, always played first) is the physically right zone next to the chips; Hand 2 (the split hand) is the left one.
The full design document lives in docs/study_design.md. Here is the short
version: the project tests five hypotheses, each backed by a Monte Carlo
experiment whose EDA is in its own notebook.
| ID | Hypothesis | Experiment | Notebook |
|---|---|---|---|
| H1 | A basic-strategy player loses less than an intuitive player, but still has negative EV. | run_experiment_1.py |
experimento_1_eda.ipynb |
| H2 | Basic strategy + Hi-Lo counting + variable bet sizing yields positive EV. | run_experiment_h2.py |
experimento_h2_conteo_eda.ipynb |
| H3 | The counter's edge shrinks with more decks and worsens with hostile rules (6:5 payout, low penetration). | (folded into H2 sweeps) | experimento_h2_conteo_eda.ipynb |
| H4 | A counter is detectable through the bet/count correlation, not through their playing decisions. | run_experiment_detector.py |
detector_contadores.ipynb |
| H5 | In a coevolution between a disguising counter and a learning vigilante, neither wins outright — an equilibrium emerges where the counter keeps positive EV while the detection rate stays appreciable. | run_experiment_adversarial.py |
(results inside 00_estudio_completo.ipynb) |
The vigilante is a RandomForest trained on session-level features
extracted from data/sim_sessions.csv (built by src/simulation/sessions.py).
The features are deliberately behavioural, not action-based: corr_bet_tc
(correlation between bet size and Hi-Lo true count), bet_spread, bet_cv,
bet_lift_high_count, bet_mean, bet_std. The empirical finding behind
H4 is that play decisions barely distinguish a counter from a basic-strategy
player (their action tables overlap on ~98% of hands), so any detector that
looks at how you play will fail. The signal lives entirely in how you
bet.
This is the layer that brings Pillars 1 and 2 onto a real table.
python main.py --session # 100-hand session, default unit = 5€
python main.py --session --session-hands 200
python main.py --session --unit 10 # change the betting unitOn startup the system trains a LiveVigilante against sim_sessions.csv
(~5 s) and instantiates three live advisors that share a single
CardCountTracker (Hi-Lo, updated from the cards observed on the table):
- BasicStrategyAdvisor — pure basic strategy, flat bet.
- NaiveCounterAdvisor — Hi-Lo, conservative bet ramp.
- ExpertCounterAdvisor — the R2 genotype evolved by the GA in
run_experiment_adversarial.py, the "winning recipe" of the study.
At the same time, the vigilante watches the real human play and updates
its probability that the human is a counter after every hand. The on-screen
dashboard makes the vigilante the protagonist: a 380-pixel top strip with
three blocks — verdict (huge P(counter), threshold bar), the six features
the RandomForest looks at (each with its current value and a verdict band),
and a sparkline of P(counter) over the session. The three advisors get a
compact strip below.
If P(counter) crosses the threshold (default 0.90), a red EXPULSED
overlay covers the lower half of the screen — the casino has flagged you.
The hand still completes; the overlay marks the moment for the post-hoc
analysis.
Two hard gates protect the experiment from messy data:
- No bet, no hand. The system refuses to start a hand if the chip detector reports 0€ in the betting zone (with a fallback when the detector is disabled). An orange banner asks the player to place a bet.
- No doubled bet, no split. Splitting is disallowed until the chip
total in the betting zone is at least
1.8 × initial_bet. Same banner pattern.
When a split is activated, the two sub-hands play out automatically by
watching which physical zone gains a new card. Hand 1 finishes first
(STAND is implicit: the next card placed lands in Hand 2's zone), then
Hand 2, then the dealer plays. The hand is logged as a single row with
is_split=True and split_outcome_1/2, split_delta_1/2 fields.
Every session writes into a fresh folder:
| File | Contents |
|---|---|
hands.csv |
One row per hand: cards, dealer upcard, bet, recommended action sequence per advisor, action sequence actually taken, outcome, delta, running count / true count snapshots. |
session.json |
Metadata: timestamp, rules, bankroll, vigilante threshold, expulsion event (if any), unit. |
report.md |
Auto-generated summary: per-advisor adherence, win rate, EV vs simulated baseline, final P(counter). |
snapshots/hand_NNN.jpg |
Annotated camera snapshot at the end of every hand (cards, dealer, totals, outcome, bet). |
The committed snapshots under data/sessions/ are kept as evidence of real
sessions played against the live system. The full CSV/JSON of each session
is regenerated locally on each run and is not committed to keep the repo
light.
The chip detector originally cropped the frame to ZONE_BETTING before
running YOLO. That worked while the zone covered the full frame. After the
zone was reduced to the bottom-right corner the cropped ROI was upscaled to
imgsz=1280, blowing up the chip features ~3× past the training
distribution → zero detections. The fix is to run YOLO on the full frame
and filter detections by the zone afterwards. Never crop before inference
if the model was trained on full frames.
Before session mode existed, python main.py (no flag) ran a
single-advisor mode: a finite-state machine inferred HIT / STAND / DOUBLE by
comparing consecutive stabilised snapshots and wrote each hand to
data/games_log.csv. That mode is still functional, but it is not how the
empirical study is being run today. Use it if you want a simple solo coach
without the vigilante or the 4-advisor dashboard.
# Setup
python3 -m venv venv --system-site-packages
source venv/bin/activate
pip install -r requirements.txt
# Tests
python -m pytest tests/ -v # 121 tests, all green
# Simulator (no camera, keyboard input)
python scripts/simulate.py
# Monte Carlo experiments (no camera)
python scripts/run_montecarlo.py
python scripts/run_experiment_1.py
python scripts/run_experiment_h2.py
python scripts/run_experiment_detector.py
python scripts/run_experiment_adversarial.py
# Live system (camera required)
python main.py # legacy single-advisor auto-mode
python main.py --session # the experiment: 4 advisors + vigilante
python main.py --session --unit 5 # 5€ per chip unit (default)- SURRENDER is visually indistinguishable from STAND and is excluded from the inference MVP.
- Card backs are not modelled. The dealer's hole card is inferred from game state (PLAYER_TURN ↔ dealer shows 1 card).
- Card placement must keep the upper-left rank of every card visible — partial occlusion of that corner makes the card invisible to the detector.
- The chip detector recognises only the five colours it was trained on (white = 1€, red = 5€, green = 25€, dark blue = 50€, black = 100€).
- Six physical decks per shoe (matches
Rules.n_decks=6). For a 100-hand session at unit = 5€, a 250€ bankroll covers the 3σ variance of the expert advisor's bet spread.
For reference: cv2.imshow does not coexist with picamera2 and torch on
Pi 5. The fix is twofold and both halves are needed:
- picamera2 in a subprocess.
src/perception/camera_ipc.pyspawns picamera2 withmultiprocessing.get_context('spawn'); frames travel viashared_memory. The main process never imports picamera2. - Display through Tkinter, not
cv2.imshow. All windows aretk.Tk()+PIL.ImageTk.PhotoImage. Tcl/Tk does not touch Qt5 or GLib.
Any new live script that combines Pi Camera + YOLO + a window must use the same combination.