Research report · Benchmark postmortem · Experiments · Evidence · Reproducibility · Русская версия

---

SplitShift documents the research and engineering behind a service-ad splitting system developed for the Avito track at Purple Hack 2026, where the submitted solution placed 2nd.

The product task was to decide whether services mentioned inside a broad renovation ad were offered independently and should become separate draft listings. The research question became more important than the final classifier:

Why did strong results on the initial benchmark fail to transfer after the evaluation distribution became more practical?

The repository reconstructs 43 experiment scripts, preserves the submitted hybrid pipeline, and adds a reproducible postmortem focused on synthetic shortcuts, near-duplicates, subset selection, and distribution shift.

My contribution

My work focused on the technical research track documented here:

• dataset analysis and failure-mode discovery;
• classical ML, rules, LLM, RAG, self-consistency, and cascade comparisons;
• error analysis and evaluation-validity review;
• integration and optimization of the submitted hybrid pipeline;
• reconstruction of the experiment record into a transparent benchmark postmortem.

The archive preserves historical branches as evidence and does not claim that I was the sole author of every archived script.

What was built

The submitted system combined:

1. a deterministic regex detector for secondary microcategories;
2. TF-IDF text features and handcrafted structural features;
3. a CatBoost split-intent classifier;
4. optional LLM draft generation after the local decision.

The clean splitshift package provides a deterministic linear baseline and reproducible evaluation tools. The competition-era CatBoost implementation and fitted artifact are preserved separately for auditability.

The benchmark postmortem

Directly reproduced findings

Finding	Reproduced evidence	Interpretation
The submitted artifact nearly memorizes the development corpus	F1 0.9968 in-sample	Invalid as a generalization estimate; the threshold was selected on the same rows
Clean 5-fold random CV is much lower	F1 0.559, PR-AUC 0.603	The development task is substantially harder than the training score suggests
Near-duplicate-aware GroupKFold is similar	F1 0.552	Near-duplicates are real, but do not alone explain the later practical-stage failure
Masking explicit shortcut cues hurts	F1 0.505	Words around “separate/partial repair” matter, but no single cue explains everything
The included initial test is unusually close to training	median nearest similarity 0.867	64/159 test ads have a training neighbor with similarity ≥ 0.90

Historical evidence

The experiment archive reports rules, classical ML, RuBERT, SetFit, augmentation, several LLMs, prompt variants, RAG, self-consistency, and ML–LLM cascades. Some selected-subset runs reached 0.91–0.92, but those values are not valid deployment estimates.

During the competition, the evaluation reportedly moved to a more practical hidden distribution and scores fell sharply. The later test labels, scorer export, and exact stage-wise score history are absent, so this repository treats that drop as historical context, not a reproduced number.

Experiment landscape

The figure deliberately places impressive but invalid numbers beside reproduced estimates. Its purpose is methodological: evaluation setting matters more than the largest number in a notebook.

The complete table is in experiments/results/experiment_manifest.csv, and the corresponding scripts are in experiments/archive/.

Quick start

CPU-side checks

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -e ".[dev]"

pytest
splitshift inspect-data
splitshift verify

Reproduce the clean benchmark table

splitshift benchmark \
  --output experiments/results/reproduced_benchmarks.csv

Reproduce train–initial-test overlap

splitshift analyze-overlap \
  --output experiments/results/train_initial_test_overlap.json

Train the clean baseline

splitshift train --output artifacts/clean_baseline.joblib

The historical CatBoost pipeline and web demo require optional dependencies:

pip install -e ".[competition]"

Evidence in the repository

Evidence type	Location
Clean reproducible research package	src/splitshift/
CPU tests	tests/
Development corpus and initial test	data/
43 historical experiment scripts	experiments/archive/
Reproduced tables	experiments/results/
Original submitted implementation	archive/original_submission/
Historical fitted artifact	artifacts/competition_split_detector.pkl
Claim-by-claim audit	docs/evidence.md

Reproducibility status

Level	Status
Dataset schemas, inventory, overlap analysis, clean CV, cue ablation, serialization, tests	Included and reproducible
Historical experiment claims	Partially supported by scripts and notes
Submitted artifact's in-sample behavior	Reproduced; invalid for generalization
Later practical hidden-test score	Not self-contained
Official second-place evidence	Historical claim; leaderboard artifact not included

Read Data and reproducibility before interpreting any benchmark number.

Repository map

.
├── src/splitshift/               # Clean data, model, analysis, evaluation, CLI
├── tests/                        # CPU-only tests
├── configs/                      # Clean baseline and historical pipeline configs
├── data/                         # Development corpus, category dictionary, initial test
├── experiments/
│   ├── archive/                  # 43 competition-era experiment scripts
│   └── results/                  # Reproduced tables and evidence manifests
├── docs/                         # Report, postmortem, architecture, evidence, reproducibility
├── archive/original_submission/  # Submitted pipeline, API, and original README
├── artifacts/                    # Historical fitted CatBoost artifact
└── pyproject.toml

Documentation

• Research report — complete investigation and conclusions
• Benchmark postmortem — synthetic shortcuts and distribution shift
• Experiment log — historical and reproduced experiments by validity
• Architecture — submitted and clean pipelines
• Code walkthrough — module-by-module guide
• Evidence map — which claims are directly inspectable
• Data and reproducibility — included and missing artifacts
• Competition task — original product setting

Scope and limitations

• The practical hidden evaluation set is unavailable.
• Historical API experiments do not all include raw responses or exact scorer logs.
• Several high historical scores were measured on selected or leaky subsets.
• The included initial test has high lexical overlap with development data.
• Grouped CV and cue ablation are diagnostics, not a complete causal explanation of the shift.
• The project is a benchmark postmortem, not a claim of production validation across all Avito traffic.

Citation

@misc{mariskin_splitshift_2026,
  title  = {SplitShift: A Benchmark Postmortem for Service-Intent Detection},
  author = {Mariskin, Stavr},
  year   = {2026},
  note   = {Purple Hack 2026 Avito track, 2nd-place solution}
}

---

_{Research principle: separate what was impressive from what was actually measured.}