bot-or-not-mcgill

Bilingual competition hardening for the McGill Bot or Not challenge.

The current codebase focuses on a practical phase-3 workflow:

• load the shipped dataset.posts&users.*.json schema and optional dataset.bots.*.txt labels
• auto-select the strongest supported English or French model by cross-validated competition utility
• optimize official-first competition models with grouped official validation and separate practice evaluation
• evaluate a saved model bundle on labeled practice data
• generate [team_name].detections.[lang].txt submission files for unlabeled datasets
• orchestrate English-only, French-only, or bilingual optimize/practice/final runs into timestamped run directories

Data Layout

The repository now separates practice, official, and self-collected data:

• data/practice/english/
• data/practice/french/
• data/official/english/
• data/official/french/
• data/self_collected/english/
• data/self_collected/french/

The shipped competition practice files live under data/practice/.... The categorized competition training files now live under data/official/.... The self_collected directories are reserved for additional training data you add later.

The repository now includes a concrete English auxiliary-source pattern for pan19-en:

• normalize the official PAN 2019 English training archive into the competition-style schema with .venv/bin/python -m bot_or_not.normalize_pan19
• reference the normalized output from data/self_collected/english/aux-manifest.pan19-en.json

The checked-in twibot20-en directory remains only as an unlabeled raw-sample reference because the public TwiBot-20 sample omits labels.

Scope

Phase 3 remains intentionally narrow:

• English and French are both supported as separate training targets
• the system still uses separate per-language models and thresholds rather than multilingual joint training
• the train CLI now defaults to --model-type auto, which compares the built-in logistic, hybridtext, histgb, extratrees, and ensemble tracks and persists only the winning artifact
• model selection now reserves a fixed stratified holdout split on the primary labeled dataset; cross-validation and threshold selection run on the remaining primary accounts, then the winning setup is reported once on that holdout before the saved model is refit on the full primary dataset
• the recommended competition path is bot_or_not.compete optimize, which discovers all labeled official shards per language, uses grouped official validation when the shipped dataset ids are available, and evaluates the selected model separately on the labeled practice dataset
• the bilingual wrapper is orchestration-only: it runs English and French side by side and aggregates artifacts in one run directory

Data Schema

The loaders expect the shipped competition JSON schema regardless of folder. The official-first optimize workflow treats data/official/... as primary training data and data/practice/... as external labeled evaluation.

Expected dataset shape:

• top-level JSON object with keys: id, lang, metadata, posts, users
• posts entries with: text, created_at, id, author_id, lang
• users entries with: id, tweet_count, z_score, username, name, description, location

Expected label shape:

• one user id per line, matching the practice dataset.bots.*.txt files

Features

Dense account features include:

• tweet_count and z_score
• profile-field lengths and empty-field flags
• mean and standard deviation of post character and token lengths
• per-post URL, hashtag, and mention rates
• duplicate-post ratio and unique-post ratio
• inter-post timing statistics
• posting-hour entropy
• richer tree-only behavioral features such as username digit patterns, vocabulary diversity, punctuation and casing ratios, retweet share, weekday entropy, and burst-vs-gap timing rates

Model tracks currently include:

• logistic: char-level TF-IDF over posts plus base dense account features
• hybridtext: account-level word and char TF-IDF timeline embeddings projected with SVD, combined with extended dense account features in a calibrated linear classifier
• histgb: calibrated histogram gradient boosting over base dense account features
• extratrees: calibrated ExtraTrees over the extended dense feature set
• ensemble: a probability blend of histgb and extratrees with cross-validated component weights

Setup

Python 3.11 is the supported baseline for this repository. Python 3.10 and older are out of scope for the checked-in lockfile and CI workflow.

Create a local virtual environment and install the pinned dependencies:

python3.11 -m venv .venv
.venv/bin/pip install -r requirements.lock

On Windows with the py launcher:

py -3.11 -m venv .venv
.\.venv\Scripts\python -m pip install -r requirements.lock

Maintaining The Lockfile

pyproject.toml is the dependency source of truth. requirements.lock is compiled from it with pip-tools.

Install the maintainer tooling:

python3.11 -m pip install "pip-tools>=7.5,<8"

Regenerate the lockfile from pyproject.toml plus the dev extra:

python3.11 -m piptools compile pyproject.toml --extra dev --strip-extras --output-file requirements.lock

CLI

The low-level train and evaluate commands are still available for manual experiments. For competition-facing runs, prefer bot_or_not.compete optimize.

Recommended official-first English optimize run:

.venv/bin/python -m bot_or_not.compete optimize \
  --lang en \
  --data-root data \
  --run-dir runs

Recommended bilingual optimize run:

.venv/bin/python -m bot_or_not.compete optimize \
  --lang both \
  --data-root data \
  --run-dir runs

The optimize workflow:

• discovers and combines all labeled official shards for the requested language
• keeps the labeled practice dataset out of training and uses it only for external evaluation
• uses grouped official validation for the shipped English ids 1, 3, 5 and French ids 2, 4, 6
• falls back to the existing stratified-random selection path when grouped official validation is unavailable

Optional English PAN sweep:

.venv/bin/python -m bot_or_not.compete optimize \
  --lang en \
  --data-root data \
  --run-dir runs \
  --pan-sweep

When --pan-sweep is enabled, the workflow evaluates PAN 2019 as low-weight auxiliary text-only augmentation at weights 0.1, 0.25, and 0.5, always with --aux-common-features-only semantics. The selected model is promoted only if it improves practice utility without increasing practice false positives and does not weaken the official validation utility. If the normalized PAN JSON is missing, the command fails with a clear error instead of silently skipping the sweep.

Train an English model bundle from the practice dataset:

.venv/bin/python -m bot_or_not.train \
  --dataset-json data/practice/english/dataset.posts\&users.30.json \
  --labels data/practice/english/dataset.bots.30.txt \
  --lang en \
  --model-out models/english-best.joblib

Train a French model bundle from the practice dataset:

.venv/bin/python -m bot_or_not.train \
  --dataset-json data/practice/french/dataset.posts\&users.31.json \
  --labels data/practice/french/dataset.bots.31.txt \
  --lang fr \
  --model-out models/french-best.joblib

Both default to --model-type auto. To force a single candidate, add --model-type logistic, --model-type hybridtext, --model-type histgb, --model-type extratrees, or --model-type ensemble.

To augment English training with one or more normalized self-collected sources, add --aux-manifest:

.venv/bin/python -m bot_or_not.train \
  --dataset-json data/practice/english/dataset.posts\&users.30.json \
  --labels data/practice/english/dataset.bots.30.txt \
  --lang en \
  --model-out models/english-best.joblib \
  --aux-manifest data/self_collected/english/aux-manifest.pan19-en.json

Auxiliary sources are used only for fit-time augmentation. Cross-validation, threshold selection, and the reported holdout metric remain anchored to the primary labeled dataset.

For source-aware mixed-source training, the English train and single-language practice flows now also support:

• --aux-weight <float> to retain only a deterministic fraction of each auxiliary source during fit-time augmentation
• --aux-common-features-only to disable profile- and timing-derived dense features during mixed-source training

Example conservative mixed-source run:

.venv/bin/python -m bot_or_not.train \
  --dataset-json data/practice/english/dataset.posts\&users.30.json \
  --labels data/practice/english/dataset.bots.30.txt \
  --lang en \
  --model-out models/english-best.joblib \
  --aux-manifest data/self_collected/english/aux-manifest.pan19-en.json \
  --aux-weight 0.5 \
  --aux-common-features-only

Evaluate a saved model on labeled data:

.venv/bin/python -m bot_or_not.evaluate \
  --dataset-json data/practice/english/dataset.posts\&users.30.json \
  --labels data/practice/english/dataset.bots.30.txt \
  --model models/english-best.joblib

.venv/bin/python -m bot_or_not.evaluate \
  --dataset-json data/practice/french/dataset.posts\&users.31.json \
  --labels data/practice/french/dataset.bots.31.txt \
  --model models/french-best.joblib

When you evaluate a saved model on the same labeled practice dataset used during training, the output now marks that result as a primary_dataset_refit score and includes the stored selected_holdout summary for a less optimistic reference point.

Generate a submission file:

.venv/bin/python -m bot_or_not.predict \
  --dataset-json data/practice/english/dataset.posts\&users.30.json \
  --model models/english-best.joblib \
  --team-name my-team \
  --output my-team.detections.en.txt

If --output is omitted, prediction defaults to [team_name].detections.[lang].txt.

.venv/bin/python -m bot_or_not.predict \
  --dataset-json data/practice/french/dataset.posts\&users.31.json \
  --model models/french-best.joblib \
  --team-name my-team \
  --output my-team.detections.fr.txt

Localhost GUI

Run the stdlib localhost GUI with the live CLI runner:

.venv/bin/python -m bot_or_not.gui serve \
  --host 127.0.0.1 \
  --port 8000 \
  --workspace .

Open http://127.0.0.1:8000 in a browser. The page exposes Train, Optimize, Practice, and Final sections, shows the exact CLI command it will run, streams logs, and serves only the artifacts created by the submitted job. Live mode is the default, but every submission requires the explicit "Confirm this expensive local run" checkbox before execution.

For safe UI work that must not execute the real model, start the mock runner instead:

.venv/bin/python -m bot_or_not.gui serve \
  --runner mock \
  --host 127.0.0.1 \
  --port 8000 \
  --workspace .

Orchestration

Create a full official-first optimize run directory with official selection reports, a saved best model, and a separate practice evaluation report:

.venv/bin/python -m bot_or_not.compete optimize \
  --lang en \
  --data-root data \
  --run-dir runs

Run the same optimize workflow for both languages in one top-level command:

.venv/bin/python -m bot_or_not.compete optimize \
  --lang both \
  --data-root data \
  --run-dir runs

Create a full single-language practice run directory with auto-selection, reports, and an optional detections file:

.venv/bin/python -m bot_or_not.compete practice \
  --dataset-json data/practice/english/dataset.posts\&users.30.json \
  --labels data/practice/english/dataset.bots.30.txt \
  --lang en \
  --run-dir runs \
  --team-name my-team

The same command works for French by changing --dataset-json, --labels, and --lang fr.

Single-language English practice runs also accept --aux-manifest, --aux-weight, and --aux-common-features-only with the same semantics as train. Bilingual practice intentionally leaves auxiliary manifests and auxiliary policy flags out of scope for this first integration.

Run a bilingual practice workflow in one top-level command:

.venv/bin/python -m bot_or_not.compete practice \
  --lang both \
  --dataset-json-en data/practice/english/dataset.posts\&users.30.json \
  --labels-en data/practice/english/dataset.bots.30.txt \
  --dataset-json-fr data/practice/french/dataset.posts\&users.31.json \
  --labels-fr data/practice/french/dataset.bots.31.txt \
  --team-name my-team \
  --run-dir runs

Competition-day bilingual runbook:

.venv/bin/python -m bot_or_not.compete optimize \
  --lang both \
  --data-root data \
  --run-dir runs

Then score the evaluation-day root datasets with the selected English and French model bundles:

.venv/bin/python -m bot_or_not.compete final \
  --lang both \
  --model-en runs/<en-optimize-run>/best-model.joblib \
  --model-fr runs/<fr-optimize-run>/best-model.joblib \
  --team-name my-team \
  --run-dir runs

When you run that final command from the repository root, the CLI auto-discovers dataset.posts&users.7.json for English and dataset.posts&users.8.json for French if you leave the dataset flags out. To override discovery, pass --dataset-json, --dataset-json-en, and/or --dataset-json-fr explicitly.

Single-language final workflow:

.venv/bin/python -m bot_or_not.compete final \
  --lang en \
  --model runs/<en-optimize-run>/best-model.joblib \
  --team-name my-team \
  --run-dir runs

Windows Batch Helper

On Windows, the repository now includes run_windows_workflows.bat, which wraps the main commands behind one entry point:

.\run_windows_workflows.bat help
.\run_windows_workflows.bat train-en-aux
.\run_windows_workflows.bat optimize-both
.\run_windows_workflows.bat final-both my-team

The batch file uses the checked-in practice datasets, writes models to models\, writes run folders to runs\, defaults the evaluation-day datasets to dataset.posts&users.7.json and dataset.posts&users.8.json, and keeps the final model paths configurable through EN_FINAL_MODEL and FR_FINAL_MODEL near the top of the file so you can point them at the chosen optimize-run artifacts.

By default, the CLI now emits human-readable status logs and progress bars to stderr while keeping the machine-readable JSON result on stdout. To silence that extra output for automation, set BOT_OR_NOT_VERBOSE=0 before invoking the commands.

Bilingual final workflow:

.venv/bin/python -m bot_or_not.compete final \
  --lang both \
  --model-en runs/<en-optimize-run>/best-model.joblib \
  --model-fr runs/<fr-optimize-run>/best-model.joblib \
  --team-name my-team \
  --run-dir runs

That bilingual final run writes the submission files to:

• runs/<timestamp>-both-<team>/en/[team_name].detections.en.txt
• runs/<timestamp>-both-<team>/fr/[team_name].detections.fr.txt

Attach those two text files directly to the competition submission email.

Single-language run directories are created under runs/<timestamp>-en-<slug>/ or runs/<timestamp>-fr-<slug>/ and contain:

• best-model.joblib
• practice-evaluation-report.json for optimize, containing the external labeled practice score for the selected model
• selection-report.json
• evaluation-report.json for practice, with both the stored primary holdout summary and a separate refit-on-practice score
• prediction-report.json for final
• [team_name].detections.en.txt when prediction output is requested

Bilingual run directories are created under runs/<timestamp>-both-<slug>/ and contain:

• summary.json
• en/ with English artifacts
• fr/ with French artifacts

Testing

Run the test suite with:

.venv/bin/python -m pytest

The default suite is non-live: it exercises the CLI contracts and the localhost GUI with mocks so pytest and CI never execute real training, evaluation, or prediction work.

To run the opt-in live smoke suite against the shipped English practice dataset, set BOT_OR_NOT_ENABLE_LIVE_SMOKE=1:

BOT_OR_NOT_ENABLE_LIVE_SMOKE=1 .venv/bin/python -m pytest tests/test_live_smoke.py

CI runs the full non-live suite on Python 3.11.