LinkedIn-Bench

Rank LinkedIn posts like an evaluator, not a fortune teller.

LinkedIn-Bench is a local benchmark runner for LLM systems that rank which LinkedIn posts are most likely to outperform an author's own historical baseline. It is built for workflow evaluation: prompts, context shaping, media handling, ranking strategy, cost, and latency.

It is not a scraper and it is not a statistical forecasting package. The core question is narrower: given only pre-post information, can a system order candidate posts better than a baseline?

At a glance

• Local-first benchmark flow for LinkedIn exports or generic CSVs
• Public smoke path that runs without API keys
• Prompt-inspection command for leakage checks
• Cost, latency, and parse-failure tracking alongside ranking metrics
• Optional media-manifest path for multimodal evaluation

Install

uv sync
cp .env.example .env

The default config uses the mock provider so the smoke path runs without API credentials. For real LLM runs, set LLM_PROVIDER, LLM_API_KEY, optional LLM_BASE_URL, and the model env vars in .env.

Public smoke path

The fastest way to verify the repo is to use the included fixture dataset. This exercises ingestion, labeling, batching, a mock benchmark run, and report generation without touching private exports or paid APIs.

uv run lb ingest-csv data/fixtures/sample_posts.csv --dataset-id fixture_public
uv run lb attach-author-metadata --dataset-id fixture_public --path data/fixtures/sample_author_metadata.csv
uv run lb attach-media --dataset-id fixture_public --manifest data/fixtures/sample_media_manifest.csv
uv run lb validate --dataset-id fixture_public
uv run lb build-labels --dataset-id fixture_public
uv run lb build-split --dataset-id fixture_public --split-id temporal_70_30
uv run lb build-batches --dataset-id fixture_public --split-id temporal_70_30
uv run lb run --config configs/v0_1.yaml
uv run lb report --run-id latest
uv run lb leaderboard --dataset-id fixture_public

Private LinkedIn export flow

Place files:

data/raw/linkedin_export.xlsx
data/raw/author_metadata.csv
data/raw/media_manifest.csv

Then run:

uv run lb ingest-linkedin-export data/raw/linkedin_export.xlsx --dataset-id creator_export --author-id creator_one
uv run lb attach-author-metadata --dataset-id creator_export --path data/raw/author_metadata.csv
uv run lb attach-media --dataset-id creator_export --manifest data/raw/media_manifest.csv
uv run lb validate --dataset-id creator_export
uv run lb build-labels --dataset-id creator_export
uv run lb build-split --dataset-id creator_export --split-id temporal_70_30
uv run lb build-batches --dataset-id creator_export --split-id temporal_70_30
uv run lb prepare-media --dataset-id creator_export
uv run lb summarize-media --dataset-id creator_export
uv run lb run-system --dataset-id creator_export --system llm_metadata_direct_v1 --split-id temporal_70_30 --max-batches 3
uv run lb report --run-id latest

If you want uv run lb run --config ... to cover your private dataset by default, add your dataset id to configs/v0_1.yaml or create a copy of that config for your own runs.

Prompt inspection

uv run lb inspect-prompt --system llm_metadata_direct_v1 --batch-id <batch_id>

Candidate prompts deliberately exclude ground-truth outcome fields. Historical aggregate stats are allowed only through explicit context fields that would have existed before the post was published.

Outputs

• Processed datasets land in data/processed/<dataset_id>/
• Run artifacts land in artifacts/runs/<run_id>/
• Markdown and HTML reports land in artifacts/reports/
• LLM caching uses artifacts/cache/llm_cache.sqlite

Reports include dataset health, leaderboard metrics, ablation deltas, failure cases, win cases, cost, and latency.

Data safety

• Do not commit private LinkedIn exports.
• Do not scrape LinkedIn.
• Media stays opt-in through data/raw/media_manifest.csv; source_url is stored but not fetched automatically.
• Treat the fixture dataset as a plumbing check, not a public performance claim.
• Read SECURITY.md before publishing your own benchmark results.

Tests

uv run pytest