JEV DataOps

Turn a dataset into a training experiment you can inspect.

Upload your data, use JEV to decide what to keep, fine-tune your model, and compare the results—all in one workbench. Every run saves the screening decisions, dataset splits, and evaluation reports so you can see what happened.

Try the demo · Website · Get started locally · User guide

Open the interactive demo → No installation, API key, or GPU needed.

1. Click Use example to load the bundled dataset.
2. Click Start workflow with the default settings.
3. Open Runs & insights to inspect the decisions, compare before/after results, and download the reports.

The example has 84 synthetic records: 80 kept, 2 for review, and 2 rejected. The kept records become 56 training, 12 validation, and 12 test records.

About the demo: Files stay in your browser tab. It uses local screening rules and a small byte-bigram statistical model; it does not call JEV or train an LLM. Try your own UTF-8 JSONL or CSV file up to 2 MiB / 1,000 rows, and download results before refreshing. Demo details

JEV screens your data. Your chosen model learns from it. This project calls JEV for judgments; it does not fine-tune JEV itself.

You need Python 3.10+, Git, and an internet connection for installation. Start with the local demo; no provider key or model download is needed.

git clone https://github.com/RenaGao/jev-dataops.git
cd jev-dataops

python3 -m venv .venv
source .venv/bin/activate
pip install -e .

jev-dataops serve

Open localhost:8000 and leave the terminal running. Click Use example → Start workflow to complete your first run. Keep both the screening engine and training backend set to Demo for this first check.

Upload a UTF-8 JSONL or CSV file, then check the preview. JSONL has one JSON object per line:

{"group_id":"chat-001","instruction":"How do I change notification preferences?","output":"Open Settings, select Notifications, and save your preferences."}
{"group_id":"chat-002","instruction":"How do I recover my password?","output":"Select Forgot password on the sign-in page and follow the verification steps."}

These two rows illustrate the format. Training requires at least 6 independent groups after screening.

Use one format per record. Give records from the same conversation or document the same group_id or conversation_id so they stay in one training, validation, or test split. The default split is approximately 70% / 15% / 15% by group.

More examples, CSV formatting, and input limits →

Set a provider key in the terminal running your backend:

export OPENROUTER_API_KEY='your-openrouter-key'
jev-dataops serve

Or set TYPESAFE_API_KEY for a direct TypeSafe connection. Restart the service if it was already running, then select JEV · OpenRouter or JEV · TypeSafe under Screening engine.

For your first live run, use a small dataset and turn off Auto-train and evaluate. Inspect the output before training:

• Keep: candidate training data that passed the configured rules and gates.
• Review: records that need closer inspection; excluded from automatic training.
• Reject: records excluded with a recorded reason.

Choose General, Finance, or Code as a starting rubric. These are screening criteria you can adapt, not certified domain-quality checks. Keys belong in the backend environment; the workbench's connection token is a separate access-control setting.

Live JEV screening sends the selected content fields to your chosen provider. Redact sensitive information before submission. Provider setup, thresholds, request budgets, and retries →

Install training dependencies in the same virtual environment and select a model:

pip install -e '.[train]'
export JEV_BASE_MODEL='HuggingFaceTB/SmolLM2-135M'
jev-dataops serve

Restart a running service, choose Hugging Face · LoRA in the workbench, and enable Auto-train and evaluate. Keep your provider key configured if you also want JEV screening. Screening and training are separate settings.

The workbench trains on kept records, keeps related records in the same split, and measures loss before and after training on held-out data. Training runs on your backend machine; the public website does not supply compute. Model files may download on the first run, and memory requirements depend on the model.

The defaults are a small trial: 1 epoch, at most 20 steps, batch size 4, sequence length 256. Adjust them before running a full experiment. Saved LoRA adapters need the original base model for inference.

Training configuration · SFT / GRPO / PPO bundles · Multi-GPU setup

Start with Runs & insights. Open Why records were held back to inspect review and rejected records by reason. In live JEV mode, you can also inspect each record’s dimension-level judgments. Download files from Artifacts & reports:

A negative delta_loss means loss decreased on that test set. Check the data split and actual training steps before interpreting the number. Retention rate and JEV confidence are not measured accuracy; lower loss does not establish better business outcomes. Use an independent task benchmark and human review to assess quality. Demo metrics use bytes; LLM metrics use tokenizer tokens, so they are not directly comparable.

The local workbench saves runs under .jev-dataops/ in your working directory. File layout and metric definitions →

Run the complete local demo from the repository root:

jev-dataops run --input examples/dialogues.jsonl --output runs/my-first-demo

Use a new output directory for each training experiment. For screening only, add --trainer none; for live JEV, configure your key and choose --provider openrouter or --provider typesafe.

CLI and HTTP examples · Python API

More troubleshooting and recovery options →

Licensed under MIT. Model weights and provider APIs have their own terms. This is an independent community project, unaffiliated with TypeSafe or OpenRouter. Security reporting