Skip to content

v0.71.5 — Ingest / data / prompt / drift

Choose a tag to compare

View all tags
@MakazhanAlpamys MakazhanAlpamys released this 02 Jun 09:46
· 49 commits to main since this release
v0.71.5
1f63393

Production-loop ergonomics: alternative model hubs for data commands, tokenizer-aware prompt pruning, webhooks across the trace toolchain, percentile curriculum bucketing, and smarter pre-flight advise.

What's New

  • Alternative model hubs for data — soup data push --hub modelscope|modelers uploads a local JSONL to ModelScope / Modelers, and soup data forge --hub … --teacher owner/name pre-fetches the teacher from that hub (warns loudly if --teacher isn't a routable repo id instead of silently dropping --hub). (#157)
  • Tokenizer-aware prompt pruning — soup prune-prompt --tokenizer <id-or-path> finds the shared token prefix and decodes only the remainder, so BPE multi-byte sequences never get truncated mid-token the way char-slicing can. (#205)
  • Webhooks across the trace toolchain — --slack-url / --discord-url now work on soup ingest, soup prune-prompt, soup ab, and soup data active-sample, sharing the same SSRF-hardened validator as soup drift-alarm. The A/B harness only pings when the sequential test actually decides (reject_h0 / accept_h0), not while it's still collecting. (#207)
  • Curriculum by difficulty percentile — dynamic curriculum can bucket by loss / perplexity percentile (rolling 512-step window) instead of length round-robin; length keeps the round-robin default. (#149)
  • Benchmark-score curves in replay — ExperimentTracker.get_metric_series transparently falls back to the eval_results table when a metric has no per-step rows, so soup runs replay can plot an mmlu / gsm8k curve the same way it plots loss. (#164)
  • Smarter pre-flight advise — soup advise nudges its confidence (never its decision) using your prior recorded verdicts for the same project + choice once ≥3 precedents exist. (#163)

Install / Upgrade

pip install --upgrade soup-cli        # 0.71.5

Security

  • New shared soup_cli/utils/webhooks.py centralises the v0.63.0 drift-alarm SSRF policy and applies it to four more commands with full parity: scheme allowlist (http/https), loopback-only HTTP, RFC1918 / link-local / reserved / multicast IPs rejected via ipaddress, 0.0.0.0 rejected, null-byte + control-char rejection, 4096-char URL cap. post_webhook never raises, so a flaky webhook can't fail the command.
  • soup data push --hub stages the upload under a cwd-contained temp dir so the existing upload_repo containment check holds.

Known Limitations

  • #204 (live SaaS trace pull) deferred — the live Langfuse / LangSmith / Helicone / OpenPipe / OTel pull adapters need paid vendor accounts to validate honestly and can't be exercised offline, so they stay open with an infra-blocked label. soup ingest continues to parse JSONL exports you pull from your dashboard / vendor API; only the in-CLI network pull is deferred.

Full history: CHANGELOG.md

Assets 2
Loading