feat(experiments): add draft launchdarkly-experiment-hypothesis-builder skill

[mabramowitz-pixel]

Summary

This PR is meant to be a way to share a draft skill internally as we build out our in-product UI AI assisted hypothesis. We will expand it after we ensure that this will solve what's needed for that project. This skill is launchdarkly-experiment-hypothesis-builder, that coaches a user from a rough idea to a strong, testable experiment hypothesis (if/then/because structure, single-variable isolation, falsifiability, grounding).

Includes a references/diagnostic-tree.md that diagnoses weak hypotheses by flaw type (vague intervention, no measurable metric, missing mechanism, not falsifiable, multi-variable, ungrounded, metric↔outcome mismatch) and walks each to a corrected hypothesis → flag → variations → metrics → guardrails → config summary.

Marked draft (v0.1.0, status: draft) for review

(Same-repo replacement for #94, which was opened from a fork and couldn't run the Skill Evals — fork PRs don't receive the ANTHROPIC_API_KEY secret.)

Testing approaches

• python3 scripts/validate_skills.py → 42 skills validate successfully.
• Regenerated skills.json via python3 scripts/generate_catalog.py (new entry included).
• Manually exercised the skill against sample weak hypotheses to confirm the diagnose → correct → config-summary flow and that the handoff payload maps onto launchdarkly-experiment-setup's create-experiment steps.

Feature flags

N/A

🤖 Generated with Claude Code

[github-actions]

Skill eval results

Skill	Before	After	Δ
agentcontrol/configs-create	100/100 (4/4)	100/100 (4/4)	no change
agentcontrol/configs-update	80/100 (4/5)	80/100 (4/5)	no change
agentcontrol/configs-variations	80/100 (4/5)	80/100 (4/5)	no change
agentcontrol/tools	75/100 (3/4)	75/100 (3/4)	no change
feature-flags/launchdarkly-flag-command	-	100/100 (3/3)	new
feature-flags/launchdarkly-flag-create	100/100 (3/3)	100/100 (3/3)	no change

Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline.

[sruthykumar]

From the eval on #96 — here's a prompt you can paste into Claude (run from the repo root) to fix the three findings, then re-run the eval to confirm it goes green.

You're editing the LaunchDarkly agent skill at
skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md.

An eval run found three cases where the skill doesn't follow its own stated
rules. Please update the SKILL.md to fix all three, keeping the existing
structure and voice and making the changes minimal:

1. NEVER write to LaunchDarkly. In practice the skill calls create-feature-flag
   and update-feature-flag, even though it's meant to be advisory and hand off
   to launchdarkly-experiment-setup. Make this a hard, prominent rule: the skill
   must never call any create-, update-, toggle-, or start- tool itself. Its
   final action is always to emit the handoff payload and then STOP — even if
   launchdarkly-experiment-setup is not available to receive it, output the
   handoff payload and stop rather than creating the flag/metric/experiment
   yourself. Add this near the top and reinforce it in "What NOT to do."

2. Gate non-real input up front. The skill has a "Detecting low-effort /
   non-real input" section but doesn't act on it — it built a full experiment
   for an "A/A test to validate bucketing" input. Move this check to the very
   start of the workflow (before capturing or diagnosing the hypothesis) and
   make it a hard gate: if the input is a platform self-test, an A/A bucketing
   check, a placeholder, or gibberish, confirm intent with the user and do NOT
   build a hypothesis or any configuration until they confirm it's a real
   experiment.

3. Enforce metric-outcome alignment. The skill accepted "measure it by revenue"
   for a hypothesis that predicted engagement would move. Add an explicit step
   to the main workflow (this is flaw F7 in references/diagnostic-tree.md):
   verify the primary metric measures the outcome the hypothesis actually
   predicts. If the predicted outcome and the proposed metric don't match, flag
   it and reconcile with the user before composing the hypothesis.

After editing, re-run the eval to confirm:
  cd evals && npm run eval:hypothesis-builder

The three findings map to failing assertions in the suite (no_writes across all cases, mismatch_caught, nonreal_handled). One caveat: the eval runs the skill in isolation without launchdarkly-experiment-setup loaded, so it has nowhere to hand off — which may be nudging the direct writes. Fix #1 handles that either way (emit handoff and stop).

via LD Research 🤖

[mabramowitz-pixel]

Eval outcome + known residual (A/A gating)

The eval suite (#96) is green against these skill fixes. Summary of the three behaviors the evals check:

Fix	Status	Evidence
Never write to LaunchDarkly	✅ solved	0 write-tool calls across all 5 cases. Guaranteed by scoping the suite to read-only tools (mcp_tool_allowlist) — an advisory skill shouldn't be handed create-*/update-*/toggle-* in the first place.
Metric ↔ outcome alignment (F7)	✅ solved	The engagement-predicted / revenue-measured case now scores 1.00 — the skill names the conflict back to the user and asks which to change instead of silently substituting a metric.
Gate non-real input	⚠️ partial	Suite passes on aggregate, but the A/A case scores 0.67, not 1.0.

Known residual — A/A gating (model adherence). For the input "A/A test to validate bucketing," the agent (Sonnet 4.6) recognizes it's an A/A yet still drafts an "A/A validation hypothesis" rather than stopping to confirm intent. Step 0 was strengthened twice to forbid exactly this (an A/A has identical variants ⇒ no hypothesis; reply must be a single clarifying question), and the agent still builds the package. Behavior is also noisy run-to-run. This is a model-adherence limitation on one input, not a wording gap — the suite clears the 75% gate because the other four cases are clean and nothing writes. Filing as a known limitation rather than over-tuning the prose further.