Add Simplified Technical English (STE) output_format variant to daily experiments - #51442
Merged
Merged
Conversation
…iments Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot created this pull request from a session on behalf of
pelikhan
August 8, 2026 18:37
View session
Collaborator
|
@copilot experiments should be paired with evals |
Collaborator
|
@copilot merge main and recompile |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
Author
Added an |
Collaborator
|
@copilot resolve the merge conflicts in this pull request |
…ments-ste-formatting # Conflicts: # .github/workflows/ab-testing-advisor.lock.yml # .github/workflows/copilot-agent-analysis.lock.yml # .github/workflows/daily-code-metrics.lock.yml # .github/workflows/daily-compiler-quality.lock.yml # .github/workflows/daily-experiment-report.lock.yml # .github/workflows/daily-issues-report.lock.yml # .github/workflows/deep-report.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
Author
Contributor
Author
Contributor
There was a problem hiding this comment.
Pull request overview
Adds Simplified Technical English variants to daily workflow experiments and related guidance.
Changes:
- Adds and rebalances STE experiment variants across six workflows.
- Adds STE prompts and format-adherence evals.
- Updates experiment documentation and regenerated workflows.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/deep-report.md |
Adds the STE briefing variant. |
.github/workflows/deep-report.lock.yml |
Regenerates the workflow. |
.github/workflows/daily-semgrep-scan.md |
Adds STE security findings. |
.github/workflows/daily-semgrep-scan.lock.yml |
Regenerates the workflow. |
.github/workflows/daily-issues-report.md |
Adds STE report instructions. |
.github/workflows/daily-issues-report.lock.yml |
Regenerates the workflow. |
.github/workflows/daily-experiment-report.md |
Documents format-variant analysis. |
.github/workflows/daily-experiment-report.lock.yml |
Regenerates the workflow. |
.github/workflows/daily-compiler-quality.md |
Adds an STE quality report. |
.github/workflows/daily-compiler-quality.lock.yml |
Regenerates the workflow. |
.github/workflows/daily-code-metrics.md |
Adds an STE metrics report. |
.github/workflows/daily-code-metrics.lock.yml |
Regenerates the workflow. |
.github/workflows/copilot-agent-analysis.md |
Adds an STE analysis summary. |
.github/workflows/copilot-agent-analysis.lock.yml |
Regenerates the workflow. |
.github/workflows/ab-testing-advisor.md |
Adds STE and paired-eval guidance. |
.github/workflows/ab-testing-advisor.lock.yml |
Regenerates the workflow. |
.github/aw/experiments.md |
Documents STE experimentation. |
Review details
Tip
Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
Suppressed comments (6)
.github/workflows/daily-compiler-quality.md:113
- BinEval receives only this static question and the agent output (
actions/setup/js/run_evals.cjs:232-257), not the selected experiment assignment. Because the report does not identify that assignment, the evaluator cannot know whether its style matches the selected variant. Pass the assignment into the eval context or expose it in the evaluated output.
question: Does the report match the writing style expected for the assigned output_format variant (e.g., short active-voice sentences with one fact per sentence when the variant is "ste")?
.github/workflows/daily-code-metrics.md:99
- BinEval receives only this static question and the agent output (
actions/setup/js/run_evals.cjs:232-257), not the selected experiment assignment. Because the report does not identify that assignment, the evaluator cannot know whether its style matches the selected variant. Pass the assignment into the eval context or expose it in the evaluated output.
question: Does the report match the writing style expected for the assigned output_format variant (e.g., short active-voice sentences with one fact per sentence when the variant is "ste")?
.github/workflows/copilot-agent-analysis.md:72
- BinEval receives only this static question and the agent output (
actions/setup/js/run_evals.cjs:232-257), not the selected experiment assignment. Because the discussion does not identify that assignment, the evaluator cannot know whether its style matches the selected variant. Pass the assignment into the eval context or expose it in the evaluated output.
question: Does the discussion summary match the writing style expected for the assigned output_format variant (e.g., short active-voice sentences with one fact per sentence when the variant is "ste")?
.github/workflows/daily-code-metrics.md:261
- This STE template places four independent metric facts on one line and uses
LOCbefore spelling it out, contradicting the rules immediately above. Split the metrics into short sentences and introduce the acronym.
**Key metrics today**: LOC: X,XXX | Quality score: XX/100 | Test ratio: X.XX | Active files (7d): XXX
.github/workflows/copilot-agent-analysis.md:289
- This STE example uses
PRrepeatedly without spelling it out on first use, directly contradicting the preceding acronym rule. Use “pull request” in the generated prose; metric identifiers can remain unchanged.
The agent created [count] PRs in the last 24 hours (`agent_prs_total`). [count] PRs were merged (`agent_prs_merged`). The merge rate is [percentage]%. The average PR duration is [time]. Each PR has an average of [count] human comments. [One short sentence on the 3-day trend, only if the success rate changed by more than 10% — otherwise omit.] [One short sentence naming notable PRs by number, only if failures, closures, or PRs open more than 24 hours exist — otherwise omit.]
.github/workflows/daily-compiler-quality.md:658
- The STE example abbreviates “Average” and combines two facts on one line, contradicting the rules above. Use two complete, short statements.
**Avg score**: 76/100 · **Files meeting threshold**: 2/3
- Files reviewed: 17/17 changed files
- Comments generated: 12
- Review effort level: Balanced
| hypothesis: "H0: no change in discussion engagement or token cost. H1: executive_brief reduces token usage by ≥20% without reducing engagement; annotated_brief improves actionability; ste improves clarity while reducing token usage." | ||
| metric: token_count | ||
| secondary_metrics: [discussion_reactions, discussion_replies, output_char_length, run_duration_ms] | ||
| secondary_metrics: [discussion_reactions, discussion_replies, output_char_length, run_duration_ms, "eval:output_format_goal_met"] |
| hypothesis: "H0: no change in alert creation rate across formats. H1: structured_sections produces ≥15% more alerts successfully created vs. baseline bullet_list; ste improves completeness via clearer, simpler language." | ||
| metric: alert_creation_rate | ||
| secondary_metrics: [run_duration_ms, output_length_chars, findings_reported] | ||
| secondary_metrics: [run_duration_ms, output_length_chars, findings_reported, "eval:output_format_adherence"] |
| hypothesis: "H0: no change in discussion engagement score. H1: inline format produces ≥20% higher reactions+replies by making charts and recommendations immediately visible; ste format improves engagement via clearer, simpler language." | ||
| metric: discussion_engagement_score | ||
| secondary_metrics: [output_length_chars, run_duration_ms] | ||
| secondary_metrics: [output_length_chars, run_duration_ms, "eval:output_format_adherence"] |
| - output_token_count | ||
| - run_duration_ms | ||
| - run_success_rate | ||
| - "eval:output_format_adherence" |
| hypothesis: "H0: no change in discussion engagement rate. H1: executive_summary variant increases discussion reactions+comments by ≥20% due to improved readability; ste variant improves readability further via simplified language." | ||
| metric: discussion_engagement_score | ||
| secondary_metrics: [output_token_count, run_duration_seconds, chart_count] | ||
| secondary_metrics: [output_token_count, run_duration_seconds, chart_count, "eval:output_format_adherence"] |
| - **Minimum detectable effect**: How large a difference matters in practice? | ||
| - **Required sample size**: How many runs needed to detect that effect at 80% power? | ||
|
|
||
| **Every experiment must be paired with an eval.** Whenever success is best judged as a YES/NO question about the output (e.g., "did the report follow the assigned variant's style?"), declare that question under the workflow's `evals:` section and reference it from `metric` or `secondary_metrics` as `eval:<id>`. This lets `gh aw experiments analyze` show observed eval outcomes alongside quantitative metrics — do not propose an experiment without at least one accompanying eval question that checks whether the assigned variant's intended effect actually shows up in the output. |
Comment on lines
+79
to
+82
| Experiments frequently test `output_format` style variants (for example `structured`, `prose`, | ||
| `table`, or `ste` for Simplified Technical English). Treat these like any other variant: compare | ||
| their `metric` and `secondary_metrics` (such as `output_length_chars` or `output_token_count`, which | ||
| serve as verbosity/readability proxies) the same way you would for any other dimension. |
| evals: | ||
| - id: output_format_goal_met | ||
| question: Does the agent output show that the objective for experiment output_format was successfully completed? | ||
| question: Does the report's writing style match the assigned output_format variant (e.g., short active-voice sentences with one fact per sentence when the variant is "ste")? |
| - id: alert_created_or_noop | ||
| question: Was a code scanning alert created for real security findings, or does the agent output confirm no vulnerabilities were found? | ||
| - id: output_format_adherence | ||
| question: Does the findings report match the writing style expected for the assigned semgrep_output_format variant (e.g., short active-voice sentences with one fact per sentence when the variant is "ste")? |
| - id: discussion-created | ||
| question: Was a daily issues discussion created successfully with the report findings and recommendations? | ||
| - id: output_format_adherence | ||
| question: Does the report match the writing style expected for the assigned output_format variant (e.g., short active-voice sentences with one fact per sentence when the variant is "ste")? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new
stevariant to the existingoutput_format/semgrep_output_formatA/B experiments so daily agentic workflows can test Simplified Technical English phrasing (short sentences, active voice, one instruction per sentence) as an alternative to structured/prose/table outputs.Experiment updates
steas a new variant to theoutput_formatexperiment incopilot-agent-analysis,deep-report,daily-issues-report,daily-code-metrics, anddaily-compiler-qualitysteto thesemgrep_output_formatexperiment indaily-semgrep-scanweight:arrays for each experiment to reflect the added variant (e.g.[50, 50]→[34, 33, 33])description/hypothesisfields to mention STE where relevantPrompt instructions
{{#if experiments.output_format == 'ste'}}blocks per workflow with concrete STE rules: sentences ≤20 words, one fact/instruction per sentence, active voice, present tense, plain vocabulary, acronyms spelled out on first useAdvisor & docs
ab-testing-advisor.md'soutput_formatdimension description to mention STE as a candidate variant.github/aw/experiments.mdwith STE guidance under "Dimensions Worth Experimenting On"daily-experiment-report.mdclarifying thatoutput_formatvariants likesteare analyzed the same way as any other dimension (viametric/secondary_metrics)Example variant block (from
daily-semgrep-scan.md):{{#if experiments.semgrep_output_format == 'ste' }} Write your findings report in Simplified Technical English (STE): - Use short sentences. Limit each sentence to 20 words or fewer. - Write one fact or instruction per sentence. - Use active voice and present tense. - Use simple, familiar words. Do not use jargon. - Spell out each acronym on first use. {{/if}}All affected workflows were recompiled to regenerate their
.lock.ymlfiles.