LLM Advice-Framing Research Harness

This repository now contains a complete proof-of-concept research harness for studying how consumer chatbot LLMs respond to interpersonal/advice-seeking prompts when both the explicit question framing and the tone of the scenario narrative are varied. It includes:

• the active prompt-and-sample structure used for the experiment
• Bash scripts for generating queries, submitting them to multiple LLM providers, and collecting outputs
• second-pass and aggregate comparison prompts and scripts
• archived generated artifacts in ./results/ for independent analysis and review
• a manuscript draft and related research documentation

Archived generated artifacts are stored in ./results/. They are intentionally separated from the active harness directories so they do not impede using this repository to run your own tests.

Recommended Reading Order

1. Readers interested in the results

Read in this order:

1. SCIENTIFIC_PAPER_DRAFT.md
2. output/analysis_all_CHATGPT.md, output/analysis_all_GEMINI.md, output/analysis_all_CLAUDE.md
3. output/analysis_001_*.md, output/analysis_003_*.md, output/analysis_008_*.md
4. ANALYSIS_CODE_CHEATSHEET.md
5. results/

2. Researchers (lay or professional) wishing to extend or replicate the research

Read in this order:

1. REPOSITORY_PROCESS_DOCUMENTATION.md
2. this README.md
3. prefixes/
4. samples/
5. build_queries.sh
6. submit_queries.sh
7. combine_series.sh
8. build_result_queries.sh
9. submit_result_queries_prompt.md
10. submit_result_queries.sh
11. compare_all_results_prompt.md
12. compare_all_results.sh
13. env.example
14. results/

3. Those wishing to audit the work done

Read in this order:

1. REPOSITORY_PROCESS_DOCUMENTATION.md
2. this README.md
3. samples/ and prefixes/
4. queries/
5. output/combined_*.md
6. output/combined_query_*.md
7. ANALYSIS_CODE_CHEATSHEET.md
8. output/analysis_*.md
9. output/analysis_all_*.md
10. results/
11. SCIENTIFIC_PAPER_DRAFT.md

AI Usage

The code in this repository has been significantly written or altered by an AI tool with human supervision. The instructions to the AI agents were detailed, step-by-step pseudocode with very specific limitations. Whenever possible, the code is also explicitly and extensively commented so that it may be audited to determine that it does what it says it does. Again, while the code in this repository works for me and I am using it, it should be considered a proof-of-concept for others to refine, fix, and build upon.

Files

convert_tsv_to_html.sh

Used for converting the TSV output from the RSS feed into something useable.

Converts a TSV export into two root-level files:

• converted_tsv.html
• converted_tsv.md

The script expects the TSV path as its first argument. It assumes the TSV has a header row and three tab-separated columns:

1. title
2. url
3. contents

The HTML output preserves the contents field as HTML and removes the trailing Reddit export footer starting at <!-- SC_ON -->. The Markdown output strips most HTML tags and converts common HTML entities into plain text.

Example:

./convert_tsv_to_html.sh 1_reference/selected-items.tsv

build_queries.sh

Combines every Markdown file in prefixes/ with every Markdown file in samples/ and writes the results to queries/.

• Input: prefixes/*.md, samples/*.md
• Output: queries/<sample_id>_<prefix_name>_<sample_suffix>.md
• Format: prefix text, then a blank line, then sample text

Example:

./build_queries.sh

If prefixes/negative-others.md and samples/008_unsympathetic.md exist, the script writes queries/008_negative-others_unsympathetic.md.

submit_queries.sh

Submits every Markdown file in queries/ to one or more LLM APIs as a fresh single-turn user message and writes the returned assistant text to output/.

• Input: queries/*.md
• Output: output/<query_name>_CHATGPT.md, output/<query_name>_GEMINI.md, and/or output/<query_name>_CLAUDE.md
• Configuration: automatically loads .env from the repo root if present
• Provider selection:
  • --chatgpt submits only to OpenAI
  • --gemini submits only to Gemini
  • --claude submits only to Claude
  • --all submits to all three
  • provider switches can be combined, for example --chatgpt --claude
  • no switch defaults to --chatgpt
• Logging and errors:
  • stdout is written to output/results.log
  • stderr is written to output/results.err
  • existing output files are skipped instead of being regenerated
  • selected providers are submitted in parallel for each query
  • request failures are logged and the script continues with the remaining query/provider combinations
  • the script exits non-zero at the end if any requests failed
• Query handling constraints:
  • every request is sent as a single fresh user turn, to mirror typing a prompt into a blank web chat
  • OpenAI requests set store: false
  • Gemini requests set store: false
  • Claude requests use the stateless Messages API with a single user message
  • OpenAI requests include a unique safety_identifier per query
  • Claude requests include a unique metadata.user_id per query
  • Gemini's generateContent endpoint does not expose a documented per-request user-ID field, so isolation there is enforced by sending each query as a standalone single-turn request with no prior history and store: false
• API behavior:
  • OpenAI uses POST /v1/responses
  • Gemini uses POST /v1beta/models/<model>:generateContent
  • Claude uses POST /v1/messages
  • each query is sent as a standalone prompt with no prior conversation state

Requirements:

• for --chatgpt: OPENAI_API_KEY
• for --gemini: GEMINI_API_KEY
• for --claude: CLAUDE_API_KEY or ANTHROPIC_API_KEY
• curl must be installed
• jq must be installed

Optional:

• OPENAI_MODEL overrides the default model, which is gpt-5
• GEMINI_MODEL overrides the default model, which is gemini-3.6-flash
• CLAUDE_MODEL overrides the default model, which is claude-sonnet-4-6
• ANTHROPIC_VERSION overrides the Claude API version header, which defaults to 2023-06-01

Example:

./submit_queries.sh --chatgpt --claude

Example .env:

OPENAI_API_KEY=your_key_here
OPENAI_MODEL=gpt-5
GEMINI_API_KEY=your_key_here
GEMINI_MODEL=gemini-3.6-flash
CLAUDE_API_KEY=your_key_here
CLAUDE_MODEL=claude-sonnet-4-6
ANTHROPIC_VERSION=2023-06-01

combine_series.sh

Recursively combines related Markdown files from a directory tree into one output file per numeric ID for each matching directory.

• Input: files named like base_<id>.md, sym_<id>.md, not_<id>.md, and result_*_<id>.md
• Directory selection: any directory containing either base_<id>.md or result_*_<id>.md files is processed
• Output: combined_<id>.md in each matching input directory by default, or in a mirrored directory tree under an optional output directory
• Format:
  • each source file starts with a Markdown header line containing its filename
  • file contents are concatenated with a blank line between sections

The script orders inputs like this:

1. base_<id>.md
2. sym_<id>.md
3. not_<id>.md
4. result_neutral_<id>.md, result_neutral_sym_<id>.md, result_neutral_not_<id>.md
5. result_positive_<id>.md, result_positive_sym_<id>.md, result_positive_not_<id>.md
6. result_negative_<id>.md, result_negative_sym_<id>.md, result_negative_not_<id>.md

For additional result families such as result_negative_me_<id>.md, the script keeps the same order within that family: no suffix first, then _sym, then _not.

Example:

./combine_series.sh output

That command processes output/ and every nested subdirectory beneath it.

To write the combined files somewhere else:

./combine_series.sh output combined_output

If output/001/ contains matching files, the combined output is written to combined_output/001/.

build_result_queries.sh

Transforms each output/combined_<id>.md file into a compact query-oriented file for downstream comparative analysis.

• Input: output/combined_<id>.md
• Output: output/combined_query_<id>.md
• Skip behavior:
  • ignores files such as combined_query_<id>.md
  • skips rebuilding if the target combined_query_<id>.md already exists
• Transformations:
  • omits the first four sections from the combined file:
    • <id>_control.md
    • <id>_neutral.md
    • <id>_sympathetic.md
    • <id>_unsympathetic.md
  • keeps headers for the remaining sections
  • rewrites those headers to a compact initials-based form
  • removes the first body line immediately under each retained header when that line is the repeated filename/provider marker

Header format:

• source header: 008_negative-me_control_CHATGPT.md
• rewritten header: 008_n-m_c_CHATGPT

That means:

• the prompt family is reduced to initials split on hyphens
• the sample variant is reduced to its first letter:
  • control -> c
  • neutral -> n
  • sympathetic -> s
  • unsympathetic -> u
• the provider suffix is preserved

Example:

./build_result_queries.sh

submit_result_queries.sh

Submits every output/combined_query_<id>.md file to one or more LLM APIs for comparative analysis and writes the returned text to output/.

• Input: output/combined_query_*.md
• Output: output/analysis_<id>_CHATGPT.md, output/analysis_<id>_GEMINI.md, and/or output/analysis_<id>_CLAUDE.md
• Configuration: automatically loads .env from the repo root if present
• Prompt source: submit_result_queries_prompt.md
• Provider selection:
  • --chatgpt submits only to OpenAI
  • --gemini submits only to Gemini
  • --claude submits only to Claude
  • --all submits to all three
  • provider switches can be combined, for example --chatgpt --claude
  • no switch defaults to --chatgpt
• Logging and errors:
  • stdout is written to output/analysis.log
  • stderr is written to output/analysis.err
  • existing output files are skipped instead of being regenerated
  • selected providers are submitted in parallel for each combined query file
  • request failures are logged and the script continues with the remaining query/provider combinations
  • the script exits non-zero at the end if any requests failed
• Query handling constraints:
  • every request is sent as a single fresh user turn, to mirror typing a prompt into a blank web chat
  • OpenAI requests set store: false
  • Gemini requests set store: false
  • Claude requests use the stateless Messages API with a single user message
  • OpenAI requests include a unique safety_identifier per query
  • Claude requests include a unique metadata.user_id per query
  • Gemini's generateContent endpoint does not expose a documented per-request user-ID field, so isolation there is enforced by sending each query as a standalone single-turn request with no prior history and store: false
• Prompt behavior:
  • the script prepends a fixed analysis prompt to the inlined contents of each combined_query_<id>.md file and submits that as one user message
  • the response is written verbatim to the corresponding analysis_<id>_<PROVIDER>.md file

Example:## AI Usage

The code in this repository has been significantly written or altered by an AI tool with human supervision. The instructions to the AI agents were detailed, step-by-step pseudocode with very specific limitations. Whenever possible, the code is also explicitly and extensively commented so that it may be audited to determine that it does what it says it does. Again, while the code in this repository works for me and I am using it, it should be considered a proof-of-concept for others to refine, fix, and build upon.

./submit_result_queries.sh --all

compare_all_results.sh

Submits all output/combined_query_<id>.md files together to one or more LLM APIs for one aggregate comparison pass per provider.

• Input: every output/combined_query_*.md
• Output: output/analysis_all_CHATGPT.md, output/analysis_all_GEMINI.md, and/or output/analysis_all_CLAUDE.md
• Configuration: automatically loads .env from the repo root if present
• Prompt source: compare_all_results_prompt.md
• Provider selection:
  • --chatgpt submits only to OpenAI
  • --gemini submits only to Gemini
  • --claude submits only to Claude
  • --all submits to all three
  • provider switches can be combined, for example --chatgpt --claude
  • no switch defaults to --chatgpt
• Logging and errors:
  • stdout is written to output/compare_all.log
  • stderr is written to output/compare_all.err
  • existing output files are skipped instead of being regenerated
  • selected providers are submitted in parallel
  • request failures are logged and the script continues with the remaining providers
  • the script exits non-zero at the end if any requests failed
• Query handling constraints:
  • every request is sent as a single fresh user turn, to mirror typing into a blank web chat
  • OpenAI requests set store: false
  • Gemini requests set store: false
  • Claude requests use the stateless Messages API with a single user message
  • OpenAI requests include a unique safety_identifier per query
  • Claude requests include a unique metadata.user_id per query
  • Gemini's generateContent endpoint does not expose a documented per-request user-ID field, so isolation there is enforced by sending each aggregate query as a standalone single-turn request with no prior history and store: false
• Prompt behavior:
  • the script prepends a fixed comparison prompt
  • it then inlines every combined_query_*.md file in one aggregate submission, separated by filename headers
  • the response is written verbatim to the corresponding analysis_all_<PROVIDER>.md file

Example:

./compare_all_results.sh --all

Size Notes

The current compare_all_results.sh implementation submits all output/combined_query_*.md files as one inline text prompt per provider rather than uploading them as files.

As of July 30, 2026, the current aggregate input size in this repo is:

• output/combined_query_001.md: 90,737 bytes, 14,839 words
• output/combined_query_003.md: 64,084 bytes, 10,487 words
• output/combined_query_008.md: 40,863 bytes, 6,382 words
• total: 195,684 bytes, 31,708 words

A rough estimate puts that aggregate prompt around 50k-65k input tokens once prompt text and separators are included.

That is currently well below the documented limits of the default large-context models used here:

• Gemini 3.6 Flash: 1,048,576 input tokens
• Claude Sonnet 4.6: up to 1M tokens

Practical caveats:

• the current corpus is unlikely to hit a hard context limit on Gemini or Claude
• output quality may still degrade before the hard limit is reached as more files are added
• transient provider capacity or timeout issues may appear before context exhaustion
• if the aggregate corpus grows several times larger, compare_all_results.sh is the script most likely to need chunking or batching first

snippets.md

Stores short commands and one-liners used in this project.

Right now it includes the SQLite command that exports selected RSSGuard items into a TSV file.

Notes

• converted_tsv.html and converted_tsv.md are generated files and are ignored by Git.
• queries/ is populated by build_queries.sh.
• output/ is populated by submit_queries.sh.
• combined review files can be generated from any directory with combine_series.sh.
• results/ stores archived generated artifacts for independent analysis and review. Those artifacts are intentionally not part of the active samples/, queries/, or output/ workflow so they do not interfere with using the harness for fresh testing.

AI Usage

The code in this repository has been significantly written or altered by an AI tool with human supervision. The instructions to the AI agents were detailed, step-by-step pseudocode with very specific limitations. Whenever possible, the code is also explicitly and extensively commented so that it may be audited to determine that it does what it says it does. Again, while the code in this repository works for me and I am using it, it should be considered a proof-of-concept for others to refine, fix, and build upon.

You are encouraged to fork and refine or rebuild this program or create something better that has the same functionality.

This is what most people would call "vibe coded".