cs376FinalProject

Final Project for CS 375/376

Reframed Objective

This repository is an evaluation harness for prompting smaller coding LLMs to solve constrained programming tasks.

The cryptography-themed datasets are used as controlled coding benchmarks, not as an end in themselves. The core question is:

How reliably can smaller LLMs generate executable, correct Python under realistic prompt constraints, and what recurring failure patterns show up in artifacts?

Why This Setup Works for Small-LLM Testing

The project isolates three dimensions that matter when evaluating smaller models:

1. Code generation quality
   • Model must emit runnable Python with the expected function contract.
2. Algorithmic correctness
   • Code must produce exact expected outputs on held-out inputs.
3. Prompt compliance and repair behavior
   • Model is asked to provide structured outputs and can receive repair prompts after failures.

Task Families (Level Structure)

1. Level 1 - Caesar and ROT style tasks
   • Lightweight algorithmic transformation with strict output matching.
2. Level 2 - Mixed classical cipher tasks
   • Vigenere, transposition, and single-byte XOR style tasks requiring key handling.
3. Level 3 - Encoding detection and decoding
   • Base64, hex, and binary decoding workflows.
4. Level 4 - Controlled hash tasks
   • Hash identification and dictionary-constrained recovery cases.

Even though the content domain is cryptography, each level functions as a coding-task difficulty ladder for small models.

What Is Logged as Artifacts

The harness writes per-sample and aggregate artifacts designed for error analysis:

• Per-sample outputs (status, retries, execution output, model-declared plaintext/key)
• Aggregate metrics (execution success, exact accuracy, runtime, key recovery)
• Failure reports and gate reports for pass/fail criteria

Primary artifact locations:

• generated/*.jsonl for datasets
• results/*_outputs.jsonl for row-level behavior
• results/*_metrics.json for summary metrics
• results/gate_report.json for gate checks

Observed Small-LLM Struggles in Recorded Artifacts

These are recurring patterns already captured in this repository's artifacts.

1. Prompt-following drift between code and declared answer

   • Execution can be correct while model-declared plaintext is wrong or generic.
   • Evidence: Level 1 outputs include many rows where exec_output matches expected but model_predicted_plaintext is empty or unrelated.

2. Strong execution reliability, weaker exact correctness on harder tasks

   • Example from Level 2 test metrics: execution_success_rate 0.99 while exec_exact_match_accuracy 0.755.
   • Interpretation: smaller models often generate runnable code that still implements the wrong algorithmic path.

3. Runtime failures cluster around code-structure mistakes

   • IndentationError, UnboundLocalError, and syntax issues appear in failed Level 2 rows.
   • Examples include unindent mismatch and unparenthesized generator expressions in failed samples.

4. Key inference is a major bottleneck

   • Level 2 key_recovery_accuracy is low (0.025 in recorded metrics), indicating difficulty recovering/using keys even when code runs.

5. Overconfident fallback text and template leakage

   • Some rows contain stock outputs like hello-world style plaintext, or malformed placeholder leakage in predicted fields.

6. Repair prompting helps some rows but does not guarantee correctness

   • Non-zero retry_count rows show recovery from certain failures, but some still end with incorrect outputs or execution errors.

How To Use This Repository as a General Small-LLM Prompting Benchmark

Run commands from repository root.

1. Generate data

   • python src/generate_level2.py

2. Validate generator correctness

   • python src/check_level2_solutions.py

3. Run a baseline model

   • python src/run_baseline.py --file generated/level_02_test.jsonl --backend ollama --model qwen2.5-coder:7b --timeout 180 --exec-timeout 10 --code-regen-interval 50

4. Check summary metrics

   • results/level_02_test_baseline_ollama_qwen2.5-coder_7b_metrics.json

5. Inspect row-level failures and retries

   • results/level_02_test_baseline_ollama_qwen2.5-coder_7b_outputs.jsonl

6. Run gate checks

   • python src/check_level2_gate.py --metrics results/level_02_test_baseline_ollama_qwen2.5-coder_7b_metrics.json

Suggested Evaluation Questions

1. How does accuracy change as prompt strictness increases?
2. Which failure types decrease with repair prompting?
3. Does stronger prompt formatting improve key recovery more than plaintext recovery?
4. Where do smaller models fail: syntax, control flow, or algorithm selection?
5. How much execution success is misleading without exact-match checks?

Reproducibility Notes

• Use fixed seeds in generation scripts.
• Keep datasets and metrics in version control.
• Compare models with the same timeout and execution constraints.
• Treat outputs.jsonl as the primary source for qualitative error taxonomy.

Ethical Policy

Allowed Use

• Educational and research use in this course project
• Synthetic datasets generated by project scripts
• Public challenge datasets where usage rights explicitly allow analysis
• Hash-recovery tasks only for known test corpora with explicit permission

Prohibited Use

• Any unauthorized decryption or account access attempts
• Processing private or stolen data
• Building or sharing tooling intended for abuse
• Attempting to evade institutional, legal, or platform safeguards

Safe Development Rules

1. Keep all experiments in isolated local/test environments.
2. Use only approved datasets and document provenance.
3. Log every run with dataset id, timestamp, and model/prompt version.
4. Do not include personal or sensitive data in prompts, logs, or results.
5. If a task appears dual-use beyond course scope, stop and require instructor approval.

Compliance Note

This repository is intended to demonstrate defensive and educational cryptography understanding. Any use outside approved legal/ethical boundaries is disallowed.