v1.0.0 – Initial release of experimental results & notebooks

• Data (/data)

  • OWASP Benchmark (spotbugs_dataset.pkl)
    • Source: OWASP Benchmark
    • 2,740+ test cases across 11 vulnerability areas, analyzed with SpotBugs + FindSecBugs
    • Train split (~80% of findings) for preliminary studies & few-shot example generation
    • Test split (~20% of findings) for validation and cross-model comparison
    • See datasets.md for full dataset details

• Preliminary Studies (/data/preliminary_study)

  1. Contextual Information Analysis
     • Impact of SpotBugs report vs. CWE database context on LLM assessments
     • Conducted on the train split
     • README & details
  2. Prompting Techniques Comparison
     • Few-Shot, Chain-of-Thought (CoT), and Self-Consistency (SC) with GPT-3.5 Turbo
     • Conducted on the train split
     • README & details

• Main Research Findings (/data/towards_efficient_complementary_security_analysis)

  • Evaluation of multiple LLM families (Qwen, GPT, Phi, Llama) on:
    • OWASP test split (403 findings)
    • Real-world Mnestix dataset (114 findings)
  • Full write-up & data

• Supporting Code (/src)

  • Few-shot example scripts: few_shot_examples.py
  • Prompt templates: prompt_templates.py

Feel free to download the JSON files and open the notebooks to reproduce our analyses!