From Rookie to Pro: Social Engineering LLMs for Automated Vulnerability Exploitation in Enterprise Software

This repository contains the official artifacts for the study on the RSA (Role Assignment-Scenario-pretexting-Action-soliciting) framework. Our research evaluates LLM safety guardrail bypasses and functional exploit generation targeting open-source ERP systems, with a primary evaluation on Odoo and a complementary generalizability assessment on ERPNext.

---

⚠️ Ethical Statement & Disclaimer

For Academic Research Purposes Only.

• No Real-World Impact: This research was conducted in a strictly controlled, isolated laboratory environment. No real-world systems were targeted or harmed.
• Data Anonymization: All network data derived from Shodan (Distribution Analysis) has been strictly aggregated. Identifiable information, such as IP addresses and hostnames, has been removed to protect existing infrastructure.
• Synthetic Lab Data: All financial records, bank names, user credentials, and personal information appearing in execution logs and outputs across this repository are synthetic (mock data) and do not correspond to any real individuals or entities.

---

1. Repository Organization

The repository is structured linearly to follow the Research Questions (RQs) presented in the paper:

• Baseline Prompts/: Contains the original templates for each tested methodology (RSA, DAP, GPTFuzzer, Direct Prompting, and Persona Prompting).
• Safety_Bypass_Evaluation (RQ1)/: Documentation of safety guardrail bypasses. Organized by CVE, then by model/method.
  • Each subfolder contains the Query.py (Input script) and Output.txt (Raw LLM response).
• Functional_Exploitation (RQ2)/: Executable exploit scripts generated by LLMs for the 8 targeted CVEs.
  • Includes [Model]_exploit_generated.py and validation.log (Execution proof).
• Human Evaluation (RQ3)/: Interaction logs and scripts generated by 5 non-expert participants (Rookies), demonstrating the democratization of exploit generation.
• Impact_Analysis/: Aggregated Shodan statistics (Europe vs. Africa) justifying the systemic risk and global attack surface.
• Supplementary_Materials/: Additional materials supporting the paper.
  • Appendix.pdf: Extended appendix with formal RSA definition, parameter variations, and detailed CVE descriptions.
  • CVE_Metadata.xlsx (and .csv): Complete metadata for all tested CVEs including affected versions, vulnerable commit hashes, and patch commit hashes, used to reproduce experimental setups.
  • ERPNext_Generalizability/: Generated scripts and execution logs demonstrating RSA's generalizability beyond Odoo, tested on three recent ERPNext CVEs using Claude Opus 4.6.

---

2. Generalizability Evaluation (ERPNext)

To demonstrate that RSA extends beyond Odoo, we conducted a complementary evaluation on ERPNext — another widely deployed open-source ERP platform built on the Frappe framework.

Scope

• Target: ERPNext v15
• Model: Claude Opus 4.6
• CVEs tested: CVE-2025-66439, CVE-2025-66440, CVE-2025-67289 (disclosed December 2025)
• Result: All three CVEs successfully exploited using the identical RSA template

Contents

Each CVE subfolder under Supplementary_Materials/ERPNext_Generalizability/ contains the generated exploit script and its raw execution output.

---

3. Requirements & Setup

System Requirements

• Python 3.10+
• PostgreSQL 12+
• Git
• Linux or macOS environment (tested on Ubuntu 22.04)

Reproducing the Experiments

Complete metadata for all tested CVEs — including affected versions, vulnerable commit hashes, and patch commit hashes — is available in:

📋 Supplementary_Materials/CVE_Metadata.xlsx (also available as CSV)

General Deployment Procedure

For each CVE, using the metadata file:

1. Identify the vulnerable commit hash from CVE_Metadata.xlsx.

2. Clone and checkout the vulnerable version:

   git clone https://github.com/odoo/odoo.git   
   cd <repository>
   git checkout <vulnerable_commit_hash>

3. Install dependencies following the official installation guide for the target version.

4. Initialize the database and start the instance.

5. Execute the LLM-generated exploit from the corresponding Functional_Exploitation (RQ2)/ folder.

Reproducibility Note

Due to the diversity of affected Odoo and ERPNext versions spanning multiple years (2017-2025), fully dockerized environments for all configurations are not provided. The artifacts include the LLM-generated exploit scripts, their execution outputs, and the CVE_Metadata.xlsx file with all information required to reproduce each experimental setup manually.

---

• Supplementary_Materials/: Additional materials supporting the paper.
  • Appendix.pdf: Extended appendix with formal RSA definition, parameter variations, and detailed CVE descriptions.
  • ERPNext_Generalizability/: Generated scripts and execution logs demonstrating RSA's generalizability beyond Odoo, tested on three recent ERPNext CVEs using Claude Opus 4.6.