Unified Vulnerability Dataset

This repository contains a unified, machine-learning-ready dataset of software vulnerabilities. The primary objective is to combine multiple disparate vulnerability datasets into a single, standardized format suitable for training vulnerability detection models (such as Large Language Models or Graph Neural Networks).

Overview

Software vulnerability datasets typically come in vastly different formats (e.g., raw source files, YAML metadata, JSONL commit diffs). This project harmonizes these formats into a single JSONL schema focused on function-level vulnerability classification.

Unified Schema

Every record in the combined dataset follows the DiverseVul schema and is stored in JSON Lines (.jsonl) format.

What is JSONL format? JSON Lines (JSONL) is a text format where each individual line is a complete, valid JSON object. This format is specifically chosen for large datasets because:

• Memory Efficient (Streamable): You can read, process, or write one record at a time without loading the entire multi-gigabyte dataset into RAM.
• Appendable: New records can simply be appended to the end of the file without needing to parse the existing JSON structure.
• Structure: Unlike standard JSON, there are no commas between objects and no enclosing array brackets [] at the start or end of the file.

An example of a single line representing one function in the dataset:

{
  "project": "Repository URL or project name",
  "commit_id": "The Git commit hash (if applicable)",
  "target": 1, 
  "func": "The raw source code of the function",
  "idx": "A unique identifier for the function",
  "cwe": "Common Weakness Enumeration ID (if available)"
}

• target: 1 indicates the function is vulnerable (bad), 0 indicates the function is patched/benign (good).

---

Dataset Sources

This dataset was constructed by aggregating and parsing data from the following three distinct sources:

1. DiverseVul (C/C++)

DiverseVul is a large-scale dataset of C/C++ vulnerabilities mined from open-source GitHub commits. It naturally follows the target JSONL schema.

• Source Format: .jsonl files (valid.jsonl, sample_1000.jsonl, etc.)
• Contents: Real-world C/C++ functions before and after security patches.

2. SARD - Software Assurance Reference Dataset (C/C++)

SARD contains synthetic and academic test cases for software vulnerabilities.

• Source Format: Raw .c and .cpp source files containing multiple function variants.
• Extraction: We developed a custom parser (process_sard.py) that uses brace-matching heuristics to extract individual functions from SARD files. Functions ending in _bad are labeled as target: 1, while functions containing good (e.g., goodG2B) are labeled as target: 0.

3. Go Vulnerability Database (Go)

The official Go vulndb repository contains highly curated vulnerability advisories.

• Source Format: .yaml metadata files detailing the repository, commit hash, and vulnerable symbol names.
• Extraction: We developed an extraction pipeline (extract_go_source.py) utilizing PyDriller to read the YAML metadata, dynamically clone the corresponding GitHub repositories, traverse to the fix commit, and extract the raw Go source code of the function both before (target: 1) and after (target: 0) the patch.

---

Processing Scripts

The following scripts are used to generate the final dataset:

1. process_sard.py: Parses the raw C/C++ SARD files, extracts functions, applies heuristic labeling, and outputs sard_processed.jsonl.
2. extract_go_source.py: Parses the Go vulndb YAML files, utilizes PyDriller to clone repos, and extracts vulnerable Go functions into go_samples.jsonl.
3. combine_datasets.py: A memory-efficient concatenation script that merges the DiverseVul .jsonl files and the newly processed sard_processed.jsonl into the final combined_dataset.jsonl.

Data Samples

Here is a visual demonstration of the structured data inside the unified vulnerability dataset. Each entry provides the source code mapped to a target classification (1 for vulnerable, 0 for patched).

1. Go Vulnerability Database (Go)

This example was dynamically extracted using PyDriller from the gin-gonic/gin repository (CVE-2020-36567).

Vulnerable Function (target: 1)

• ID: GO-2020-0001_LoggerWithConfig_bad
• Commit: a71af9c144f9579f6dbe945341c1df37aaf09c0d

func LoggerWithConfig(conf LoggerConfig) HandlerFunc {
	// ... (setup code)
	return func(c *Context) {
		start := time.Now()
		path := c.Request.URL.Path
		raw := c.Request.URL.RawQuery

		c.Next()

		if _, ok := skip[path]; !ok {
			param := LogFormatterParams{
				Request: c.Request,
				isTerm:  isTerm,
				Keys:    c.Keys,
			}
			
			// POTENTIAL VULNERABILITY:
			// Unsanitized input from 'path' and 'raw' could allow log injection.
			if raw != "" {
				path = path + "?" + raw
			}
			param.Path = path

			fmt.Fprint(out, formatter(param))
		}
	}
}

2. SARD - Software Assurance Reference Dataset (C/C++)

This example was extracted using our custom Python brace-matcher from the raw synthetic test cases.

Vulnerable Function (target: 1)

• ID: sard_1
• File: CWE114_Process_Control__w32_char_connect_socket_01.c
• CWE: CWE114

void CWE114_Process_Control__w32_char_connect_socket_01_bad()
{
    char * data;
    char dataBuffer[100] = "";
    data = dataBuffer;
    
    // ... (Socket setup and network reading code)
    recvResult = recv(connectSocket, (char *)(data + dataLen), sizeof(char) * (100 - dataLen - 1), 0);
    data[dataLen + recvResult / sizeof(char)] = '\0';
    // ...

    {
        HMODULE hModule;
        /* POTENTIAL FLAW: If the path to the library is not specified, an attacker may be able to
         * replace his own file with the intended library */
        hModule = LoadLibraryA(data);
        if (hModule != NULL)
        {
            FreeLibrary(hModule);
            printLine("Library loaded and freed successfully");
        }
        else
        {
            printLine("Unable to load library");
        }
    }
}

---

Usage

The final combined dataset can be found at combined_dataset.jsonl. Because it is in standard JSON Lines format, it can easily be loaded into Python using pandas or standard json libraries:

import json

with open('combined_dataset.jsonl', 'r') as f:
    for line in f:
        record = json.loads(line)
        print(f"Project: {record['project']}, Vulnerable: {record['target'] == 1}")