Virtual Hardware Security Module (HSM)

A comprehensive, feature-rich virtual Hardware Security Module implementation providing cryptographic key management, digital signatures, multi-user authentication, audit logging, and secure file storage.

Table of Contents

• Overview
• Features
• Architecture
• Quick Start
• Installation
• Usage
  • Library API
  • CLI Usage
  • Enhanced HSM
  • REST API Server
  • Python API
• Docker & Kubernetes
• Key Management
• Digital Signatures
• File Storage
• Security Features
• GitHub Actions Integration
• Building from Source
• Testing
• Documentation
• Known Limitations
• License

Overview

This Virtual HSM provides enterprise-grade cryptographic key management and security services in software. While not providing hardware-level security, it implements comprehensive HSM functionality including:

• Multi-user authentication and authorization
• Complete key lifecycle management
• Audit logging for compliance
• Secure file storage with encryption
• Digital signature operations (ED25519)
• Multiple access interfaces (Library API, CLI, REST API, Python API)

⚠️ Important: This is a virtualized HSM for educational and development purposes. For production systems, use hardware-backed HSMs or cloud HSM services.

Features

🔐 Authentication & Access Control

• Multi-user support with password-based authentication (PBKDF2-HMAC-SHA512)
• Role-based access control (User, Operator, Admin, Auditor)
• Session management with timeout protection
• Optional PIN support for two-factor authentication

🔑 Comprehensive Key Management

• Key generation: AES-128, AES-256, ED25519, HMAC, RSA, ECDSA
• Key lifecycle: Creation, activation, rotation, deprecation, destruction
• Key versioning with automatic rotation policies
• Key import/export with secure key wrapping
• Key usage policies and access controls

📝 Audit & Compliance

• Complete audit trail of all operations
• Tamper-evident logging with timestamps
• User attribution for all actions
• Queryable audit logs by time range and event type
• 13 tracked event types including key creation, access, rotation, signing

💾 Secure File Storage

• Chunked encryption for large files
• AES-256-GCM encryption per chunk
• Token-based retrieval system
• Compression support (zlib)
• Progress tracking for file operations

🛡️ Enhanced Security

• Secure memory management with memory locking
• Explicit key erasure from memory
• Thread-safe operations throughout
• Input validation and bounds checking
• Hardware security module detection (TPM, YubiKey, FIDO2, PKCS#11)

🌐 Multiple Interfaces

• Library API: Clean C API for programmatic access
• Python API: Complete Python bindings using ctypes
• CLI Tool: Command-line interface for scripting
• Enhanced CLI: Interactive mode with extended features
• REST API Server: JSON HTTP API for remote access
• TLS REST Server: Encrypted remote access

Architecture

virtual_hsm/
├── include/
│   ├── vhsm.h           # Main API header
│   └── vhsm_types.h     # Type definitions and constants
├── src/
│   ├── core/            # Library initialization and context management
│   ├── auth/            # Authentication and session management
│   ├── storage/         # Key storage and file operations
│   ├── crypto/          # Cryptographic primitives
│   ├── audit/           # Audit logging system
│   └── utils/           # Secure memory utilities
├── cli/                 # Command-line interface
├── server/              # REST API servers
├── python/              # Python API bindings
├── examples/            # Example applications
├── tests/               # Test suites
├── lib/                 # Built libraries (output)
└── bin/                 # Built executables (output)

Quick Start

Installation

# Install dependencies (Ubuntu/Debian)
sudo apt-get install build-essential libssl-dev zlib1g-dev uuid-dev

# Build everything
make all

# Install system-wide (optional)
sudo make install

Basic Usage (CLI)

# Initialize HSM and create admin user
bin/vhsm init ./my_hsm

# Create a user (interactive)
bin/vhsm user-create

# Generate an encryption key
bin/vhsm key-generate mykey AES-256 --encrypt

# Encrypt a file
bin/vhsm encrypt mykey input.txt encrypted.bin

# Decrypt the file
bin/vhsm decrypt mykey encrypted.bin output.txt

Basic Usage (Library API)

#include "vhsm.h"

int main() {
    vhsm_ctx_t ctx;
    vhsm_session_t session;

    // Initialize library
    vhsm_init(VHSM_LOG_LEVEL_INFO);

    // Create context
    vhsm_ctx_create(&ctx, "./hsm_storage");

    // Generate master key
    uint8_t master_key[32];
    vhsm_ctx_generate_master_key(ctx, master_key);

    // Create admin user
    vhsm_user_create(ctx, "admin", "password123", NULL, VHSM_ROLE_ADMIN);

    // Login
    vhsm_session_login(ctx, &session, "admin", "password123", NULL);

    // Generate encryption key
    vhsm_key_handle_t key;
    vhsm_key_generate(session, "my_aes_key", VHSM_KEY_TYPE_AES_256,
                      VHSM_KEY_USAGE_ENCRYPT | VHSM_KEY_USAGE_DECRYPT, &key);

    // Encrypt data
    uint8_t plaintext[] = "Secret message";
    uint8_t ciphertext[256];
    size_t ciphertext_len = sizeof(ciphertext);
    vhsm_encrypt(session, key, plaintext, strlen(plaintext),
                 ciphertext, &ciphertext_len);

    // Cleanup
    vhsm_session_logout(session);
    vhsm_ctx_destroy(ctx);
    vhsm_cleanup();

    return 0;
}

Usage

Library API

The library provides a comprehensive C API for all HSM operations:

Initialization:

vhsm_error_t vhsm_init(vhsm_log_level_t log_level);
vhsm_error_t vhsm_ctx_create(vhsm_ctx_t* ctx, const char* storage_path);
vhsm_error_t vhsm_ctx_generate_master_key(vhsm_ctx_t ctx, uint8_t* master_key);

User Management:

vhsm_error_t vhsm_user_create(vhsm_ctx_t ctx, const char* username,
                               const char* password, const char* pin,
                               vhsm_role_t role);
vhsm_error_t vhsm_session_login(vhsm_ctx_t ctx, vhsm_session_t* session,
                                 const char* username, const char* password,
                                 const char* pin);

Key Operations:

vhsm_error_t vhsm_key_generate(vhsm_session_t session, const char* name,
                                vhsm_key_type_t type, vhsm_key_usage_t usage,
                                vhsm_key_handle_t* handle);
vhsm_error_t vhsm_encrypt(vhsm_session_t session, vhsm_key_handle_t key,
                          const uint8_t* plaintext, size_t plaintext_len,
                          uint8_t* ciphertext, size_t* ciphertext_len);
vhsm_error_t vhsm_sign(vhsm_session_t session, vhsm_key_handle_t key,
                       const uint8_t* data, size_t data_len,
                       uint8_t* signature, size_t* signature_len);

File Storage:

vhsm_error_t vhsm_file_store(vhsm_session_t session, vhsm_key_handle_t key,
                              const char* file_path, char* token_out);
vhsm_error_t vhsm_file_retrieve(vhsm_session_t session, vhsm_key_handle_t key,
                                 const char* token, const char* dest_path);

See include/vhsm.h for complete API documentation.

CLI Usage

The original virtual_hsm binary provides basic command-line operations:

Compilation:

gcc -o virtual_hsm virtual_hsm.c -lcrypto

Master Key Generation:

# Generate master key (display only)
./virtual_hsm -generate_master_key

# Generate and store master key
./virtual_hsm -generate_master_key store_key

Key Management:

# Store a symmetric key (64 hex characters = 32 bytes)
echo -n "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" | \
  ./virtual_hsm -store mykey

# Generate random key
openssl rand -hex 32 | ./virtual_hsm -store randomkey

# Retrieve a key
./virtual_hsm -retrieve mykey

# List all keys
./virtual_hsm -list

Digital Signatures:

# Generate key pair
./virtual_hsm -generate_key_pair signing_key

# Sign data
echo -n "Important message" | ./virtual_hsm -sign signing_key -o signature.bin

# Verify signature
./virtual_hsm -verify signing_key -i message.txt -s signature.bin

Custom Keystore:

# Use custom keystore and master key files
./virtual_hsm -keystore custom.dat -master custom.key -list

# Use master key from GitHub Secrets
./virtual_hsm -master_key ${{ secrets.MASTER_KEY }} -store api_key

Enhanced HSM

The enhanced HSM (hsm_enhanced) provides additional security features:

Build:

gcc -o hsm_enhanced hsm_enhanced.c -lcrypto -lssl

Features:

• Key lifecycle management with rotation
• Comprehensive audit logging
• Access control and user identification
• Hardware security module detection
• Secure memory management

Usage:

# Scan for hardware security modules
./hsm_enhanced -scan_hardware

# Set user identity for audit logging
./hsm_enhanced -set_user "admin@example.com"

# View key metadata
./hsm_enhanced -key_info mykey

# Rotate encryption key
./hsm_enhanced -rotate_key mykey

# View audit logs (last 7 days)
./hsm_enhanced -audit_logs 7

REST API Server

Start the REST API server for remote access:

# Build server
make server

# Run server
bin/vhsm-server --port 8443 --storage ./hsm_data

# Run with TLS
bin/vhsm-server-tls --port 8443 --storage ./hsm_data \
  --cert server.crt --key server.key

API Endpoints:

# Get version
curl http://localhost:8443/api/version

# Create user
curl -X POST http://localhost:8443/api/user/create \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"pass123","role":"admin"}'

# Login
curl -X POST http://localhost:8443/api/session/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"pass123"}'

Python API

The Virtual HSM provides a complete Python library for easy integration with Python applications.

Installation:

# Build the C library first
make lib

# Install Python package
cd python
pip install -e .

Quick Example:

import vhsm

# Initialize HSM
hsm = vhsm.HSM('/tmp/hsm_storage')
hsm.generate_master_key()

# Create admin user
hsm.create_user('admin', 'password123', role=vhsm.ROLE_ADMIN)

# Login and use HSM
with hsm.login('admin', 'password123') as session:
    # Generate encryption key
    key_handle = session.generate_key('my_key', vhsm.KEY_TYPE_AES_256)

    # Encrypt data
    ciphertext, iv = session.encrypt(key_handle, b'secret data')

    # Decrypt data
    plaintext = session.decrypt(key_handle, ciphertext, iv)
    print(plaintext)  # b'secret data'

    # Generate signing key
    sign_key = session.generate_key('sign_key', vhsm.KEY_TYPE_ED25519)

    # Sign document
    signature = session.sign(sign_key, b'Important document')

    # Verify signature
    is_valid = session.verify(sign_key, b'Important document', signature)

Features:

• Pure Python interface using ctypes (no compilation required)
• Context managers for automatic session cleanup
• Full error handling with descriptive exceptions
• Support for all HSM operations (encryption, signatures, key management)

Documentation:

• Full API reference: python/README.md
• Complete example: examples/python/hsm_example.py

Run the example:

cd examples/python
python hsm_example.py

Key Management

Key Types Supported

• AES-128: 128-bit symmetric encryption
• AES-256: 256-bit symmetric encryption (recommended)
• ED25519: Edwards-curve digital signatures
• HMAC: Message authentication codes
• RSA: RSA public-key cryptography (2048-4096 bit)
• ECDSA: Elliptic curve digital signatures

Key Lifecycle States

• PRE_ACTIVE: Key created but not yet active
• ACTIVE: Key is currently usable
• DEPRECATED: Key still usable but scheduled for retirement
• COMPROMISED: Key suspected of compromise, should not be used
• DESTROYED: Key has been securely erased

Key Rotation

Automatic key rotation based on policies:

// Rotate key when needed
vhsm_key_rotate(session, key_handle, &new_key_handle);

// Enhanced HSM automatic rotation (90-day default)
./hsm_enhanced -rotate_key encryption_key

Digital Signatures

ED25519 Signatures

# Generate signing key pair
./virtual_hsm -generate_key_pair signer

# Sign a file
./virtual_hsm -sign signer -i document.pdf -o document.sig

# Verify signature
./virtual_hsm -verify signer_public -i document.pdf -s document.sig

Public Key Export/Import

# Export public key
./virtual_hsm -export_public_key signer_public -o public.pem

# Import public key
./virtual_hsm -import_public_key trusted_key -i public.pem

File Storage

The Virtual HSM provides secure file storage with encryption and chunking:

Using the Library API

// Store a file
char token[128];
vhsm_file_store(session, encryption_key, "sensitive.doc", token);
printf("File stored with token: %s\n", token);

// Retrieve the file
vhsm_file_retrieve(session, encryption_key, token, "./retrieved.doc");

Using the Token Tool

# Generate encryption key
./token/token generate-key master.key

# Store a file
./token/token store document.pdf --key master.key
# Returns: Token: 550e8400-e29b-41d4-a716-446655440000

# Retrieve file
./token/token retrieve 550e8400-e29b-41d4-a716-446655440000 ./output/ --key master.key

Features:

• Random-size chunking (4KB-16KB) for obfuscation
• Per-chunk AES-256-GCM encryption
• Optional zlib compression
• Progress tracking for large files
• UUID-based token system

Security Features

Secure Memory Management

// Memory is locked to prevent swapping
secure_lock_memory(key_buffer, key_size);

// Keys are explicitly wiped after use
secure_wipe(key_buffer, key_size);

Hardware Security Module Detection

$ ./hsm_enhanced -scan_hardware

=== Scanning for HSM Hardware ===

1. FIDO2/U2F Devices (YubiKey, etc.):
   ✗ No HID devices found

2. Trusted Platform Module (TPM):
   ✓ TPM device detected at /dev/tpm0

3. PKCS#11 Tokens:
   ✗ No PKCS#11 libraries found

4. Smart Cards (OpenSC):
   ✗ OpenSC not found

=== Software Security Features ===
✓ AES-256-GCM encryption
✓ ED25519 digital signatures
✓ Secure memory management
✓ Audit logging enabled

Audit Logging

All operations are logged with:

• Timestamp
• Event type (13 types tracked)
• Key/resource affected
• User who performed the action
• Operation details
• Success/failure status

Event Types:

• KEY_CREATED, KEY_ACCESSED, KEY_ROTATED
• SIGN_OPERATION, VERIFY_OPERATION
• ENCRYPT_OPERATION, DECRYPT_OPERATION
• FILE_STORED, FILE_RETRIEVED
• USER_CREATED, SESSION_LOGIN, SESSION_LOGOUT
• CONFIG_CHANGE

Query Audit Logs:

# View last 30 days
./hsm_enhanced -audit_logs 30

# Example output:
2025-11-21 10:30:45|KEY_CREATED|encryption_key|admin@example.com|AES-256 key generated|SUCCESS
2025-11-21 10:31:12|ENCRYPT_OPERATION|encryption_key|user@example.com|File encrypted|SUCCESS

GitHub Actions Integration

Using Master Key from Secrets

name: HSM Operations
on: [push]

jobs:
  secure-build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build HSM
        run: gcc -o virtual_hsm virtual_hsm.c -lcrypto

      - name: Store API Key
        run: |
          echo ${{ secrets.API_KEY }} | ./virtual_hsm \
            -master_key ${{ secrets.MASTER_KEY }} \
            -store service_key

      - name: Sign Release
        run: |
          ./virtual_hsm \
            -master_key ${{ secrets.MASTER_KEY }} \
            -sign release_key \
            -i release.zip \
            -o release.sig

Generating Master Key for Secrets

# Generate and display master key for GitHub Secrets
./virtual_hsm -generate_master_key

# Output:
# Generated Master Key (hex format for GitHub Secret):
# a1b2c3d4e5f6...

Add this value to your repository secrets as MASTER_KEY.

Docker & Kubernetes

Full Docker and Kubernetes support with production-ready configurations and multiple secrets management solutions.

See k8s/README.md for comprehensive Kubernetes documentation including HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and Google Secret Manager integration.

Quick Start - Docker

# Build and run
docker build -t virtual-hsm .
docker run -d -p 8443:8443 virtual-hsm

# Or use docker-compose
docker-compose up -d

Quick Start - Kubernetes (Minikube)

cd k8s
./minikube-setup.sh

Building from Source

Quick Build (All Platforms)

Linux/macOS:

# Automated build script (recommended)
./build.sh

# Or build specific components
./build.sh library    # Library only
./build.sh cli        # CLI only
./build.sh standalone # Standalone tools only
./build.sh test       # Build and run tests
./build.sh package    # Create distribution package
./build.sh clean      # Clean build artifacts

macOS (with Homebrew):

# macOS-specific build (handles Homebrew dependencies)
./build-macos.sh

# Or install dependencies and build
./build-macos.sh deps     # Install dependencies
./build-macos.sh all      # Build everything
./build-macos.sh bundle   # Create .app bundle

Windows:

REM Using MinGW or Visual Studio
build.bat

REM Or specific targets
build.bat standalone  # Build standalone tools
build.bat clean       # Clean build artifacts

Prerequisites

Linux (Ubuntu/Debian):

sudo apt-get install build-essential libssl-dev zlib1g-dev uuid-dev libsodium-dev

Linux (Fedora/RHEL):

sudo dnf install gcc make openssl-devel zlib-devel libuuid-devel libsodium-devel

macOS:

brew install openssl zlib ossp-uuid libsodium

Windows:

• Install MinGW-w64 or Visual Studio
• Install OpenSSL for Windows

Optional (for FIDO2/YubiKey support):

sudo apt-get install libfido2-dev libjson-c-dev  # Linux
brew install libfido2 json-c                      # macOS

Manual Build Commands

# Build everything
make all

# Build specific components
make lib          # Libraries only
make cli          # CLI binary
make server       # REST API server
make server-tls   # TLS REST API server
make examples     # Example programs

# Build with tests
make test-all     # Build and run all tests

# Clean build
make clean

Build Output

lib/libvhsm.a       # Static library
lib/libvhsm.so      # Shared library (Linux)
lib/libvhsm.dylib   # Shared library (macOS)
bin/vhsm            # CLI application
bin/vhsm-server     # REST API server
bin/vhsm-server-tls # TLS REST server
bin/example_basic   # Example program
virtual_hsm         # Standalone CLI tool
hsm_enhanced        # Enhanced HSM tool

Environment Variables

Build scripts support these environment variables:

BUILD_TYPE=Debug ./build.sh      # Debug build
ENABLE_TESTS=0 ./build.sh        # Skip tests
ENABLE_PYTHON=0 ./build.sh       # Skip Python library
JOBS=8 ./build.sh                # Parallel jobs

Cross-Platform Notes

Linux: Full support for all features macOS: Full support, uses Homebrew for dependencies Windows: Standalone tools supported, library API requires MinGW or MSVC

Configuration

The HSM can be configured via configuration file hsm_config.conf:

# Create default configuration
./hsm_enhanced -create_config hsm_config.conf

# View current configuration
./hsm_enhanced -show_config

Key Configuration Options:

# Key Rotation
key_rotation_days=90          # Days before automatic rotation
max_key_age_days=365          # Maximum key age
rotation_enabled=1            # Enable auto-rotation

# Session Management
session_timeout_seconds=3600  # 1 hour timeout
max_concurrent_sessions=0     # Unlimited

# Security
max_failed_auth_attempts=3    # Lockout after 3 failures
password_min_length=8         # Minimum password length

# Audit
audit_enabled=1               # Enable audit logging
audit_log_file=hsm_audit.log  # Log file path

# Advanced
fips_mode=0                   # FIPS 140-2 mode
secure_memory=1               # Lock keys in memory

Change Rotation Frequency:

# Set rotation period to 30 days
./hsm_enhanced -set_rotation_period 30

# Set rotation period to 180 days
./hsm_enhanced -set_rotation_period 180

# Disable automatic rotation
./hsm_enhanced -set_rotation_enabled 0

# Enable automatic rotation
./hsm_enhanced -set_rotation_enabled 1

Programmatic Configuration (Python):

import vhsm

# Load and modify configuration
hsm = vhsm.HSM('/tmp/hsm_storage')

# Set rotation period (requires enhanced HSM)
# Configuration is managed through hsm_config.conf file

Testing

Run All Tests

# Run all unit and integration tests
make test-all

# Run specific test suites
make test-crypto        # Cryptographic operations
make test-he            # Homomorphic encryption
make test-integration   # Integration tests
make test-cli           # CLI interface tests
make test-rest          # REST API tests

Test Coverage

• ✅ Cryptographic operations (encryption, signing, verification)
• ✅ Key generation and management
• ✅ User authentication and sessions
• ✅ Audit logging
• ✅ File storage operations
• ✅ CLI interface
• ✅ REST API endpoints

CI/CD Testing

All tests run automatically on every push via GitHub Actions:

• Unit tests
• Integration tests
• Cross-platform compilation (Linux, Windows)
• Security scanning
• API compliance checks

Documentation

API Documentation

Complete API documentation is available in header files:

• include/vhsm.h - Main API functions
• include/vhsm_types.h - Type definitions and constants

Examples

See the examples/ directory for:

• Basic usage examples
• Multi-user scenarios
• File encryption workflows
• Integration examples

Additional Resources

• GitHub Actions Workflows
• Test Suite
• Server Implementation

Known Limitations

This is a virtual HSM for development and educational purposes. It lacks several features of production HSMs:

Security Limitations

• ❌ No hardware-backed key storage
• ❌ No tamper-resistant hardware
• ❌ No physical security protections
• ❌ Limited protection against side-channel attacks
• ❌ Keys stored in process memory (though with secure wiping)

Performance Limitations

• Single-threaded for most operations (though thread-safe)
• No hardware acceleration
• Limited to software cryptography performance

Feature Limitations

• No PKCS#11 interface (custom API only)
• Limited key types compared to enterprise HSMs
• No clustering or high-availability features
• No backup/restore mechanisms (manual file copy only)

Recommended Alternatives for Production

For production systems, consider:

• Hardware HSMs: Thales, Entrust, Utimaco
• Cloud HSM: AWS CloudHSM, Azure Key Vault, Google Cloud KMS
• TPM: For device-specific key storage
• Hardware Security Keys: YubiKey, SoloKey for authentication

License

This project is for educational and development purposes. See individual source files for license information.

Support and Contributing

• Issues: Report bugs or request features via GitHub Issues
• Pull Requests: Contributions are welcome
• Testing: All PRs must pass CI/CD tests

Version Information

• Current Version: 2.0.0
• API Version: 2.0
• Protocol Version: 2.0

What's New in 2.0.0 (November 2024)

Security Enhancements:

• Fixed 6 critical vulnerabilities
• Fixed 20 high severity issues
• Enhanced cryptographic key derivation (PBKDF2 with 100k iterations)
• Secure random session IDs (no more predictable IDs)
• Strengthened TLS configuration
• Fixed CORS vulnerability
• Removed test key fallbacks

New Features:

• Docker support with security hardening
• Kubernetes deployment manifests
• Multiple secrets management solutions (Vault, AWS, Azure, GCP)
• Minikube integration
• Docker Compose configuration
• Comprehensive security documentation

See SECURITY.md for detailed security information.

For version history and changelog, see commit history on GitHub.