Skip to content

Home

Jump to bottom
Vyacheslav Meyerzon edited this page Aug 1, 2025 · 8 revisions

Git Security Scanner Wiki

Welcome to the Git Security Scanner documentation!

Table of Contents

Installation

System Requirements

  • Python: 3.7 or higher
  • Operating Systems: Windows, macOS, Linux
  • Git: Any recent version
  • Memory: ~100MB for typical repositories
  • Disk Space: ~50MB for installation

Installation Methods

Install from PyPI (Recommended)

pip install git-security-scanner

Install in Virtual Environment

# Create virtual environment
python -m venv scanner-env

# Activate it
# On Windows:
scanner-env\Scripts\activate
# On macOS/Linux:
source scanner-env/bin/activate

# Install
pip install git-security-scanner

Install from Source

git clone https://github.com/vyacheslavmeyerzon/security-scanner.git
cd security-scanner
pip install -e .

Verify Installation

git-security-scanner --version

Getting Started

Your First Scan

Navigate to any Git repository and run:

git-security-scanner

This will scan:

  • All files in the working directory
  • The last 100 commits in history
  • Display findings in the console

Understanding the Output

The scanner uses color-coded severity levels:

  • 🔴 CRITICAL (Red): Database passwords, private keys
  • 🟡 HIGH (Yellow): API keys, access tokens
  • 🟣 MEDIUM (Purple): Potential secrets
  • 🔵 LOW (Blue): Configuration issues

Example output:

[CRITICAL] MongoDB Connection
  Description: MongoDB Connection String with credentials
  File: config.py
  Line: 15
  Secret: mongodb://user:****@localhost/db

Common Use Cases

Pre-commit Scanning

git add .
git-security-scanner --pre-commit

Quick Repository Audit

git-security-scanner --no-history

Export Findings

git-security-scanner --export security-report.html

CI/CD Integration

git-security-scanner --quiet || exit 1

Configuration

Configuration File

Create .gitscannerrc.json or .gitscannerrc.yaml in your repository:

{
  "patterns": {
    "custom": [
      {
        "name": "Company API Key",
        "pattern": "COMP-[A-Z0-9]{32}",
        "severity": "HIGH",
        "description": "Company internal API key"
      }
    ],
    "disabled": ["Generic Secret", "Environment Variable"]
  },
  "scan": {
    "history_limit": 50,
    "max_file_size_mb": 5,
    "parallel_workers": 4,
    "show_progress": true
  },
  "output": {
    "format": "console",
    "color": true,
    "quiet": false,
    "min_severity": "MEDIUM"
  },
  "ignore": {
    "paths": ["vendor/", "third_party/"],
    "patterns": ["*.test.js", "*.spec.ts"]
  },
  "cache": {
    "enabled": true,
    "ttl_hours": 48,
    "show_hits": false
  }
}

Environment Variables

  • SCANNER_HISTORY_LIMIT - Limit commit history scan
  • SCANNER_MAX_FILE_SIZE - Maximum file size to scan (MB)
  • SCANNER_PARALLEL_WORKERS - Number of parallel workers
  • SCANNER_SHOW_PROGRESS - Show progress bars (true/false)
  • SCANNER_OUTPUT_FORMAT - Output format (console/json/html/csv)
  • SCANNER_NO_COLOR - Disable colored output
  • SCANNER_QUIET - Enable quiet mode
  • SCANNER_MIN_SEVERITY - Minimum severity to report
  • SCANNER_CACHE_ENABLED - Enable/disable caching
  • SCANNER_CACHE_TTL - Cache time-to-live in hours

Ignore Files

Create .gitscannerignore:

# Ignore test files
tests/
*.test.py
*.spec.js

# Ignore vendor directories
vendor/
node_modules/
third_party/

# Ignore specific files
config.example.json
.env.example

# Ignore by pattern
**/temp/*
*.log

Usage Examples

Basic Scanning

# Scan current repository
git-security-scanner

# Scan specific repository
git-security-scanner /path/to/repo

# Scan with custom config
git-security-scanner -c custom-config.json

Filtering Results

# Show only high severity and above
git-security-scanner --min-severity HIGH

# Quiet mode (minimal output)
git-security-scanner --quiet

# No color output
git-security-scanner --no-color

Performance Options

# Skip commit history
git-security-scanner --no-history

# Limit history scan
git-security-scanner --history-limit 10

# Disable progress bars
git-security-scanner --no-progress

# Disable cache
git-security-scanner --no-cache

Export Options

# Export to JSON
git-security-scanner --export findings.json

# Export to HTML
git-security-scanner --export report.html

# Export to CSV
git-security-scanner --export results.csv

# Export to Markdown
git-security-scanner --export summary.md

# Force output format
git-security-scanner --output-format json

Pre-commit Hook

# Check only staged files
git-security-scanner --pre-commit

# Pre-commit with severity filter
git-security-scanner --pre-commit --min-severity HIGH

Cache Management

# Clear cache
git-security-scanner --clear-cache

# Show cache statistics
git-security-scanner --cache-stats

# Disable cache for this scan
git-security-scanner --no-cache

Pattern Management

# Show all patterns
git-security-scanner --show-patterns

# Generate example config
git-security-scanner --generate-config

# Validate configuration
git-security-scanner --validate-config

API Reference

Using as a Python Library

from security_scanner import SecurityScanner
from security_scanner.patterns import Severity

# Initialize scanner
scanner = SecurityScanner(
    repo_path="/path/to/repo",
    ignore_file=".gitscannerignore",
    config=None  # Use default config
)

# Scan methods
result = scanner.scan_full(include_history=True)
result = scanner.scan_staged_files()
result = scanner.scan_working_directory()
result = scanner.scan_commit_history(limit=50)
result = scanner.scan_file("path/to/file.py")

# Access results
print(f"Found {len(result.findings)} potential secrets")
print(f"Scanned {result.scanned_files} files")
print(f"Skipped {result.skipped_files} files")

# Filter results
high_severity = result.filter_by_severity(Severity.HIGH)
unique_findings = result.get_unique_findings()

# Add custom patterns
scanner.add_custom_pattern(
    name="Custom Pattern",
    pattern=r"PATTERN_[0-9]+",
    severity="HIGH",
    description="Description of what this detects"
)

# Remove patterns
scanner.remove_pattern("Generic Secret")

# Get all patterns
patterns = scanner.get_patterns()

Export Results Programmatically

from security_scanner.export import ReportGenerator
from pathlib import Path

# Create report generator
report = ReportGenerator(result.findings, scan_stats={
    "scanned_files": result.scanned_files,
    "skipped_files": result.skipped_files
})

# Export to different formats
report.export_to_html(Path("report.html"))
report.export_to_csv(Path("findings.csv"))
report.export_to_json_with_stats(Path("report.json"))
report.export_to_markdown(Path("summary.md"))

Custom Configuration

from security_scanner.config import ScannerConfig

# Load custom config
config = ScannerConfig(Path("custom-config.json"))

# Modify config programmatically
config.set("scan.history_limit", 20)
config.set("output.min_severity", "HIGH")

# Use with scanner
scanner = SecurityScanner(config=config)

CI/CD Integration

GitHub Actions

Create .github/workflows/security-scan.yml:

name: Security Scan

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  scan:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v3
      with:
        fetch-depth: 0  # Full history for commit scanning
    
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
    
    - name: Install Git Security Scanner
      run: pip install git-security-scanner
    
    - name: Run Security Scan
      run: |
        git-security-scanner --quiet --export scan-results.json
        
    - name: Upload scan results
      uses: actions/upload-artifact@v3
      if: failure()
      with:
        name: security-scan-results
        path: scan-results.json
        
    - name: Comment PR
      uses: actions/github-script@v6
      if: failure() && github.event_name == 'pull_request'
      with:
        script: |
          github.rest.issues.createComment({
            issue_number: context.issue.number,
            owner: context.repo.owner,
            repo: context.repo.repo,
            body: '⚠️ Security scan found potential secrets. Please review the scan results.'
          })

GitLab CI

Create .gitlab-ci.yml:

stages:
  - test

security-scan:
  stage: test
  image: python:3.11-slim
  before_script:
    - apt-get update && apt-get install -y git
    - pip install git-security-scanner
  script:
    - git-security-scanner --quiet --export report.html
  artifacts:
    expose_as: 'Security Scan Report'
    paths:
      - report.html
    when: on_failure
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH == "main"'
    - if: '$CI_COMMIT_BRANCH == "develop"'

Jenkins

pipeline {
    agent any
    
    stages {
        stage('Security Scan') {
            steps {
                script {
                    try {
                        sh '''
                            pip install git-security-scanner
                            git-security-scanner --quiet --export scan-results.json
                        '''
                    } catch (Exception e) {
                        currentBuild.result = 'UNSTABLE'
                        archiveArtifacts artifacts: 'scan-results.json', allowEmptyArchive: true
                        error("Security scan found potential secrets")
                    }
                }
            }
        }
    }
    
    post {
        always {
            publishHTML([
                allowMissing: true,
                alwaysLinkToLastBuild: true,
                keepAll: true,
                reportDir: '.',
                reportFiles: 'scan-results.json',
                reportName: 'Security Scan Report'
            ])
        }
    }
}

Pre-commit Hook

Create .pre-commit-config.yaml:

repos:
  - repo: local
    hooks:
      - id: git-security-scanner
        name: Git Security Scanner
        entry: git-security-scanner --pre-commit
        language: system
        pass_filenames: false
        always_run: true

Install pre-commit:

pip install pre-commit
pre-commit install

Docker Integration

FROM python:3.11-slim

RUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/*
RUN pip install git-security-scanner

WORKDIR /repo

ENTRYPOINT ["git-security-scanner"]

Usage:

docker build -t security-scanner .
docker run -v $(pwd):/repo security-scanner --quiet

Custom Patterns

Pattern Structure

Patterns are defined using regular expressions:

{
  "name": "Pattern Name",
  "pattern": "regex-pattern-here",
  "severity": "HIGH",
  "description": "What this pattern detects"
}

Adding Custom Patterns

Via Configuration File

{
  "patterns": {
    "custom": [
      {
        "name": "Acme API Key",
        "pattern": "acme_[a-f0-9]{40}",
        "severity": "HIGH",
        "description": "Acme service API key"
      },
      {
        "name": "Internal Token",
        "pattern": "INT_TOKEN_[A-Z0-9]{32}",
        "severity": "CRITICAL",
        "description": "Internal authentication token"
      }
    ]
  }
}

Via Python API

scanner.add_custom_pattern(
    name="Database URL",
    pattern=r"postgres://[^:]+:[^@]+@[^/]+/\w+",
    severity="CRITICAL",
    description="PostgreSQL connection URL with credentials"
)

Pattern Examples

API Keys

{
  "name": "Stripe Secret Key",
  "pattern": "sk_(test|live)_[a-zA-Z0-9]{24,}",
  "severity": "CRITICAL",
  "description": "Stripe payment API secret key"
}

Tokens

{
  "name": "JWT Token",
  "pattern": "eyJ[a-zA-Z0-9_-]+\\.[a-zA-Z0-9_-]+\\.[a-zA-Z0-9_-]+",
  "severity": "HIGH",
  "description": "JSON Web Token"
}

Passwords

{
  "name": "Hardcoded Password",
  "pattern": "(?i)(password|passwd|pwd)\\s*=\\s*[\"'][^\"']{8,}[\"']",
  "severity": "HIGH",
  "description": "Hardcoded password assignment"
}

Severity Guidelines

  • CRITICAL: Immediate security risk

    • Database credentials
    • Private keys
    • Production passwords

  • HIGH: Significant security concern

    • API keys with write access
    • OAuth tokens
    • Cloud service credentials

  • MEDIUM: Potential security issue

    • Read-only API keys
    • Development tokens
    • Weak secrets

  • LOW: Minor concern

    • Configuration values
    • Non-sensitive tokens
    • Example credentials

Testing Custom Patterns

# Test your pattern
import re

pattern = re.compile(r"acme_[a-f0-9]{40}")
test_string = "api_key = 'acme_1234567890abcdef1234567890abcdef12345678'"

if pattern.search(test_string):
    print("Pattern matched!")

Disabling Built-in Patterns

{
  "patterns": {
    "disabled": [
      "Generic Secret",
      "Environment Variable",
      "Generic API Key"
    ]
  }
}

Troubleshooting

Common Issues

"Not a Git repository" error

Problem: Scanner can't find Git repository

Error: Not a Git repository: /path/to/directory

Solutions:

  • Ensure you're in a directory with .git folder
  • Initialize Git: git init
  • Specify correct path: git-security-scanner /correct/path

No output / Silent execution

Problem: Scanner runs but shows no output

Solutions:

# Check if repository has files
git ls-files

# Run with verbose output
git-security-scanner --no-quiet

# Check for ignore files
cat .gitscannerignore

# Disable quiet mode in config
git-security-scanner --output-format console

Import errors

Problem: Module import failures

ImportError: No module named 'security_scanner'

Solutions:

# Reinstall package
pip uninstall git-security-scanner
pip install git-security-scanner

# Check Python version
python --version  # Should be 3.7+

# Install in correct environment
which python
which pip

Performance issues

Problem: Scanning takes too long

Solutions:

# Skip history scanning
git-security-scanner --no-history

# Limit commit depth
git-security-scanner --history-limit 10

# Check repository size
git count-objects -vH

# Use cache (enabled by default)
git-security-scanner --cache-stats

# Reduce parallel workers if system is constrained
git-security-scanner -c config.json
# config.json: {"scan": {"parallel_workers": 2}}

False positives

Problem: Scanner reports non-secrets as findings

Solutions:

  1. Use .gitscannerignore:
# Ignore test files
tests/
*_test.py
*.example

# Ignore specific patterns
docs/examples/
  1. Disable specific patterns:
{
  "patterns": {
    "disabled": ["Environment Variable", "Generic Secret"]
  }
}
  1. Increase minimum severity:
git-security-scanner --min-severity HIGH

Memory issues

Problem: Scanner runs out of memory on large repositories

Solutions:

# Limit file size scanning
# config.json
{
  "scan": {
    "max_file_size_mb": 5
  }
}

# Scan in batches
git-security-scanner --no-history
git-security-scanner --history-limit 50 --no-cache

Error Messages

"Permission denied"

  • Check file permissions
  • Run with appropriate user privileges
  • Ensure Git repository is accessible

"File too large"

  • File exceeds max_file_size_mb limit
  • Increase limit in configuration
  • Add file to .gitscannerignore

"Cache database is locked"

  • Another scan is running
  • Clear cache: git-security-scanner --clear-cache
  • Delete cache manually: rm -rf .git/.scanner-cache

Debug Mode

For detailed debugging information:

# Set log level via environment
export LOG_LEVEL=DEBUG
git-security-scanner

# Python debugging
python -m pdb -m security_scanner.cli

FAQ

General Questions

Q: How is this different from other tools like TruffleHog or GitLeaks?

A: Git Security Scanner offers several unique features:

  • Native Python implementation with pip installation
  • Built-in progress bars for better UX
  • Interactive HTML reports with filtering
  • Intelligent caching system
  • Extensive pattern library including AI/ML services
  • Comprehensive configuration options

Q: What types of secrets can it detect?

A: The scanner detects 25+ types including:

  • Cloud service credentials (AWS, Azure, GCP)
  • API keys (OpenAI, Stripe, GitHub, etc.)
  • Database connection strings
  • Private keys and certificates
  • Generic passwords and secrets
  • Custom patterns you define

Q: Is it safe to use on sensitive repositories?

A: Yes, the scanner:

  • Runs locally only
  • Doesn't send data anywhere
  • Truncates secrets in output for safety
  • Can be configured to minimize exposure

Usage Questions

Q: How do I scan only my latest changes?

A: Use pre-commit mode:

git-security-scanner --pre-commit

Q: Can I use this in CI/CD pipelines?

A: Yes! The scanner is designed for CI/CD use:

git-security-scanner --quiet || exit 1

Q: How do I ignore false positives?

A: Three methods:

  1. Create .gitscannerignore file
  2. Disable specific patterns in config
  3. Use --min-severity to filter results

Q: Can I scan non-Git directories?

A: No, the scanner requires a Git repository to function properly.

Performance Questions

Q: Why is scanning slow on large repositories?

A: Try these optimizations:

# Skip history
git-security-scanner --no-history

# Limit commits
git-security-scanner --history-limit 10

# Check cache usage
git-security-scanner --cache-stats

Q: How does caching work?

A: The scanner caches:

  • File scan results (24-hour TTL by default)
  • Commit scan results
  • Cache is stored in .git/.scanner-cache

Q: Can I run multiple scans in parallel?

A: Yes, but they'll share the same cache database which may cause lock conflicts. Use --no-cache for parallel scans.

Configuration Questions

Q: Where should I put the configuration file?

A: The scanner looks for config files in this order:

  1. Path specified with -c flag
  2. Current directory: .gitscannerrc, .gitscannerrc.json, .gitscannerrc.yaml
  3. Parent directories (up to 5 levels)

Q: Can I use environment variables for configuration?

A: Yes, all major options have environment variables:

  • SCANNER_HISTORY_LIMIT
  • SCANNER_MIN_SEVERITY
  • SCANNER_QUIET
  • etc.

Q: How do I add custom patterns?

A: Add them to your config file:

{
  "patterns": {
    "custom": [{
      "name": "My Pattern",
      "pattern": "PATTERN_[0-9]+",
      "severity": "HIGH",
      "description": "Description"
    }]
  }
}

Troubleshooting Questions

Q: I get "command not found" error

A: Ensure:

  • Package is installed: pip install git-security-scanner
  • Python scripts directory is in PATH
  • Or use: python -m security_scanner.cli

Q: The scanner finds no results but I know there are secrets

A: Check:

  • Files aren't in .gitscannerignore
  • Patterns aren't disabled in config
  • Files are tracked by Git: git ls-files
  • Minimum severity isn't too high

Q: Can I contribute new patterns?

A: Yes! See our Contributing Guide for details on adding patterns.

Support

License

MIT License - see LICENSE for details.

Clone this wiki locally