Build CVE Matter-Analysis OS: Python 3.11 blue-team platform with ML, K8s, and GPU support

.copilot/tasks.md

@@ -0,0 +1,171 @@
+# GitHub Copilot Tasks Configuration
+
+## Project Tasks
+
+### Development Tasks
+
+- **Ingest CVE Data**
+  - Command: `cve-matter ingest --source nvd --output data/cve_data.json`
+  - Description: Fetch and ingest CVE data from NVD
+  - Category: Data Collection
+
+- **Run Alignment Analysis**
+  - Command: `cve-matter align --method procrustes --input data/cve_data.json`
+  - Description: Perform Procrustes alignment on CVE features
+  - Category: Analysis
+
+- **Run Super-Learner Prediction**
+  - Command: `cve-matter arbiter --input data/cve_data.json --n-folds 5`
+  - Description: Execute super-learner ensemble predictions
+  - Category: Machine Learning
+
+- **Compute Epsilon Values**
+  - Command: `cve-matter refract --input data/cve_data.json --use-gpu`
+  - Description: Calculate epsilon refraction values (GPU-accelerated)
+  - Category: Advanced Analysis
+
+- **Evaluate Model Evidence**
+  - Command: `cve-matter evidence --input data/cve_data.json --criteria bic waic`
+  - Description: Compute BIC and WAIC for model selection
+  - Category: Model Evaluation
+
+### Testing Tasks
+
+- **Run Unit Tests**
+  - Command: `pytest tests/ -v`
+  - Description: Execute all unit tests
+  - Category: Testing
+
+- **Run Tests with Coverage**
+  - Command: `pytest tests/ -v --cov=cve_matter --cov-report=html`
+  - Description: Run tests and generate coverage report
+  - Category: Testing
+
+- **Lint Code**
+  - Command: `ruff check cve_matter/ tests/`
+  - Description: Check code style with ruff
+  - Category: Quality
+
+- **Format Code**
+  - Command: `black cve_matter/ tests/`
+  - Description: Auto-format code with black
+  - Category: Quality
+
+- **Type Check**
+  - Command: `mypy cve_matter/`
+  - Description: Run static type checking
+  - Category: Quality
+
+### Docker Tasks
+
+- **Build CPU Image**
+  - Command: `docker build --target cpu -t cve-matter-analysis:cpu .`
+  - Description: Build CPU-only Docker image
+  - Category: Containers
+
+- **Build CUDA Image**
+  - Command: `docker build --target cuda -t cve-matter-analysis:cuda .`
+  - Description: Build GPU-enabled Docker image
+  - Category: Containers
+
+- **Run with Docker Compose**
+  - Command: `docker-compose up cve-matter-cpu`
+  - Description: Start CVE Matter with Docker Compose
+  - Category: Containers
+
+- **Scan Container with Trivy**
+  - Command: `trivy image cve-matter-analysis:cpu`
+  - Description: Scan Docker image for vulnerabilities
+  - Category: Security
+
+### Kubernetes Tasks
+
+- **Apply RuntimeClass**
+  - Command: `kubectl apply -f k8s/gvisor-runtime.yaml`
+  - Description: Deploy gVisor RuntimeClass
+  - Category: Kubernetes
+
+- **Deploy CRD**
+  - Command: `kubectl apply -f k8s/policy-trigger-crd.yaml`
+  - Description: Deploy PolicyTrigger CRD
+  - Category: Kubernetes
+
+- **Deploy Webhook**
+  - Command: `kubectl apply -f k8s/admission-webhook.yaml`
+  - Description: Deploy admission webhook
+  - Category: Kubernetes
+
+- **Submit Argo Workflow**
+  - Command: `argo submit argo/epsilon-sweep-workflow.yaml`
+  - Description: Run GPU epsilon sweep workflow
+  - Category: Workflows
+
+### Terraform Tasks
+
+- **Initialize Terraform**
+  - Command: `cd terraform && terraform init`
+  - Description: Initialize Terraform configuration
+  - Category: Infrastructure
+
+- **Plan Infrastructure**
+  - Command: `cd terraform && terraform plan`
+  - Description: Preview infrastructure changes
+  - Category: Infrastructure
+
+- **Apply Infrastructure**
+  - Command: `cd terraform && terraform apply`
+  - Description: Deploy GKE cluster and GPU nodes
+  - Category: Infrastructure
+
+- **Destroy Infrastructure**
+  - Command: `cd terraform && terraform destroy`
+  - Description: Tear down infrastructure
+  - Category: Infrastructure
+
+### Security Tasks
+
+- **Run CodeQL Analysis**
+  - Command: `codeql database analyze`
+  - Description: Perform static security analysis
+  - Category: Security
+
+- **Scan Dependencies**
+  - Command: `pip-audit`
+  - Description: Check for vulnerable dependencies
+  - Category: Security
+
+- **Generate SBOM**
+  - Command: `syft packages . -o json > sbom.json`
+  - Description: Create Software Bill of Materials
+  - Category: Security
+
+## Task Categories
+
+- **Data Collection**: CVE data ingestion and preprocessing
+- **Analysis**: Statistical and alignment analysis
+- **Machine Learning**: Prediction and model training
+- **Advanced Analysis**: Epsilon calculations and GPU workloads
+- **Model Evaluation**: Information criteria and validation
+- **Testing**: Unit, integration, and system tests
+- **Quality**: Code quality and formatting tools
+- **Containers**: Docker builds and management
+- **Security**: Vulnerability scanning and analysis
+- **Kubernetes**: K8s deployment and management
+- **Workflows**: Argo workflows for batch processing
+- **Infrastructure**: Terraform IaC operations
+
+## Quick Start Workflow
+
+1. Install dependencies: `pip install -e ".[dev]"`
+2. Run tests: `pytest tests/ -v`
+3. Ingest data: `cve-matter ingest --output data/cve_data.json`
+4. Run analysis: `cve-matter align --input data/cve_data.json`
+5. Build container: `docker build --target cpu -t cve-matter-analysis:cpu .`
+
+## Notes
+
+- All tasks follow defensive blue-team security principles
+- No offensive capabilities or cryptographic breaking included
+- GPU tasks require CUDA-enabled hardware
+- Kubernetes tasks require configured cluster access
+- Terraform tasks require GCP credentials

.github/workflows/ci.yml

@@ -0,0 +1,131 @@
+name: CI
+
+on:
+  push:
+    branches: [ main, develop ]
+  pull_request:
+    branches: [ main, develop ]
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    name: Test Python ${{ matrix.python-version }}
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    strategy:
+      matrix:
+        python-version: ["3.11"]
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Cache pip packages
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-${{ hashFiles('**/pyproject.toml') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+
+      - name: Lint with ruff
+        run: |
+          ruff check cve_matter/ tests/
+
+      - name: Format check with black
+        run: |
+          black --check cve_matter/ tests/
+
+      - name: Type check with mypy
+        run: |
+          mypy cve_matter/ || true
+
+      - name: Run tests with pytest
+        run: |
+          pytest tests/ -v --cov=cve_matter --cov-report=xml --cov-report=term
+
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v4
+        with:
+          file: ./coverage.xml
+          flags: unittests
+          name: codecov-umbrella
+          fail_ci_if_error: false
+
+  build-docker:
+    name: Build Docker Images
+    runs-on: ubuntu-latest
+    needs: test
+    permissions:
+      contents: read
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build CPU image
+        uses: docker/build-push-action@v5
+        with:
+          context: .
+          target: cpu
+          tags: cve-matter-analysis:cpu
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+          push: false
+
+      - name: Build CUDA image
+        uses: docker/build-push-action@v5
+        with:
+          context: .
+          target: cuda
+          tags: cve-matter-analysis:cuda
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+          push: false
+
+  integration-test:
+    name: Integration Tests
+    runs-on: ubuntu-latest
+    needs: test
+    permissions:
+      contents: read
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python 3.11
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+
+      - name: Test CLI commands
+        run: |
+          cve-matter --version
+          cve-matter --help
+          cve-matter ingest --help
+          cve-matter align --help
+          cve-matter arbiter --help
+          cve-matter refract --help
+          cve-matter evidence --help

.github/workflows/codeql.yml

@@ -0,0 +1,42 @@
+name: CodeQL Security Analysis
+
+on:
+  push:
+    branches: [ main, develop ]
+  pull_request:
+    branches: [ main, develop ]
+  schedule:
+    - cron: '0 0 * * 1'  # Weekly on Monday
+
+permissions:
+  actions: read
+  contents: read
+  security-events: write
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [ 'python' ]
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Initialize CodeQL
+        uses: github/codeql-action/init@v3
+        with:
+          languages: ${{ matrix.language }}
+          queries: +security-and-quality
+
+      - name: Autobuild
+        uses: github/codeql-action/autobuild@v3
+
+      - name: Perform CodeQL Analysis
+        uses: github/codeql-action/analyze@v3
+        with:
+          category: "/language:${{ matrix.language }}"