v0.3.0 - Observability, async scan persistence, CIS fixes and validation framework#170
Merged
Conversation
… compliance mappings, docs
* feat: add sentinel/ingest.py — Log Analytics ingestion via HMAC-SHA256 * feat: add sentinel/__init__.py * feat: add KQL rule — HIGH severity finding detected * feat: add KQL rule — misconfiguration wave detection * feat: add KQL rule — new resource type critical detection * Delete sentinel/rules directory * Create rules * Delete sentinel/rules * Add KQL rule for high severity findings * Add Misconfiguration Wave detection rule * Add KQL rule for persistent misconfiguration detection * Add KQL rule for new critical resource types This rule identifies new resource types with critical findings that have occurred in the last 24 hours, excluding known types from the last 30 days. * Add script to generate test findings in JSON format This script generates test findings related to security compliance and saves them in a JSON file. * Add Sentinel integration test plan and results Added a comprehensive test plan for Sentinel integration, detailing test objectives, results, and acceptance criteria for various KQL rules and data ingestion. * docs: add sentinel integration setup guide Added a comprehensive setup guide for integrating Sentinel with Azure, covering prerequisites, workspace creation, activation, environment variable setup, ingestion, log verification, KQL rules deployment, and incident verification.
* Add az_net_003.py to check NSG rules for port 443 This script detects Network Security Groups (NSGs) with unrestricted inbound access on port 443 and provides remediation guidance. * Add AZ-NET-004 rule for empty NSG detection This script detects Network Security Groups (NSGs) that have no custom security rules configured, providing details for remediation. * Add AZ-NET-005 rule for DDoS protection check This script detects virtual networks in Azure that do not have DDoS protection enabled and provides remediation steps. * feat: add rule AZ-NET-006 — public IP unassociated with any resource This rule detects public IP addresses that are not associated with any resource, providing details for remediation. * feat: add rule AZ-NET-007 — Application Gateway without WAF enabled This rule detects Application Gateways that do not have WAF enabled, logging findings and providing remediation steps. * feat: add rule AZ-NET-008 — load balancer with no backend pool This rule detects load balancers in Azure that are not configured with a backend pool, indicating potential misconfiguration or unnecessary costs. * feat: add rule AZ-NET-009 — VPN gateway using outdated IKE version This script detects VPN gateways using the outdated IKEv1 protocol and provides remediation steps to migrate to IKEv2. * feat: add rule AZ-NET-010 — subnet with no NSG attached This script detects subnets in Azure that do not have a Network Security Group (NSG) attached, logging findings and providing remediation guidance. * feat: add playbook fix_az_net_003.sh This script updates the NSG rule to restrict inbound traffic on port 443 to a specified IP range. * feat: add playbook fix_az_net_004.sh This script adds a default deny-all inbound rule to a specified NSG. * feat: add playbook fix_az_net_005.sh This script enables DDoS protection on a specified virtual network in Azure. It checks for required parameters and provides usage instructions if they are missing. * feat: add playbook fix_az_net_006.sh This script deletes unassociated public IP addresses in Azure. * feat: add playbook fix_az_net_007.sh This script enables WAF on an Application Gateway, ensuring compliance with the AZ-NET-007 rule. * feat: add playbook fix_az_net_008.sh Script to remediate AZ-NET-008 by deleting empty load balancers. * feat:add script to update VPN connection to IKEv2 This script updates a VPN connection to use IKEv2, ensuring compliance with the AZ-NET-009 rule. * feat: add playbook fix_az_net_010.sh This script attaches a specified network security group to a given subnet in a virtual network, ensuring compliance with the AZ-NET-010 rule. * Clarify description and add note for public-facing services Updated the description to clarify the risk of exposing port 443 and added a note regarding public-facing services. * Change severity level from MEDIUM to HIGH * fix: AZ-NET-005 severity changed to LOW — DDoS Standard high cost on small subscriptions * Add note about NetworkManagementClient usage Added a note regarding the creation of NetworkManagementClient directly and suggested a follow-up for consistency. * Add note about NetworkManagementClient usage Added a note regarding the use of NetworkManagementClient and suggested a follow-up for consistency. * Add additional security controls to CIS Azure benchmark * Refine control descriptions in nist_csf.json Updated descriptions for various controls to enhance clarity and specificity regarding remote access management, data protection, and security measures. * fix: add AZ-NET-003 to AZ-NET-010 to ISO27001 compliance framework Updated descriptions for various controls to clarify compliance requirements and improve security guidance. --------- Co-authored-by: Vishnu Ajith <86302373+Vishnu2707@users.noreply.github.com>
* feat: add rule AZ-STOR-003 storage lifecycle policy check * feat: add rule AZ-STOR-003 storage lifecycle policy check
* docs: add SOC 2 Type II compliance framework mapping for all 20 rules Added SOC 2 Type II framework with detailed controls for security measures and compliance requirements. * feat: add soc2 to FRAMEWORK_FILE_MAP in finding.py add soc2.json to FRAMEWORK_FILE_MAP in finding.py * feat: add soc2 to SUPPORTED_FRAMEWORKS in compliance.py Added 'soc2' to the list of supported compliance frameworks. * Add SOC 2 controls for data protection and management
* refactor: add get_virtual_networks() and get_public_ip_addresses() to AzureClient * Refactor DDoS protection check to use azure_client * refactor: AZ-NET-006 now uses azure_client.get_public_ip_addresses()
- Python syntax check on all rule files - Rule structure validation (RULE_ID, SEVERITY, FRAMEWORKS) + RULE_ID uniqueness - Hardcoded credential scan - Playbook existence + bash syntax check for every rule - Compliance JSON validation for all four framework files (inc. soc2.json) - API syntax check - Compliance vs rule cross-reference check - CI summary step with per-check pass/fail table (if: always) - Fix duplicate DESCRIPTION assignment in az_net_003.py - Add pyyaml to requirements.txt for local YAML validation - Add docs/ci-pipeline.md with local run commands and design rationale - Update CI_PIPELINE_GUIDE.md with final PR description Closes #30
) * feat: implement asynchronous scan execution with background worker * chore: async scan architecture with 100% verified test suite * feat: complete transition to async scan architecture with verified E2E suite and docs * fix: addressed the requested changes * fix: address security and architecture issues in async scan processing - Sanitize worker error messages to prevent sensitive exception details from being exposed through the public API - Revert unrelated schema and search_path changes to maintain compatibility with existing public-schema deployments - Add column to preserve as the original queue timestamp - Improve migration logic to correctly backfill historical scans and repair incorrect statuses - Update worker tests to reflect generic error handling and the new atomic scan-claiming workflow
* feat(scanner): add AZ-NET-015 public DNS zone enumeration rule - Add scanner/rules/az_net_015.py to detect public DNS zones - Add get_dns_zones() to azure_client.py using DnsManagementClient - Add playbooks/cli/fix_az_net_015.sh remediation script - Add azure-mgmt-dns==8.0.0 to requirements.txt - Update all 4 compliance framework JSONs with AZ-NET-015 mappings * feat(scanner): add get_dns_record_sets to AzureClient * feat(scanner): rewrite AZ-NET-015 to detect RFC1918 IPs and internal hostnames in public DNS zones * feat(compliance): add AZ-NET-015 mappings to all four framework JSON files
* Update finding.py total_findings > 0 skips any scan where 0 issues were found. So after you fix all misconfigs and run a clean scan, the dashboard ignores it and keeps showing the old dirty scan. status = 'completed' picks the latest scan that finished successfully, whether it found 0 things or 100 things. * Update finding.py * Update ci.yml * Create test_clean_scan.py all confirmed passing locally. Zero real DB or Azure calls pure mock. * Update ci.yml * Update finding.py
* fix: skip DB migrations when DATABASE_URL is unset and add full pytest step to CI * fix: add PYTEST to CI Summary env section
* feat: integrate azure offensive skills with dynamic grounding and metadata standardization * feat(ai): harden RAG grounding logic and stabilize vectorstore rebuild flow * fix(ci): decouple app from database during test collection and execution * fix(ci): prevent side-effect create_app() call during pytest collection * fix(tests): mock database manager in jwt config integration tests --------- Co-authored-by: Vishnu Ajith <86302373+Vishnu2707@users.noreply.github.com>
… demo mode (#144) * Update public access paths and database migration logic Refactor public access logic and improve database migration handling. * Add tests for JWT authentication middleware This file contains tests for JWT authentication middleware, including both production and demo modes. It tests various endpoints for JWT requirements and public access. * Enhance API reference with authentication details Added authentication details and public demo mode information to the API reference. * Add auth tests to regression test suite * Remove DATABASE_URL from CI workflow Removed DATABASE_URL environment variable from rule regression tests. * Update ci.yml * fix: mock DatabaseManager in test_auth.py fixtures to prevent CI DB connection
…ics, readiness probe (#167) Implements issue #159 (INFRA 6). - api/observability.py: shared module with JSON logging (python-json-logger), optional Sentry init (only when SENTRY_DSN set), request-ID middleware, Prometheus metrics, and the /metrics endpoint. - api/app.py: wire observability middleware first; split /health (liveness, no DB) from /ready (DB connectivity → 200/503); echo X-Request-ID; include request_id in JSON error responses and auth failures; /ready + /metrics public. - api/models/finding.py: add DatabaseManager.ping() for the readiness probe. - scanner/worker.py: shared JSON logging, conditional Sentry, scan_id as a structured log field, scan success/failure counters, scan duration, queue depth. - scanner/engine.py: increment per-rule error counter when a rule raises. - scanner/nvd_client.py, api/services/ai_provider.py: record external-call latency. - requirements.txt: add prometheus-client, python-json-logger, sentry-sdk. - tests/test_observability.py: probes, /metrics, request IDs, auth request_id, conditional Sentry, worker metrics. No metric is labelled with request_id, scan_id, subscription_id, resource_id, error_message or user input. Co-authored-by: Shaurya K Sharma <Sharma-SK@ulster.ac.uk>
…compare, Graph pagination (#166) * fix(compliance): correct duplicate CIS control mappings, TLS version compare, Graph pagination - Fix AZ-CMP-001 sharing CIS control 7.2 with unrelated AZ-CMP-002 - De-duplicate 13 rules that shared CIS control_id values across cis_azure_benchmark.json, their rule files, docs, and the website - Replace lexicographic TLS version comparison in az_pqc_001 with a numeric parser so "1.10" is no longer misread as below "1.3" - Follow @odata.nextLink pagination in get_conditional_access_policies instead of dropping all results past the first page * style: apply ruff format to new compliance/pagination tests
Bumps the npm_and_yarn group with 1 update in the / directory: [react-router](https://github.com/remix-run/react-router/tree/HEAD/packages/react-router). Updates `react-router` from 7.15.0 to 7.18.0 - [Release notes](https://github.com/remix-run/react-router/releases) - [Changelog](https://github.com/remix-run/react-router/blob/main/packages/react-router/CHANGELOG.md) - [Commits](https://github.com/remix-run/react-router/commits/react-router@7.18.0/packages/react-router) --- updated-dependencies: - dependency-name: react-router dependency-version: 7.18.0 dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the npm_and_yarn group with 1 update in the /frontend directory: [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite). Updates `vite` from 8.0.14 to 8.0.16 - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v8.0.16/packages/vite) --- updated-dependencies: - dependency-name: vite dependency-version: 8.0.16 dependency-type: direct:development dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Collaborator
|
We have to make the Deploy to Render disabled or only manaul as the old Render deployment will fail cause of the expiration of the DB and that would be the best fit for newer merges in my opinion |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| try: | ||
| context, sources = _context_for(findings_text) | ||
| except VectorStoreNotBuilt as exc: | ||
| return jsonify({"error": str(exc)}), 503 |
| try: | ||
| raw = get_completion(body["provider"], body["api_key"], prompt, model=body.get("model")) | ||
| except ValueError as exc: | ||
| return jsonify({"error": str(exc)}), 400 |
| except ValueError as exc: | ||
| return jsonify({"error": str(exc)}), 400 | ||
| except RuntimeError as exc: | ||
| return jsonify({"error": str(exc)}), 502 |
| return jsonify(scan) | ||
| except Exception as exc: | ||
| logger.error("Failed to get scan status: %s", exc) | ||
| return jsonify({"error": "Database error", "detail": str(exc)}), 500 |
| logger.error("Failed to save scan result: %s", exc, exc_info=True) | ||
| return jsonify({"error": "Database save failed", "detail": str(exc)}), 500 | ||
| logger.error("Failed to create pending scan: %s", exc, exc_info=True) | ||
| return jsonify({"error": "Database error", "detail": str(exc)}), 500 |
ritiksah141
reviewed
Jul 8, 2026
ritiksah141
left a comment
ritiksah141
left a comment
Collaborator
There was a problem hiding this comment.
We need to disable the auto deploy trigger of the Render as the current Deployment has been suspended the Database will never work unless need to create new deployment to new subscription
Member
Author
Thanks @ritiksah141 , i did note this issue was creating a PR for that : ) deploy: I am going to update the trigger in deploy.yml file |
ritiksah141
previously approved these changes
Jul 8, 2026
ritiksah141
approved these changes
Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Merges dev into main for the v0.3.0 release.
Type of change
What's included
Testing
Related issues
Closes #131
Closes #159
Closes #168