Skip to content

test(plugins): add result normalization contract tests for all bundled parsers (#231)#292

Open
vikas-6 wants to merge 1 commit into
utksh1:mainfrom
vikas-6:parser-contract
Open

test(plugins): add result normalization contract tests for all bundled parsers (#231)#292
vikas-6 wants to merge 1 commit into
utksh1:mainfrom
vikas-6:parser-contract

Conversation

@vikas-6
Copy link
Copy Markdown

@vikas-6 vikas-6 commented May 24, 2026

📝 Description

This PR implements a centralized integration contract test suite validating that every single one of the 60 custom bundled parsers complies strictly with the required parser interface and normalization constraints (resolves #231).


🚀 Key Features & Changes

1. Centralized Contract Verification Suite (testing/backend/integration/test_parser_output_contract.py)

  • Dynamic Parser Discovery & Loading: Scans the plugins/ directory, dynamically imports parser.py modules, and verifies the required parse(output: str) entrypoint exists and is callable.
  • Empty Shape Resilience: Asserts that empty strings or empty structures passed to the normalization logic are handled gracefully without raising unhandled server crashes.

2. Strict Input & Structure Guardrails (Negative Tests)

  • Malformed Shape Fallbacks: Ensures that incomplete or malformed finding records default safely (falling back to the capitalized plugin category, default titles, empty string remediations, and empty metadata dictionaries).
  • Valid Severity Mappings: Asserts that standard parser severities (critical, high, medium, moderate, warning, low, info, informational, error) are accurately translated.
  • Unsafe/Unknown Severity Sanitization: Verifies that unknown or malicious severities (e.g. critical-critical, severe, malicious, None, integers, or blank strings) are safely defaulted to info to protect database inserts.
  • Shape Type Filtering: Verifies that any malformed elements in the findings list that are not represented as dictionaries (e.g. strings, integers, or lists) are strictly filtered out and rejected.

🧪 Verification & Formatting Hygiene

  • Formatting Hygiene: The new test file has been cleaned of all trailing whitespaces. Verified locally with git diff --check, resulting in a perfectly clean pass.
  • Test Session Success: All 60 custom parsers dynamically loaded and passed the strict contract validations successfully:
    $ pytest testing/backend -k test_parser_contract_compliance
    testing/backend/integration/test_parser_output_contract.py ............................................................ [100%]
    ================ 60 passed in 0.15s =================
  • Test Baseline: Confirmed that all 336 existing unit and integration tests continue to pass with absolute success.

📋 Checklist

  • Dynamic discovery of all 60 bundled custom parsers implemented.
  • Required parse callable entrypoint verified across all parsers.
  • Normalization layer resilience to empty/null shapes validated.
  • Severity mappings and malformed fallbacks verified.
  • Explicit negative tests for unknown severities and malformed shapes implemented.
  • Staged, committed, and pushed clean code without unrelated lockfile/doc churn.

…d parsers

- Implemented dynamic parser discovery to locate and verify all custom parser.py modules.
- Added callable check for the required parse entrypoint across all 60 parsers.
- Implemented robust normalization verification for empty, incomplete, or malformed inputs.
- Created explicit negative tests to verify safe severity sanitization and type-based shape filtering.
@vikas-6
Copy link
Copy Markdown
Author

vikas-6 commented May 24, 2026

Hi @utksh1!

I have resolved this issue by implementing a centralized integration contract test suite that dynamically loads and verifies all 60 bundled custom parsers against the backend normalization constraints.

Here is the PR: #292 (or your PR number once created!)

Please review and merge it. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[PLUGINS] Add plugin result normalization contract tests for all bundled parsers

1 participant