feat(audit): implement append-only audit trail and lifecycle tracking #327

[rajesh-puripanda]

Description

Implements a persistent, append-only audit log system for SecuScan to track scan lifecycle events (created, started, completed, failed, cancelled, deleted).

---

Closes #327

---

Changes

• Backend:
  • Added audit_log table with indexed fields and JSON metadata support.
  • Implemented log_event() helper for lifecycle state machine hooks in executor.py.
  • Added paginated GET /api/v1/audit and memory-efficient streamed GET /api/v1/audit/export.
  • Enforced immutability by omitting update/delete routes.
• Frontend:
  • Created AuditLog page with debounced filtering and AuditTable for metadata inspection.
  • Added navigation and API integration.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

How Has This Been Tested?

• Verified schema migration with existing data.
• Confirmed lifecycle hooks (scan_created to scan_deleted) trigger audit entries.
• Verified non-existence of PUT/PATCH/DELETE endpoints.
• Manual verification of export streaming.
• Unit tests passed via pytest.

Checklist

• My code follows the code style of this project.
• I have performed a self-review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.

Closes #327

[utksh1]

Requesting changes. Append-only audit logging is valuable, but backend-tests are failing and the PR adds SQLAlchemy as a runtime dependency while the app still uses aiosqlite. Please remove unnecessary ORM scaffolding or make it clearly dev-only, fix the failing tests, and add migration/backward-compatibility coverage for existing audit_log tables.

[utksh1]

Thanks for following up. Clarifying the change request so it is actionable:

Why this is blocked:
Requesting changes. Append-only audit logging is valuable, but backend-tests are failing and the PR adds SQLAlchemy as a runtime dependency while the app still uses aiosqlite. Please remove unnecessary ORM scaffolding or make it clearly dev-only, fix the failing tests, and add migration/backward-compatibility coverage for existing audit_log tables.

What to do next:

• Fix the specific issues called out above.
• Push the updated branch and make sure the relevant CI checks pass.
• Reply here when ready for re-review.

[utksh1]

Re-reviewed latest state. The branch is conflicting after recent main changes, so it cannot merge. Please rebase and make sure the audit-log implementation stays aiosqlite-native, keeps migration/backward-compatibility coverage, and does not reintroduce unnecessary ORM/runtime dependencies.

[rajesh-puripanda]

Re-reviewed latest state. The branch is conflicting after recent main changes, so it cannot merge. Please rebase and make sure the audit-log implementation stays aiosqlite-native, keeps migration/backward-compatibility coverage, and does not reintroduce unnecessary ORM/runtime dependencies.

Ready for re-review. Conflicts with upstream/main resolved.

Summary of changes:

• SQLAlchemy moved to requirements-dev.txt only, runtime stays aiosqlite-native

• Append-only contract enforced , delete_task_records no longer deletes from audit_log; logs SCAN_DELETED events instead

• Migration/backward-compat coverage added database.py now migrates legacy audit_log tables (adds target, actor, metadata columns)

• CI checks passing:

  • Ruff lint
  • 537 backend tests pass (only pre-existing waf_detector failure, unrelated)
  • Formatting hygiene

All 24 audit-specific tests pass. No new warnings, no new runtime dependencies.

[utksh1]

Re-reviewed after the latest push. Still blocked: please keep the audit implementation aiosqlite-native, make sure migration/backward-compatibility coverage passes on current main, and avoid reintroducing unnecessary ORM/runtime dependency changes.

[rajesh-puripanda]

Conflicts resolved. Ready for re-review.

[utksh1]

Re-reviewed after the latest push. Still blocked: please keep the audit trail implementation aiosqlite-native, avoid unrelated runtime dependency churn, and add/keep migration compatibility coverage for existing audit_log tables on current main.

[rajesh-puripanda]

@utksh1 I've addressed all your review comments:

1. Rebased against latest main — all merge conflicts resolved
2. Removed SQLAlchemy entirely — no SQLAlchemy in requirements.txt or requirements-dev.txt. Removed the SQLAlchemy model/imports from audit.py — the implementation is pure aiosqlite-native now
3. Kept migration/backward-compatibility coverage — test_audit_log_migration.py tests that old audit_log tables get the new columns added via ALTER TABLE, using only aiosqlite
4. Append-only contract preserved — audit_log rows are retained on task deletion (only findings, reports, and tasks are deleted)
5. SCAN_DELETED events logged — each deletion triggers a log_event call before the task row is removed

All 4 commits rebased cleanly onto current main. Please re-review when you get a chance!

[utksh1]

Re-reviewed the latest push. This is still not mergeable: the branch is behind current main, and backend-tests has not completed on the latest visible run. Please update the branch against current main and get a completed green backend-tests run before requesting review again.