DL-18: Manage Change Log (Event Sourcing)

[spuentesp]

Category: data-layer | Epic: 0 | Priority: high

Summary

Implement append-only change log in MongoDB for recording all changes to canonical
data. Provides audit trail and enables historical state reconstruction. All Neo4j
write operations automatically log to this collection. Supports filtering by subject,
time range, change type, and author.

Acceptance Criteria

• mongodb_log_change creates append-only change record
• All Neo4j write operations auto-capture to change_log
• Change records include subject_type, subject_id, change_type
• Change records include old_value and new_value
• Change records include author and authority level
• Change records include evidence_type and evidence_id
• Related changes grouped by transaction_id
• mongodb_get_change_history returns paginated history for subject
• mongodb_get_changes_by_time returns changes in time range
• mongodb_get_changes_by_author returns changes by specific author
• Indexes on subject_id+timestamp, timestamp, transaction_id, author
• Collection is append-only (no updates or deletes)
• Unit tests achieve >= 80% coverage

Dependencies

This use case depends on:

• DL-1
• DL-2
• DL-3
• DL-4

Blocks

This use case blocks:

• DL-19
• M-32

Implementation

Layer: 1

Files to create:

• packages/data-layer/src/monitor_data/schemas/change_log.py
• packages/data-layer/tests/test_tools/test_change_log_tools.py
  Files to modify:
• packages/data-layer/src/monitor_data/tools/mongodb_tools.py
• packages/data-layer/src/monitor_data/middleware/auth.py
• packages/data-layer/src/monitor_data/schemas/__init__.py

MONGODB Operations:

• mongodb_log_change (authority: *)
• mongodb_get_change_history (authority: *)
• mongodb_get_changes_by_time (authority: *)
• mongodb_get_changes_by_author (authority: *)
• mongodb_get_transaction_changes (authority: *)

Notes:

• This is the foundation for event sourcing pattern
• All write operations MUST call log_change() before returning
• Middleware integration ensures no writes escape logging
• Field_path uses dot notation: "properties.hp", "state_tags"
• For complex changes, store full state_before/state_after snapshots
• Keep change records forever for compliance and debugging

Testing Requirements

Minimum coverage: 80%

Unit tests:

• test_log_change_success: creates change record
• test_log_change_fields: all required fields present
• test_get_history_by_subject: returns changes for entity
• test_get_history_pagination: limit/offset works
• test_get_changes_by_time: time range filter works
• ... and 3 more

Integration tests:

• test_neo4j_write_logs_change: entity create logs automatically
• test_update_logs_old_and_new: both values captured
• test_multiple_changes_same_transaction: atomic grouping

References

Documentation:

• docs/DATA_LAYER_USE_CASES.md#dl-18
• docs/architecture/DATABASE_INTEGRATION.md
  Code:
• packages/data-layer/src/monitor_data/db/mongodb.py
• packages/data-layer/src/monitor_data/middleware/auth.py

---

Generated from /home/sebas/monitor2/docs/use-cases/data-layer/DL-18.yml