Enterprise Integration Lab

Enterprise Integration Lab is a local portfolio project that demonstrates how independent enterprise systems can exchange, normalize, synchronize, and observe business data through an event-driven integration lifecycle.

The project is intentionally industry-neutral. It uses generic customers, agreements, service requests, operational cases, documents, canonical objects, lineage records, and sync logs.

Project Overview

This lab simulates a common enterprise integration pattern:

1. A user submits a service request through an intake portal.
2. The backend creates raw intake and source-system records.
3. Integration events are created for asynchronous processing.
4. A worker synchronizes the request into a downstream operational system.
5. The worker creates or updates an enterprise canonical object.
6. The dashboard shows the full lifecycle, lineage, status history, documents, and sync logs.

The goal is not to build a production product. The goal is to make enterprise architecture concepts concrete, observable, and easy to discuss in a portfolio or demo setting.

What This Demonstrates

• Logical system boundaries using PostgreSQL schemas
• Event-driven lifecycle orchestration
• Separation of raw intake, source-system records, integration events, operational records, canonical objects, and dashboard views
• Idempotent worker processing for duplicate events
• Downstream operational workflow simulation
• Canonical status mapping from operational workflow events
• Document repository metadata linked to business records
• MinIO object storage for uploaded supporting documents
• Reference data for controlled portal selections
• Lineage and auditability across system boundaries
• A simple dashboard for explaining the integration lifecycle

Architecture Overview

flowchart LR
    A["Intake Portal"] --> B["Source System"]
    A --> D["MinIO Document Repository"]
    B --> C["Integration Events"]
    C --> W["Worker"]
    W --> O["Operational System"]
    W --> K["Canonical Layer"]
    O --> C
    O --> K
    B --> L["Lineage / Audit Records"]
    O --> L
    K --> L
    D --> L
    B --> X["Dashboard"]
    C --> X
    O --> X
    K --> X
    D --> X
    L --> X

Loading

Containerized Runtime Architecture

The project runs as a containerized enterprise simulation platform with isolated services for the frontend portal and dashboard, backend orchestration, asynchronous worker processing, relational enterprise data modeling, and document repository storage. The worker continuously polls integration events and synchronizes operational and canonical lifecycle state.

System Boundaries

Layer	Responsibility
reference	Generic master/reference data used by the intake form
intake	Raw user-submitted requests
source_system	First internal system-of-record request
document_repo	Document metadata and object-storage references
integration	Events and sync logs
operational	Downstream operational case records and status history
canonical	Enterprise-normalized business objects and lineage
dashboard	Reserved schema for future read models

Event-Driven Enterprise Lifecycle

Intake submissions generate source requests, integration events trigger asynchronous orchestration, worker processing creates downstream operational workflow state, and canonical lifecycle visibility is synchronized continuously. Auditability and lineage preserve traceability across the full flow. This project demonstrates enterprise lifecycle orchestration rather than simple CRUD processing.

The minimal lifecycle is deterministic:

1. POST /intake/submissions
2. Create intake.submissions
3. Create source_system.service_requests
4. Optionally upload a supporting document to MinIO and write document_repo.documents
5. Create a pending RequestCreated event in integration.events
6. Worker creates or reuses operational.operation_cases
7. Worker creates or reuses canonical.business_objects
8. Worker writes canonical.lineage_records
9. Worker writes integration.sync_logs
10. Worker marks the event as completed

For operational workflow changes:

1. Dashboard or API patches an operational case status.
2. Backend updates operational.operation_cases.
3. Backend writes operational.case_status_history.
4. Backend creates an OperationalCaseStatusChanged event.
5. Worker maps operational status back to canonical lifecycle status.
6. Worker writes a sync log and completes the event.

Semantic Mapping and Enterprise Lineage

Local systems maintain their own semantic models, while the canonical layer creates enterprise-wide semantic consistency. Semantic normalization maps local objects into a shared enterprise business model, and lineage records preserve relationship history and lifecycle traceability. The canonical layer behaves like a shared enterprise semantic coordinate system across otherwise independent operational systems.

Canonical Data Model Explanation

The canonical layer is the enterprise-normalized view of the business object. It is not the raw portal submission and it is not the downstream operational case.

Canonical lifecycle mapping:

Operational Status	Canonical Status
open	active
in_progress	in_progress
completed	completed
rejected	rejected

This keeps downstream operational activity synchronized with the enterprise-level representation without making the operational system the only place to understand the lifecycle.

Document Repository Flow

The intake form supports one optional supporting document.

Document handling:

1. Backend validates extension and content type.
2. Backend enforces a 10MB default size limit.
3. File bytes are stored in MinIO bucket enterprise-documents.
4. Metadata is stored in document_repo.documents.
5. Metadata links to both intake.submissions and source_system.service_requests.
6. Dashboard displays metadata only. It does not preview or download files.

Allowed file types in the MVP:

• PDF
• TXT
• CSV
• PNG
• JPEG

Validation checks filename extension and browser-provided content type. It does not inspect magic bytes.

Operational Workflow and Lifecycle Synchronization

Operational workflow state changes generate synchronization events, worker processing propagates lifecycle updates asynchronously, and canonical lifecycle visibility reflects downstream operational execution state. Synchronization history is recorded for auditability and observability. The operational workflow remains locally independent while enterprise lifecycle visibility stays synchronized centrally.

Operational cases simulate downstream work records. New cases start as open.

Allowed transitions:

• open -> in_progress
• open -> rejected
• in_progress -> completed
• in_progress -> rejected

Terminal statuses:

• completed
• rejected

Each transition records:

• previous status
• new status
• changed by
• change reason
• changed timestamp

Lineage / Auditability

The project demonstrates multiple audit paths:

• integration.events records business events.
• integration.sync_logs records worker synchronization outcomes.
• canonical.lineage_records maps source request to operational case to canonical object.
• operational.case_status_history records human workflow transitions.
• document_repo.documents records document metadata and object-storage references.

Together, these tables make the lifecycle explainable without relying on dashboard-only state.

Dashboard Explainability

The dashboard is intended as the main demo surface. It shows:

• submission list
• lifecycle stages
• schema role legend
• event flow explanation
• status meaning legend
• document metadata
• operational workflow action panel
• status history
• sync logs
• lineage records
• events

The dashboard includes a controlled operational workflow action. It is not a full administration console.

Getting Started

Create a local environment file:

cp .env.example .env

Start the stack:

docker compose up --build

Open:

• Dashboard: http://localhost:8080
• Backend API docs: http://localhost:8000/docs
• Backend health check: http://localhost:8000/health
• MinIO console: http://localhost:9001

From another machine on the same LAN:

http://<vm-ip-address>:8080

Clean Reset / Reseed

Use this before screenshots or portfolio recording to remove local test data and restore generic demo records:

docker compose down -v
docker compose up --build

The reset removes local PostgreSQL, Redis, and MinIO volumes. PostgreSQL then runs:

• docker/postgres/init/01_schemas.sql
• docker/postgres/init/02_demo_seed.sql

The demo seed includes only generic example.test users, generic request descriptions, and deterministic demo lifecycle records. It does not seed uploaded document objects.

Demo Workflow

1. Open the dashboard.
2. Review the seeded demo submissions.
3. Submit a new request from the portal form.
4. Optionally attach a small supporting document.
5. Watch the worker create operational and canonical records.
6. Select an operational case.
7. Move it from open to in_progress.
8. Move it to completed or rejected.
9. Review status history, events, sync logs, and lineage records.

API Examples

Create a JSON submission:

curl -s -X POST http://localhost:8000/intake/submissions \
  -H "Content-Type: application/json" \
  -d '{
    "intake_method": "manual_form",
    "requester_name": "Demo User",
    "requester_email": "demo.user@example.test",
    "request_description": "Create a generic service request.",
    "request_type": "GENERAL_INQUIRY",
    "priority": "medium",
    "assigned_team": "OPERATIONS_TEAM"
  }' | jq

Update operational status:

curl -s -X PATCH "http://localhost:8000/operational/cases/<operation_case_id>/status" \
  -H "Content-Type: application/json" \
  -d '{
    "new_status": "in_progress",
    "changed_by": "dashboard-user",
    "change_reason": "Start downstream processing"
  }' | jq

Inspect lifecycle records:

docker compose exec postgres psql -U enterprise_lab -d enterprise_lab -c "SELECT submission_id, requester_name, submission_status FROM intake.submissions ORDER BY created_at DESC LIMIT 5;"
docker compose exec postgres psql -U enterprise_lab -d enterprise_lab -c "SELECT request_number, lifecycle_status FROM source_system.service_requests ORDER BY created_at DESC LIMIT 5;"
docker compose exec postgres psql -U enterprise_lab -d enterprise_lab -c "SELECT event_type, source_system, event_status FROM integration.events ORDER BY created_at DESC LIMIT 10;"
docker compose exec postgres psql -U enterprise_lab -d enterprise_lab -c "SELECT external_reference, processing_status FROM operational.operation_cases ORDER BY updated_at DESC LIMIT 5;"
docker compose exec postgres psql -U enterprise_lab -d enterprise_lab -c "SELECT object_type, canonical_category, lifecycle_status FROM canonical.business_objects ORDER BY updated_at DESC LIMIT 5;"

Recommended Screenshots

For a portfolio page, capture:

• Dashboard overview with lifecycle legend visible
• Submit Service Request form with reference data dropdowns
• Selected lifecycle showing intake, source, operational, canonical, and document metadata
• Operational Workflow panel with status history
• Sync Logs and Lineage Records tables
• Backend /docs showing API endpoints

Project Structure

.
├── backend
│   ├── Dockerfile
│   ├── app
│   │   ├── core
│   │   │   ├── __init__.py
│   │   │   └── config.py
│   │   ├── services
│   │   │   ├── __init__.py
│   │   │   ├── dashboard.py
│   │   │   ├── documents.py
│   │   │   ├── intake.py
│   │   │   ├── integration.py
│   │   │   ├── operational.py
│   │   │   ├── reference_data.py
│   │   │   └── source_system.py
│   │   ├── __init__.py
│   │   ├── db.py
│   │   ├── main.py
│   │   └── models.py
│   └── requirements.txt
├── docker
│   └── postgres
│       └── init
│           ├── 01_schemas.sql
│           └── 02_demo_seed.sql
├── docs
│   ├── assets
│   │   ├── 1-containerized-runtime-architecture.png
│   │   ├── 2-event-driven-lifecycle-flow.png
│   │   ├── 3-semantic-mapping-and-lineage-model.png
│   │   └── 4-operational-workflow-and-lifecycle-state-synchronization.png
│   ├── 00_project_brief.md
│   ├── 01_architecture_overview.md
│   ├── 02_system_modules.md
│   ├── 03_business_object_model.md
│   ├── 04_data_flow_and_lifecycle.md
│   ├── 05_database_schema_design.md
│   └── README.md
├── frontend
│   ├── Dockerfile
│   ├── index.html
│   └── nginx.conf
├── worker
│   ├── Dockerfile
│   ├── app
│   │   ├── core
│   │   │   ├── __init__.py
│   │   │   └── config.py
│   │   ├── __init__.py
│   │   ├── db.py
│   │   └── main.py
│   └── requirements.txt
├── .env.example
├── .gitignore
└── docker-compose.yml

Environment Setup

Only .env.example should be committed. Local .env files are ignored by Git.

The included values are local demo defaults:

• PostgreSQL user/password
• MinIO user/password
• local service ports
• document upload size limit

Change them for local experimentation, but do not use this project as-is for public internet deployment.

Security Notes

This is a local demo project.

• No authentication
• No role-based authorization
• No TLS termination
• No production secret management
• No malware scanning for uploads
• MVP file validation only
• MinIO objects are local demo objects
• Uploaded files are not previewed or downloaded through the dashboard

Current Scope

Implemented:

• Docker Compose environment
• FastAPI backend
• PostgreSQL schemas for system boundaries
• Redis container
• MinIO document object storage
• Static frontend dashboard behind nginx
• Reference data APIs
• Intake portal form
• Optional document upload
• Event-driven worker
• Operational case creation
• Operational workflow simulation
• Canonical object synchronization
• Lineage records
• Sync logs
• Status history audit trail
• Generic demo seed data

Not implemented:

• Authentication
• Role-based permissions
• AI normalization
• Document preview/download
• Complex retries
• Production observability stack
• Public internet deployment hardening

Not Production Ready

This repository is designed for local architecture demonstration and portfolio discussion. It is not production ready and should not be deployed publicly without authentication, authorization, secret management, TLS, upload scanning, operational monitoring, and a formal migration strategy.