Skip to content

Architecture

Manuel Luján edited this page Nov 6, 2026 · 1 revision

Architecture

LineageIQ has three stages: discovery reads the tenant, the build turns what it found into a graph and explorer payloads, and the web server serves that graph to people (web UI), scripts (REST API) and AI agents (MCP). This page is a summary; the README "How It Works" section and the guides in docs/ have the details.

flowchart LR
    APIS["Fabric REST API and Power BI Scanner API"] --> LOCAL["Local scripts in backend/scripts"]
    APIS --> NB["Fabric notebooks in a Data Pipeline"]
    LOCAL --> FILES["Local CSV and JSON artefacts"]
    NB --> FAB["Warehouse bronze tables and OneLake Files"]
    FILES -->|optional publish| FAB
    FILES --> WEB["lineage_web_server.py"]
    FAB -->|live reads with local fallback| WEB
    WEB --> UI["Web UI"]
    WEB --> REST["REST API (/api)"]
    WEB --> MCPHTTP["mcp_http_server.py behind /mcp (Streamable HTTP)"]
    MCPHTTP --> REST
    REMOTE["mcp_server_remote.py (stdio)"] --> REST
    FILES --> MCPLOCAL["mcp_server.py (stdio)"]
Loading

1. Discovery

There are two ways to run discovery. Both produce the same shape of data and feed the same viewer.

  • Locally, with the numbered PowerShell and Python scripts in backend/scripts/, signed in through the Azure CLI or the Fabric CLI. The phases cover workspaces, items, item definitions (pipelines, notebooks, semantic model TMDL, reports), executions, deployment pipelines, object-level lineage, SQL endpoint and column catalogues, notebook source, Git connections, role assignments, Fabric connections and the governance Scanner API pass. 99_refresh_lineage_project.ps1 runs them in dependency order, in one of three modes: local-artifacts (no API calls), read-only-inventory or full-read-only. See Phased execution system.
  • Inside Fabric, with the notebooks in fabric_notebooks/ chained in a Data Pipeline on your own capacity. They sign in as a service principal and write dbo.bronze_* tables to a Warehouse and the explorer JSON files to a Lakehouse's Files/generated/ folder. See Fabric-native discovery setup.

Discovery is read-only against the scanned tenant. Lineage is extracted from definitions and code with regular expressions and structured JSON traversal, and object types are confirmed against SQL endpoint catalogues where they can be reached.

2. Lineage build

Build steps assemble the dependency graph, the reverse-impact index and the explorer payloads:

  • 11_build_lineage_explorer_fast.py writes lineage-explorer.json, the main explorer graph.
  • 17_build_object_impact_index_fast.py writes object-impact.json, reverse impact by source object.
  • build_graph_map_index.py writes the item-level and object-level Graph Map indexes.
  • Notebook parsing produces notebook-super-lineage.json.

Some build steps also exist as T-SQL stored procedures in backend/scripts/warehouse_sql/, reading dbo.bronze_* and writing dbo.silver_* tables.

The graph model

  • Nodes are workspaces, items and source objects (tables, views, stored procedures, files). Each node has a readable identity, Workspace / Item Type / Item Name; Fabric GUIDs are kept as metadata.
  • Edges have a relationship type (reads_from, writes_to, executes, refreshes, depends_on, uses_model, serves_report, contains) and a confidence level (confirmed, likely, unknown).
  • Lineage stops at object level. Column catalogues and profiles exist, but they are not lineage edges.

3. Storage and publishing

Data Local path Fabric destination
Catalogue and lineage tables index/*.csv, analysis/ Warehouse dbo.bronze_*
Explorer JSON files frontend/semantic-lineage/data/ Lakehouse Files/generated/

Local results can be pushed to Fabric with 67_upload_bronze_to_warehouse.ps1 and 72_publish_web_data_to_onelake.py, so a web server running elsewhere reads current data without a redeploy. See Publishing explorer data to OneLake and the medallion Lakehouse guide.

4. Serving

backend/scripts/lineage_web_server.py (Python standard library http.server) serves everything from one host and port, 8765 by default:

  • Web UI: vanilla JavaScript with a 2D flow explorer (lineage, reverse impact and planning views), a DAG view laid out with dagre, and a WebGL 3D Graph Map built on 3d-force-graph and Three.js. Data loads in a Web Worker.
  • REST API under /api/*, described by a live OpenAPI 3.0 spec at /api/openapi.json, generated from the route registry in lineage_web/routes.py.
  • MCP at /mcp: a reverse proxy to mcp_http_server.py, which the web server starts as a subprocess on 127.0.0.1 (port + 1 by default). Two stdio servers, mcp_server.py and mcp_server_remote.py, run separately on the agent's side. See MCP tools reference.

Live reads with fallback:

  • With a Warehouse configured, the catalogue routes query it (with a 30-second cache per table); otherwise, or when a query fails, they read the local CSV files.
  • With a Lakehouse configured, the five explorer JSON files are downloaded from OneLake and cached for five minutes; otherwise, or when a download fails, the server serves a cached or local copy.

5. Scans from the web

Scans can be started from the UI, the REST API or the MCP scan tools. A running scan queues further requests (up to 20). The scan mode is set in config/admin_settings.json:

  • fabric_pipeline (the default) triggers, polls and cancels an existing Fabric Data Pipeline through the Job Scheduler API, so the web host needs no PowerShell or Azure CLI.
  • local runs the PowerShell scripts on the server machine and needs pwsh and a valid az session.

6. Access control

  • People sign in with Microsoft Entra ID through the device-code flow (MSAL). Sessions are kept in memory and expire after 8 hours idle or 24 hours in total.
  • Automation uses admin-issued API keys, stored only as hashes, with read_only or full scope.
  • Admin routes check an allowlist in config/admin_settings.json.
  • /api/* is rate limited per client IP.
  • LINEAGE_DEMO_MODE=1 opens reads without sign-in, for the synthetic demo data only.

Details: admin panel and user authentication and the security policy.

Clone this wiki locally