Repository navigation
Architecture
LineageIQ has three stages: discovery reads the tenant, the build turns what it found into a graph and explorer payloads, and the web server serves that graph to people (web UI), scripts (REST API) and AI agents (MCP). This page is a summary; the README "How It Works" section and the guides in docs/ have the details.
flowchart LR
APIS["Fabric REST API and Power BI Scanner API"] --> LOCAL["Local scripts in backend/scripts"]
APIS --> NB["Fabric notebooks in a Data Pipeline"]
LOCAL --> FILES["Local CSV and JSON artefacts"]
NB --> FAB["Warehouse bronze tables and OneLake Files"]
FILES -->|optional publish| FAB
FILES --> WEB["lineage_web_server.py"]
FAB -->|live reads with local fallback| WEB
WEB --> UI["Web UI"]
WEB --> REST["REST API (/api)"]
WEB --> MCPHTTP["mcp_http_server.py behind /mcp (Streamable HTTP)"]
MCPHTTP --> REST
REMOTE["mcp_server_remote.py (stdio)"] --> REST
FILES --> MCPLOCAL["mcp_server.py (stdio)"]
There are two ways to run discovery. Both produce the same shape of data and feed the same viewer.
-
Locally, with the numbered PowerShell and Python scripts in
backend/scripts/, signed in through the Azure CLI or the Fabric CLI. The phases cover workspaces, items, item definitions (pipelines, notebooks, semantic model TMDL, reports), executions, deployment pipelines, object-level lineage, SQL endpoint and column catalogues, notebook source, Git connections, role assignments, Fabric connections and the governance Scanner API pass.99_refresh_lineage_project.ps1runs them in dependency order, in one of three modes:local-artifacts(no API calls),read-only-inventoryorfull-read-only. See Phased execution system. -
Inside Fabric, with the notebooks in
fabric_notebooks/chained in a Data Pipeline on your own capacity. They sign in as a service principal and writedbo.bronze_*tables to a Warehouse and the explorer JSON files to a Lakehouse'sFiles/generated/folder. See Fabric-native discovery setup.
Discovery is read-only against the scanned tenant. Lineage is extracted from definitions and code with regular expressions and structured JSON traversal, and object types are confirmed against SQL endpoint catalogues where they can be reached.
Build steps assemble the dependency graph, the reverse-impact index and the explorer payloads:
-
11_build_lineage_explorer_fast.pywriteslineage-explorer.json, the main explorer graph. -
17_build_object_impact_index_fast.pywritesobject-impact.json, reverse impact by source object. -
build_graph_map_index.pywrites the item-level and object-level Graph Map indexes. - Notebook parsing produces
notebook-super-lineage.json.
Some build steps also exist as T-SQL stored procedures in backend/scripts/warehouse_sql/, reading dbo.bronze_* and writing dbo.silver_* tables.
-
Nodes are workspaces, items and source objects (tables, views, stored procedures, files). Each node has a readable identity,
Workspace / Item Type / Item Name; Fabric GUIDs are kept as metadata. -
Edges have a relationship type (
reads_from,writes_to,executes,refreshes,depends_on,uses_model,serves_report,contains) and a confidence level (confirmed,likely,unknown). - Lineage stops at object level. Column catalogues and profiles exist, but they are not lineage edges.
| Data | Local path | Fabric destination |
|---|---|---|
| Catalogue and lineage tables |
index/*.csv, analysis/
|
Warehouse dbo.bronze_*
|
| Explorer JSON files | frontend/semantic-lineage/data/ |
Lakehouse Files/generated/
|
Local results can be pushed to Fabric with 67_upload_bronze_to_warehouse.ps1 and 72_publish_web_data_to_onelake.py, so a web server running elsewhere reads current data without a redeploy. See Publishing explorer data to OneLake and the medallion Lakehouse guide.
backend/scripts/lineage_web_server.py (Python standard library http.server) serves everything from one host and port, 8765 by default:
-
Web UI: vanilla JavaScript with a 2D flow explorer (lineage, reverse impact and planning views), a DAG view laid out with dagre, and a WebGL 3D Graph Map built on
3d-force-graphand Three.js. Data loads in a Web Worker. -
REST API under
/api/*, described by a live OpenAPI 3.0 spec at/api/openapi.json, generated from the route registry inlineage_web/routes.py. -
MCP at
/mcp: a reverse proxy tomcp_http_server.py, which the web server starts as a subprocess on127.0.0.1(port + 1 by default). Two stdio servers,mcp_server.pyandmcp_server_remote.py, run separately on the agent's side. See MCP tools reference.
Live reads with fallback:
- With a Warehouse configured, the catalogue routes query it (with a 30-second cache per table); otherwise, or when a query fails, they read the local CSV files.
- With a Lakehouse configured, the five explorer JSON files are downloaded from OneLake and cached for five minutes; otherwise, or when a download fails, the server serves a cached or local copy.
Scans can be started from the UI, the REST API or the MCP scan tools. A running scan queues further requests (up to 20). The scan mode is set in config/admin_settings.json:
-
fabric_pipeline(the default) triggers, polls and cancels an existing Fabric Data Pipeline through the Job Scheduler API, so the web host needs no PowerShell or Azure CLI. -
localruns the PowerShell scripts on the server machine and needspwshand a validazsession.
- People sign in with Microsoft Entra ID through the device-code flow (MSAL). Sessions are kept in memory and expire after 8 hours idle or 24 hours in total.
- Automation uses admin-issued API keys, stored only as hashes, with
read_onlyorfullscope. - Admin routes check an allowlist in
config/admin_settings.json. -
/api/*is rate limited per client IP. -
LINEAGE_DEMO_MODE=1opens reads without sign-in, for the synthetic demo data only.
Details: admin panel and user authentication and the security policy.