diff --git a/pages/memgraph-zero/memgql/changelog.mdx b/pages/memgraph-zero/memgql/changelog.mdx index 466e9d8f8..d72e6f95d 100644 --- a/pages/memgraph-zero/memgql/changelog.mdx +++ b/pages/memgraph-zero/memgql/changelog.mdx @@ -9,65 +9,58 @@ description: MemGQL release notes ### 🍃 New features & Improvements -- **MongoDB connector.** New `mongodb` connector type brings a document store - into the federation. It is the first backend that is neither SQL nor Cypher: - queries are translated to **MongoDB aggregation pipelines** and executed - server-side. Node labels map to collections and relationship types to their own - collections holding `from`/`to` reference fields, using the same +- **Microsoft Fabric connector.** New `fabric` connector type translates GQL to + T-SQL over a Fabric Warehouse's SQL endpoint — a pure-Rust driver, so there's no + ODBC layer to install. The same connector also reaches Lakehouse SQL analytics + endpoints, SQL databases in Fabric, and mirrored databases, which need no + connector of their own. It uses the same [mapping](/memgraph-zero/memgql/reference#mapping-schema) format as every other - backend — a mapping written for PostgreSQL works unchanged. A single hop - becomes `$lookup`; a **variable-length hop becomes `$graphLookup`**, MongoDB's - native recursive traversal, so `(a)-[:KNOWS*1..3]->(b)` runs on the server — - and unlike the SQL backends the **unbounded** `*` form is supported too. - Filters that compare a property to a literal emit the plain query form so - MongoDB can serve them from an index. **Both reads and writes** are supported, - including `DETACH DELETE`, which clears incident edge documents. Connect with a - standard connection string, so replica sets and MongoDB Atlas - (`mongodb+srv://`) work as-is. `WITH` boundaries, `OPTIONAL MATCH` (including a - predicate inside the optional pattern), `FOR x IN […]`, and the full aggregate - set all run server-side; `UNION` and scalar functions inside `RETURN` are the - notable gaps. MongoDB runs the same cross-backend parity corpus as every other - connector, so its answers are checked against the same golden rows. See the + SQL backend, references tables with three-part `item.schema.table` names — so one + query can join across items in a workspace — and supports **both reads and + writes**. Authenticate with a Microsoft Entra ID access token or a service + principal. Variable-length paths are rejected, since Fabric has no recursive + queries. See the + [Microsoft Fabric connector page](/memgraph-zero/memgql/connect/fabric) for setup. + +- **MongoDB connector.** New `mongodb` connector type brings a document store into + the federation — the first backend that is neither SQL nor Cypher. Queries + translate to **MongoDB aggregation pipelines** and run server-side: a single hop + becomes `$lookup`, a variable-length hop `$graphLookup`, so unlike the SQL + backends the unbounded `*` form works too. Labels map to collections and + relationship types to their own collections, using the same + [mapping](/memgraph-zero/memgql/reference#mapping-schema) format as every other + backend, and **both reads and writes** are supported, including `DETACH DELETE`. + Connect with a standard connection string, so replica sets and MongoDB Atlas work + as-is; `UNION` and scalar functions inside `RETURN` are the notable gaps. See the [MongoDB connector page](/memgraph-zero/memgql/connect/mongodb) for setup. ### ⚠️ Behavior changes -- **A vertex's or edge's `metaFields.id` is now exposed as a property.** - `RETURN n` / `RETURN r` include it, and `n.` reads it by name — - including after a `WITH n` boundary, which previously resolved only when the - id column happened to be called `id`. This aligns relational sources with native-graph ones and - with a graph's Memgraph cache, which stores the id as its merge key. -- **The cache now answers only for properties it holds.** A fragment holds a - label's id and its declared `attributes`; a query reading anything else is a - cache miss and reads the source, so results never differ from the uncached - run. A label declaring no `attributes` still caches its ids and topology, so - traversals and graph algorithms work against it. `SHOW GRAPH CACHES` gained - `hits`, `misses` and `cached_properties` columns — enough to tell a cache that - is serving from one that never engages, and a property-driven miss from a cold - one. See +- **A vertex's or edge's `metaFields.id` is now exposed as a property.** `RETURN n` + / `RETURN r` include it, and `n.` reads it by name — including after a + `WITH n` boundary, which previously worked only when the column was called `id`. +- **The cache answers only for properties it holds.** A fragment holds a label's id + and its declared `attributes`; a query reading anything else misses and reads the + source, so results never differ from the uncached run. `SHOW GRAPH CACHES` gained + `hits`, `misses` and `cached_properties` columns. See [Caching a graph in Memgraph](/memgraph-zero/memgql/multiple-graphs#caching-a-graph-in-memgraph). -### Fixed +### 🐞 Bug fixes -- **ClickHouse returned result columns in alphabetical order** rather than the - order the query projected them. Column names travelled with their values, so - most clients were unaffected, but anything reading positionally was not: a - `RETURN n` whose mapping had a property sorting before `id` came back with that - property's value as the node's element id. +- **ClickHouse returned result columns in alphabetical order** rather than the order + the query projected them. Names travelled with their values, so most clients were + unaffected — but anything reading positionally was not. - **The Memgraph cache returned different rows than its source.** Cached edges - pointed at the wrong endpoints or were missing entirely, edge properties were - never copied at all, and a cache holding only a label's id answered property - reads with `NULL` instead of declining them — so a query silently changed its - answer once the cache was warm. + pointed at the wrong endpoints or were missing, edge properties were never copied, + and a cache holding only a label's id answered property reads with `NULL`. - **`count(DISTINCT )` and `count(DISTINCT )` produced invalid SQL** on relational backends. Counting distinct properties was unaffected. - **Grouping by a whole element produced invalid SQL** — `WITH n, count(…)` and `WITH r, count(…)` on relational backends. -- **A warm table was re-scanned from the source** every time a newly-touched - edge type pulled it in as an endpoint, repeating the full scan the cache - exists to avoid. -- **The cache's warm-up log counted rows read from the source**, so it reported - success even when no rows reached the cache. +- **A warm table was re-scanned from the source** whenever a newly-touched edge type + pulled it in as an endpoint. +- **The cache's warm-up log counted rows read from the source**, reporting success + even when no rows reached the cache. ## MemGQL v0.8.0 - July 19th, 2026 diff --git a/pages/memgraph-zero/memgql/connect.mdx b/pages/memgraph-zero/memgql/connect.mdx index 9321050c2..a495686e6 100644 --- a/pages/memgraph-zero/memgql/connect.mdx +++ b/pages/memgraph-zero/memgql/connect.mdx @@ -8,6 +8,7 @@ description: MemGQL connector details and configuration. - [ClickHouse](/memgraph-zero/memgql/connect/clickhouse) - [DuckDB](/memgraph-zero/memgql/connect/duckdb) - [Iceberg](/memgraph-zero/memgql/connect/iceberg) +- [Microsoft Fabric](/memgraph-zero/memgql/connect/fabric) - [Memgraph](/memgraph-zero/memgql/connect/memgraph) - [MongoDB](/memgraph-zero/memgql/connect/mongodb) - [MySQL](/memgraph-zero/memgql/connect/mysql) diff --git a/pages/memgraph-zero/memgql/connect/_meta.ts b/pages/memgraph-zero/memgql/connect/_meta.ts index c3f903b79..b66bd403c 100644 --- a/pages/memgraph-zero/memgql/connect/_meta.ts +++ b/pages/memgraph-zero/memgql/connect/_meta.ts @@ -1,6 +1,7 @@ export default { "clickhouse": "to Clickhouse", "duckdb": "to DuckDB", + "fabric": "to Microsoft Fabric", "iceberg": "to Iceberg", "memgraph": "to Memgraph", "mongodb": "to MongoDB", diff --git a/pages/memgraph-zero/memgql/connect/fabric.mdx b/pages/memgraph-zero/memgql/connect/fabric.mdx new file mode 100644 index 000000000..fd81a4c34 --- /dev/null +++ b/pages/memgraph-zero/memgql/connect/fabric.mdx @@ -0,0 +1,251 @@ +--- +title: Microsoft Fabric +description: Connect MemGQL to Microsoft Fabric. +--- + +# Microsoft Fabric + +The Fabric connector (type `fabric`) translates GQL queries into T-SQL and runs +them over a Fabric Warehouse's SQL endpoint (TDS, port 1433). It requires a +[mapping file](/memgraph-zero/memgql/reference#mapping-schema) that maps graph +patterns to relational tables — the same format as every other SQL backend — +and references tables with three-part `item.schema.table` names. + +**One connector, four surfaces.** Fabric exposes the same T-SQL endpoint shape +for a **Warehouse**, a **Lakehouse SQL analytics endpoint** (read-only), a **SQL +database in Fabric**, and every **mirrored database** (Snowflake, Azure SQL, +Cosmos DB, Databricks, Oracle, SQL Server replicated into OneLake). All four are +reachable through this one connector — a mirrored database needs no connector of +its own. + +The driver is **pure-Rust**: there is no ODBC layer or SQL client to install. +The connector supports **both reads and writes**. Fabric is cloud-only, so there +is no local container to run. It is available both in +[`multi` mode](/memgraph-zero/memgql/multiple-graphs) (`ADD CONNECTOR … TYPE +fabric`) and as the standalone `CONNECTOR_TYPE=fabric` mode. + +## 1. Prepare your Fabric workspace + +Create a workspace and a warehouse in the Fabric portal, then note two values +from the warehouse's **Settings → SQL endpoint**: + +- the **connection string**, e.g. `abc123….datawarehouse.fabric.microsoft.com` +- the **item name** of the warehouse (or lakehouse) you want to query + +Fabric accepts **Microsoft Entra ID only** — there is no SQL authentication, so +there is no username/password to create. For an unattended service such as +MemGQL, register an Entra **application (service principal)** and grant it +access to the workspace: + +1. In Entra ID, register an application and create a client secret. Note the + **tenant ID**, **application (client) ID**, and **secret value**. +2. In the Fabric admin portal, enable **Service principals can use Fabric + APIs** (tenant setting), optionally scoped to a security group containing + your application. +3. In the workspace, add the service principal with at least the **Viewer** + role, then grant it rights on the warehouse: + +```sql +CREATE USER [memgql-app] FROM EXTERNAL PROVIDER; +GRANT SELECT ON SCHEMA::dbo TO [memgql-app]; +-- Only if MemGQL should write: +GRANT INSERT, UPDATE, DELETE ON SCHEMA::dbo TO [memgql-app]; +``` + +A warehouse's **collation is fixed when it is created** and cannot be changed +afterwards. The default, `Latin1_General_100_BIN2_UTF8`, is case-sensitive — +which matches Cypher's case-sensitive string comparison more closely than SQL +Server's case-insensitive default. Choose deliberately: the alternative is +`Latin1_General_100_CI_AS_KS_WS_SC_UTF8`. + +## 2. Write the mapping + +Save as `mapping.json`, the standard +[mapping format](/memgraph-zero/memgql/reference#mapping-schema). Table +references resolve against the connector's item and schema (default `dbo`); +declare every property you plan to query — in `multi` mode the mapping is also +the routing schema, so an undeclared property is a routing error, not a +passthrough. + +```json +{ + "vertices": [ + { + "label": "Person", + "mappedTableSource": { + "connector": "fab", + "table": "persons", + "metaFields": { "id": "id" } + }, + "attributes": [ + { "name": "name" }, + { "name": "age", "type": "Int" } + ] + }, + { + "label": "Company", + "mappedTableSource": { + "connector": "fab", + "table": "companies", + "metaFields": { "id": "id" } + }, + "attributes": [ { "name": "name" } ] + } + ], + "edges": [ + { + "label": "KNOWS", + "from": "Person", + "to": "Person", + "mappedTableSource": { + "connector": "fab", + "table": "knows", + "metaFields": { "id": "id", "from": "from_id", "to": "to_id" } + } + }, + { + "label": "WORKS_AT", + "from": "Person", + "to": "Company", + "mappedTableSource": { + "connector": "fab", + "table": "works_at", + "metaFields": { "id": "id", "from": "person_id", "to": "company_id" } + } + } + ] +} +``` + +## 3. Start MemGQL in multi mode + +Mount your mapping. Every connection parameter is given on the `ADD CONNECTOR` +statement in the next step, so no Fabric environment variables are required +here: + +```bash +docker run --rm \ + --name memgql \ + --stop-timeout 2 \ + -p 7688:7688 \ + --env CONNECTOR_TYPE=multi \ + --env BOLT_LISTEN_ADDR=0.0.0.0:7688 \ + -v ./mapping.json:/data/mapping.json \ + memgraph/memgql:latest +``` + +## 4. Connect and register the backend + +```bash +mgconsole --port 7688 +``` + +Provide the endpoint, the warehouse item, and the service principal inline: + +```gql +ADD CONNECTOR fab TYPE fabric + URI 'abc123.datawarehouse.fabric.microsoft.com' + DATABASE 'memgql_wh' SCHEMA 'dbo' + TENANT_ID '' CLIENT_ID '' CLIENT_SECRET ''; +CREATE GRAPH social FROM FILE '/data/mapping.json'; +MATCH (n:Person) RETURN n.name LIMIT 5; +``` + +Options: `URI` (the SQL endpoint host), `DATABASE` (the warehouse or lakehouse +item), `SCHEMA` (default `dbo`), and either `TOKEN` or the +`TENANT_ID`/`CLIENT_ID`/`CLIENT_SECRET` triple. Any option you omit falls back +to the corresponding `FABRIC_*` environment variable, so you can mix inline +values with env defaults. + +Because the parameters live on the connector, **rotating a secret needs no +MemGQL restart** — re-register the connector: + +```gql +DROP CONNECTOR fab; +ADD CONNECTOR fab TYPE fabric + URI 'abc123.datawarehouse.fabric.microsoft.com' + DATABASE 'memgql_wh' + TENANT_ID '' CLIENT_ID '' CLIENT_SECRET ''; +``` + +Point a connector at a **Lakehouse SQL analytics endpoint** the same way, using +the lakehouse item name. That endpoint is read-only, so declare the graph read +only and let MemGQL reject writes before they reach Fabric: + +```gql +ALTER GRAPH social SET READ ONLY; +``` + +## 5. Query + +```gql +MATCH (p:Person) RETURN p.name, p.age; +``` + +```gql +MATCH (p:Person)-[:WORKS_AT]->(c:Company) RETURN p.name, c.name; +``` + +```gql +MATCH (a:Person)-[:KNOWS]->(b:Person) RETURN a.name, b.name; +``` + +## Authentication + +Fabric supports Microsoft Entra ID only. Supply exactly one method (precedence: +token → service principal). Each can be given inline on `ADD CONNECTOR` or via +an environment variable; the inline option wins when both are set: + +| Method | Inline option | Environment fallback | +|---|---|---| +| **Service principal** — recommended | `TENANT_ID` / `CLIENT_ID` / `CLIENT_SECRET` | `FABRIC_TENANT_ID` / `FABRIC_CLIENT_ID` / `FABRIC_CLIENT_SECRET` | +| **Access token** | `TOKEN ''` | `FABRIC_TOKEN` | + +**Tokens expire.** Entra access tokens are short-lived (about an hour). A token +is consumed when the SQL session logs in, so an open session keeps working after +its token expires — but the next reconnect needs a fresh one. Only the service +principal can mint a new token by itself; a connector given a bare `TOKEN` stops +working once that token expires and has to be re-registered. Use the service +principal for anything unattended. + +The standalone `CONNECTOR_TYPE=fabric` mode has no `ADD CONNECTOR`, so it takes +all of these from the environment. + +## Dialect notes + +- Table references are three-part `item.schema.table`, where the item is the + warehouse or lakehouse. Set `catalog` / `schema` on a `mappedTableSource` to + pin a table explicitly; the schema defaults to `dbo`. +- Because the item is part of the name, a single query can **join across items + in one workspace** — a Warehouse table to a Lakehouse SQL analytics endpoint + table — and MemGQL pushes that join down as one statement. +- Parameters are TDS binds (`@P1`, `@P2`, …); booleans are `BIT` `1`/`0`. +- Pagination uses `ORDER BY … OFFSET … ROWS FETCH NEXT … ROWS ONLY`. +- `AVG()` over integers is cast to `FLOAT` so results arrive as floats. +- Fabric is billed in capacity units, so pushing filters, aggregations and joins + down saves cost as well as latency. Expect the first query after an idle + period to be slower. + +## Known limitations + +These are Fabric engine limits, not MemGQL gaps: + +- **Variable-length paths** (`(){1,3}`, `-[:R*]-`) are **not supported**: Fabric + Warehouse does not support recursive queries. MemGQL rejects such a query with + a clear error rather than returning a partial answer — use a fixed number of + hops, or run it against a Cypher backend. +- **`collect()` and map projections** (`RETURN n {.a, .b}`) are not pushed down: + Fabric's `FOR JSON` must be the last operator in a statement and is invalid in + subqueries. +- **Generated ids are not returned** on insert. Fabric has no usable + `IDENTITY_INSERT`, so write a natural key and read the row back by it. +- **Column types** are limited to what a Fabric table can store: `nvarchar`, + `text`, `xml`, `json`, `money`, `datetime`, `datetimeoffset`, `tinyint` and + the spatial types do not exist there. Use `varchar` in a UTF-8 collation and + `datetime2`. +- **Constraints are metadata only** — `PRIMARY KEY`, `UNIQUE` and + `FOREIGN KEY` must be declared `NOT ENFORCED`, so Fabric will not reject a row + that violates them. + +For connector configuration, see the +[mapping reference](/memgraph-zero/memgql/reference#mapping-schema). diff --git a/pages/memgraph-zero/memgql/features.mdx b/pages/memgraph-zero/memgql/features.mdx index c95032441..a296e5d7a 100644 --- a/pages/memgraph-zero/memgql/features.mdx +++ b/pages/memgraph-zero/memgql/features.mdx @@ -24,6 +24,7 @@ description: MemGQL Community and Enterprise feature comparison. | [Oracle](/memgraph-zero/memgql/connect/oracle) | Yes | Yes | | [SQL Server](/memgraph-zero/memgql/connect/sqlserver) | Yes | Yes | | [Snowflake](/memgraph-zero/memgql/connect/snowflake) | Yes | Yes | +| [Microsoft Fabric](/memgraph-zero/memgql/connect/fabric) | Yes | Yes | | [MongoDB](/memgraph-zero/memgql/connect/mongodb) | Yes | Yes | | **Multi-Connection Mode** | Yes | Yes | | Max connectors | 2 | Unlimited | diff --git a/pages/memgraph-zero/memgql/multiple-graphs.mdx b/pages/memgraph-zero/memgql/multiple-graphs.mdx index 035f6b969..56d929878 100644 --- a/pages/memgraph-zero/memgql/multiple-graphs.mdx +++ b/pages/memgraph-zero/memgql/multiple-graphs.mdx @@ -11,7 +11,7 @@ If you want a running stack to try these queries against, the [Docker Compose - ## Where the catalog DSL works -The catalog statements (`ADD CONNECTOR`, `CREATE GRAPH`, `SHOW GRAPHS`, `SHOW CONNECTORS`, `DROP GRAPH`, `USE `, …) are available in **`CONNECTOR_TYPE=multi`** mode. Single-backend modes (`memgraph-gql`, `neo4j-gql`, `postgres`, `mysql`, `oracle`, `duckdb`, `clickhouse`, `iceberg`, `pinot`, `snowflake`, `mongodb`) connect to one backend configured via env vars and don't expose the catalog. +The catalog statements (`ADD CONNECTOR`, `CREATE GRAPH`, `SHOW GRAPHS`, `SHOW CONNECTORS`, `DROP GRAPH`, `USE `, …) are available in **`CONNECTOR_TYPE=multi`** mode. Single-backend modes (`memgraph-gql`, `neo4j-gql`, `postgres`, `mysql`, `oracle`, `duckdb`, `clickhouse`, `iceberg`, `pinot`, `snowflake`, `fabric`, `mongodb`) connect to one backend configured via env vars and don't expose the catalog. | Statement | `multi` | Cypher single-backend | SQL single-backend | |---|---|---|---| @@ -84,7 +84,7 @@ MATCH (me:Person {id: 1})-[:FRIEND_OF]->(f:Person) RETURN f.id; The schema index is built from two sources: - **SQL-family connectors** (PostgreSQL, MySQL, Oracle, DuckDB, ClickHouse, - Iceberg, Pinot, Snowflake, MongoDB): taken from the registered **mapping** (labels, + Iceberg, Pinot, Snowflake, Fabric, MongoDB): taken from the registered **mapping** (labels, rel-types, and properties are known exactly). - **Cypher-family connectors** (Memgraph, Neo4j): **introspected at `CONNECT` time** and cached. Memgraph uses `SHOW SCHEMA INFO` (the server must run with diff --git a/pages/memgraph-zero/memgql/reference.mdx b/pages/memgraph-zero/memgql/reference.mdx index 922ab8856..b73db1c90 100644 --- a/pages/memgraph-zero/memgql/reference.mdx +++ b/pages/memgraph-zero/memgql/reference.mdx @@ -116,7 +116,8 @@ ADD CONNECTOR TYPE [USER ''] [PASSWORD ''] [DATABASE ''] [CATALOG ''] [SCHEMA ''] [GRAPH ''] [WAREHOUSE ''] [ROLE ''] - [PRIVATE_KEY_PATH ''] [TOKEN '']; + [PRIVATE_KEY_PATH ''] [TOKEN ''] + [TENANT_ID ''] [CLIENT_ID ''] [CLIENT_SECRET '']; DROP CONNECTOR ; PING ; @@ -139,11 +140,12 @@ apply depends on the type: | Option | Read by | |-----------------------------|----------------------------------------------------------------------| | `GRAPH ''` | Memgraph, Neo4j — selects the Cypher database (it is not a mapping) | -| `DATABASE ''` | PostgreSQL, MySQL, SQL Server, Oracle (service name), ClickHouse, MongoDB, Snowflake | +| `DATABASE ''` | PostgreSQL, MySQL, SQL Server, Oracle (service name), ClickHouse, MongoDB, Snowflake, Fabric (warehouse/lakehouse item) | | `CATALOG ''` | Iceberg (Trino catalog), Iceberg Direct (warehouse) | -| `SCHEMA ''` | Iceberg, Snowflake; MongoDB accepts it as a fallback for `DATABASE` | +| `SCHEMA ''` | Iceberg, Snowflake, Fabric (default `dbo`); MongoDB accepts it as a fallback for `DATABASE` | | `WAREHOUSE` / `ROLE` | Snowflake session settings | -| `PRIVATE_KEY_PATH` / `TOKEN`| Snowflake auth (key-pair JWT / programmatic access token) | +| `PRIVATE_KEY_PATH` / `TOKEN`| Snowflake auth (key-pair JWT / programmatic access token); `TOKEN` is also a Fabric Entra ID access token | +| `TENANT_ID` / `CLIENT_ID` / `CLIENT_SECRET` | Fabric service-principal auth (Microsoft Entra ID) | | `PATH ''` | DuckDB (database file; `:memory:` by default) | An option a connector doesn't read is ignored, and one that is omitted falls diff --git a/pages/memgraph-zero/memgql/schema-file.mdx b/pages/memgraph-zero/memgql/schema-file.mdx index 387c651eb..011413d97 100644 --- a/pages/memgraph-zero/memgql/schema-file.mdx +++ b/pages/memgraph-zero/memgql/schema-file.mdx @@ -137,12 +137,12 @@ Every `connection` field is optional; supply what a given backend needs: | `user` / `password`| Credentials (override / complement the URI). | | `database` | Database name. | | `schema` | Middle namespace (PostgreSQL schema, MySQL / ClickHouse database, Iceberg schema). | -| `catalog` | Native root for three-level backends (Iceberg catalog, SQL Server database). | +| `catalog` | Native root for three-level backends (Iceberg catalog, SQL Server database, Fabric warehouse/lakehouse item). | | `path` | File-backed backends (DuckDB). | **Supported `type` values:** `memgraph`, `neo4j`, `postgres` (`postgresql`), `mysql`, `sqlserver`, `oracle`, `duckdb`, `iceberg`, `iceberg-direct`, -`clickhouse`, `pinot`. +`clickhouse`, `pinot`, `fabric`. ## `graphs`