diff --git a/profile/README.md b/profile/README.md index 061998b..2fc29a9 100644 --- a/profile/README.md +++ b/profile/README.md @@ -24,13 +24,13 @@ The companion datasets and reproducible scripts live in [MobilityDataScienceBook ## Repository map -MobilityDB Ecosystem โ€” MEOS core; MobilityDB, MobilityDuck and MobilitySpark as peer SQL surfaces; MEOS-API as a side codegen catalog projected to OpenAPI/MCP/runtime; a teal portable-data interchange band (Arrow C Data Interface, Parquet / Temporal Data Lake) beneath MEOS; and a dashed, ghosted PLANNED Stream-layers box (MobilityNebula, MobilityKafka, MobilityFlink) as a future fourth peer +MobilityDB Ecosystem โ€” MEOS core; MobilityDB, MobilityDuck and MobilitySpark as peer SQL surfaces; MEOS-API as a side codegen catalog projected to OpenAPI/MCP/runtime; a teal portable-data interchange band (Arrow C Data Interface, Parquet / Temporal Data Lake) beneath MEOS; and a Stream-layers box (MobilityNebula, MobilityKafka, MobilityFlink) as a fourth peer The sections below follow the figure's boxes **bottom โ†’ top, left โ†’ right**. ### ๐Ÿ”„ Portable data โ€” the interchange band beneath MEOS -This is the portable-data interchange band drawn beneath MEOS in the figure. MEOS exposes a zero-copy [Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html) and a Parquet / *TemporalParquet* on-disk form, so trajectories move between the engines and a **Temporal Data Lake** without re-encoding. It is the data-side complement of the *portable-computation* property of the SQL layers below. Both properties โ€” and their reproducible companions โ€” are catalogued in the mobility-platform interoperability index (in [MobilityDB](https://github.com/MobilityDB/MobilityDB), `doc/temporal-parquet/`). +This is the portable-data interchange band drawn beneath MEOS in the figure. MEOS exposes a zero-copy [Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html) and a Parquet / *TemporalParquet* on-disk form, so trajectories move between the engines and a **Temporal Data Lake** without re-encoding. It is the data-side complement of the *portable-computation* property of the SQL layers below. Both properties โ€” and their reproducible companions โ€” are catalogued in the mobility-platform interoperability index (in [MobilityDB](https://github.com/MobilityDB/MobilityDB), `doc/temporal-parquet/`). On disk, the **Temporal Data Lake** stores trajectories as [Apache Iceberg](https://iceberg.apache.org/) tables under an [Apache Polaris](https://polaris.apache.org/) REST catalog (RBAC, multi-tenancy, credential vending); columnar engines such as [Polars](https://pola.rs/) read them straight off the Arrow stream without copies โ€” see the Iceberg / Polaris catalog ([MobilityDuck#179](https://github.com/MobilityDB/MobilityDuck/pull/179)) and the Polars round-trip ([PyMEOS-Examples#6](https://github.com/MobilityDB/PyMEOS-Examples/pull/6)). ### โฌ› Core C library @@ -42,7 +42,7 @@ This is the portable-data interchange band drawn beneath MEOS in the figure. MEO | Repository | Description | |---|---| -| [MEOS-API](https://github.com/MobilityDB/MEOS-API) | Machine-readable description of the MEOS C-library API (an IDL JSON plus a shape-metadata catalog), generated from the MEOS headers via libclang. Beyond binding code-generation, the enriched catalog is projected into service contracts: an OpenAPI 3.1 contract, a Model Context Protocol (MCP) tool manifest (so LLMs/agents can call the MEOS spatiotemporal algebra directly), and a contract-driven runtime HTTP server. | +| [MEOS-API](https://github.com/MobilityDB/MEOS-API) | Machine-readable description of the MEOS C-library API (an IDL JSON plus a shape-metadata catalog), generated from the MEOS headers via libclang. It is the single source the language bindings (Python, Java, Rust, Go, .NET, JavaScript) are generated from โ€” including the JMEOS binding behind the JVM streaming runtimes โ€” and from which the enriched catalog is projected into service contracts: an OpenAPI 3.1 contract, a Model Context Protocol (MCP) tool manifest (so LLMs/agents call the MEOS spatiotemporal algebra directly), and a contract-driven runtime HTTP server. | ### ๐ŸŸฆ SQL layers (peers above MEOS) @@ -56,9 +56,19 @@ This is the platform's **portable computation** property โ€” one query text, thr | [MobilityDuck](https://github.com/MobilityDB/MobilityDuck) | DuckDB extension โ€” peer SQL layer for analytics / columnar workloads. | | [MobilitySpark](https://github.com/MobilityDB/MobilitySpark) | Apache Spark plugin โ€” peer SQL layer for distributed and large-scale workloads, with MEOS-backed UDFs and DataFrame integration. | -### ๐ŸŒŠ Stream layers (planned) +### ๐ŸŒŠ Stream layers -**Planned โ€” not yet built.** The same edge-to-cloud model is planned for the streaming side of the ecosystem, each tool in its canonical role: [MobilityNebula](https://github.com/MobilityDB/MobilityNebula) ([NebulaStream](https://nebula.stream/)) on the **edge**, [MobilityKafka](https://github.com/MobilityDB/MobilityKafka) ([Apache Kafka](https://kafka.apache.org/)) as the streaming **transport backbone** in between, and [MobilityFlink](https://github.com/MobilityDB/MobilityFlink) ([Apache Flink](https://flink.apache.org/)) for **stream processing in the cloud** โ€” reproducing the SNCB benchmark from [*MobilityNebula* (EDBT 2026)](https://docs.mobilitydb.com/pub/MobilityNebula_EDBT_2026.pdf). It is drawn as a dashed, ghosted box โ€” same format as the other peer boxes, second in the peers row of the figure above (dashed = not yet built). +The same edge-to-cloud model runs on the streaming side of the ecosystem, each tool in its canonical role: [MobilityNebula](https://github.com/MobilityDB/MobilityNebula) ([NebulaStream](https://nebula.stream/)) on the **edge**, [MobilityKafka](https://github.com/MobilityDB/MobilityKafka) ([Apache Kafka](https://kafka.apache.org/)) as the streaming **transport backbone** in between, and [MobilityFlink](https://github.com/MobilityDB/MobilityFlink) ([Apache Flink](https://flink.apache.org/)) for **stream processing in the cloud**. The published reference architecture is [*MobilityNebula* (EDBT 2026)](https://docs.mobilitydb.com/pub/MobilityNebula_EDBT_2026.pdf), with real railway data (SNCB) as the application demonstration. It is the second peer in the row of peer surfaces in the figure above. + +The parity contract matches the SQL-layer one: **the same BerlinMOD reference queries run across all three platforms in three streaming forms โ€” continuous (always-on), windowed (tumbling / sliding / session), and snapshot (query at time T, โ‰ก the batch result at the same scale factor)** โ€” with the snapshot form anchored to the batch BerlinMOD outputs in [MobilityDB-BerlinMOD](https://github.com/MobilityDB/MobilityDB-BerlinMOD). The same generator and scale-factor axis as the batch side are reused. + +The Flink and Kafka platforms use [JMEOS](https://github.com/MobilityDB/JMEOS); MobilityNebula calls MEOS directly through its C ABI. + +The streaming-form parity matrix is scaffolded across all three runtimes โ€” [MobilityFlink#3](https://github.com/MobilityDB/MobilityFlink/pull/3), [MobilityKafka#1](https://github.com/MobilityDB/MobilityKafka/pull/1) and [MobilityNebula#15](https://github.com/MobilityDB/MobilityNebula/pull/15) โ€” each implementing the full 27-cell BerlinMOD-Q ร— 3-form matrix in the runtime's native operator surface. **81 / 81 cells full, 0 partial.** + +On the JVM platforms the spatial-predicate surface routes through a single `MEOSBridge` class โ€” [MobilityFlink#4](https://github.com/MobilityDB/MobilityFlink/pull/4) and [MobilityKafka#2](https://github.com/MobilityDB/MobilityKafka/pull/2) โ€” calling MEOS' `geog_dwithin` over WGS84 geographies via [JMEOS#18](https://github.com/MobilityDB/JMEOS/pull/18)'s `utils.spatial.Haversine` and `utils.spatial.PointToSegment` wrappers when libmeos is loadable, with a pure-Java great-circle fallback for the mini-cluster local-test runs. The 27 cells ร— 2 platforms are MEOS-backed at every predicate AND distance site. + +On the NebulaStream side, three stacked PRs add the BerlinMOD-specific aggregations across the four pipeline layers (logical / physical / parser / lowering): [MobilityNebula#16](https://github.com/MobilityDB/MobilityNebula/pull/16) adds `TEMPORAL_LENGTH` (Q6), and [MobilityNebula#17](https://github.com/MobilityDB/MobilityNebula/pull/17) adds `PAIR_MEETING` (Q5) + `CROSS_DISTANCE` (Q9), all calling MEOS C directly. Together with PR#15, the MobilityNebula matrix-row is 27 / 27 full. | Repository | Engine | |---|---| @@ -99,7 +109,7 @@ Each binding follows its language community's naming convention. | Repository | Stack | |---|---| | [MobilityDeck](https://github.com/MobilityDB/MobilityDeck) | [deck.gl](https://deck.gl/) | -| [MobilityFlink-Deck](https://github.com/MobilityDB/MobilityFlink-Deck) | [deck.gl](https://deck.gl/) on the planned Flink stream layer | +| [MobilityFlink-Deck](https://github.com/MobilityDB/MobilityFlink-Deck) | [deck.gl](https://deck.gl/) on the Flink stream layer | | [MobilityOpenLayers](https://github.com/MobilityDB/MobilityOpenLayers) | [OpenLayers](https://openlayers.org/) | | [MobilityLeaflet](https://github.com/MobilityDB/MobilityLeaflet) | [Leaflet](https://leafletjs.com/) | | [MobilityQGIS](https://github.com/MobilityDB/MobilityQGIS) | [QGIS](https://qgis.org/) integration | diff --git a/profile/images/mobilitydb_ecosystem.svg b/profile/images/mobilitydb_ecosystem.svg index f651ec6..b31513c 100644 --- a/profile/images/mobilitydb_ecosystem.svg +++ b/profile/images/mobilitydb_ecosystem.svg @@ -1,7 +1,7 @@ MobilityDB Ecosystem - Architecture diagram. MEOS (canonical C library) at the center; SQL layers (MobilityDB, MobilityDuck, MobilitySpark), HTTP/API (MobilityAPI), and language bindings (PyMEOS, JMEOS, meos-rs, GoMEOS, MEOS.NET, MEOS.js) as peer layers above; MEOS-API as a side codegen catalog projected to OpenAPI, MCP and a runtime HTTP server; application platforms, visualization, and cloud deployment as the upper layer; a portable-data interchange band (Arrow C Data Interface; Parquet / Temporal Data Lake) beneath MEOS. A dashed, ghosted PLANNED (roadmap) Stream-layers box โ€” same format and same peers row as the SQL, HTTP/API and Language-Bindings boxes โ€” not yet built โ€” shows the same edge-to-cloud model extending to streaming: MobilityNebula on the edge, MobilityKafka as the streaming transport backbone, MobilityFlink for cloud stream processing. The portable-data interchange band is shown in teal. + Architecture diagram. MEOS (canonical C library) at the center; SQL layers (MobilityDB, MobilityDuck, MobilitySpark), Stream layers (MobilityNebula on the edge, MobilityKafka as the streaming transport backbone, MobilityFlink for cloud stream processing), HTTP/API (MobilityAPI), and language bindings (PyMEOS, JMEOS, meos-rs, GoMEOS, MEOS.NET, MEOS.js) as peer surfaces above; MEOS-API as a side codegen catalog projected to OpenAPI, MCP and a runtime HTTP server; application platforms, visualization, and cloud deployment as the upper layer; a portable-data interchange band (Arrow C Data Interface; Parquet / Temporal Data Lake) beneath MEOS in teal. The Stream-layers box is the second peer in the row, matching the SQL, HTTP/API and Language-Bindings boxes in shape. @@ -37,7 +37,7 @@ - + SQL Layers @@ -48,17 +48,15 @@ MobilitySpark ยท Spark - - - - Stream Layers โ€” planned - - MobilityNebula ยท NebulaStream - - MobilityKafka ยท Kafka - - MobilityFlink ยท Flink - + + + Stream Layers + + MobilityNebula ยท NebulaStream + + MobilityKafka ยท Kafka + + MobilityFlink ยท Flink @@ -85,8 +83,9 @@ - - + + + @@ -125,8 +124,8 @@ Cloud Portable data - - Stream (planned) + + Stream layers