Repository navigation
Integrating with lake table stores (e.g., Apache Paimon): a table type or index format instead of a second copy of graph data? #988
lipengming
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Background
GraphAr is positioned as "a standardized system-independent format for storing graph data" (README), with its own metadata and chunked data files. Meanwhile, Apache Paimon already supports third-party table types (
ICEBERG_TABLE,LANCE_TABLEinorg.apache.paimon.TableType) and has mechanisms for maintaining secondary index structures over its tables.The concern: a second physical copy of the data
In a typical lakehouse, the source of truth lives once in the lake — e.g., a Paimon vertex table and an edge table. To use GraphAr today, the data has to be re-materialized into GraphAr chunks (fixed-size vertex chunks, adjacency layout with offset tables), because lake-managed Parquet files cannot be directly pointed to by GraphAr metadata. That means two physical copies of the same graph data (plus two metadata trees), which seems to conflict with the data-lake principle of "one copy of data, many views/materializations".
Possible integration shapes
Has the community considered making GraphAr work with lake table stores rather than alongside them? Two concrete shapes:
GRAPHAR_TABLE-style table type (similar to Paimon's existing Iceberg/Lance table types): the graph is stored once in GraphAr layout, while the lake engine handles registration, schema evolution, compaction, and snapshot-based time travel for graph versions.For context on demand: I've been prototyping graph indexes directly in Paimon — adjacency index tables, node-map reverse index, and a k-hop traversal SDK over Paimon tables (
paimon-graphmodule on thefeature/graph-indexbranch of my fork, lipengming/paimon). The experiment confirms there is demand for graph retrieval on lake tables, but the index layout is Paimon-internal. A system-independent layout spec — which is very close to what GraphAr already specifies — would let this interoperate across engines instead of being Paimon-specific.Questions
Happy to work on a PoC if there's interest, and glad to move this discussion to the dev mailing list (dev@graphar.apache.org) if that's a better venue.
中文摘要
问题背景:GraphAr 定位为"标准化的、系统无关的图数据存储格式",有自己的元数据和分块数据文件。但在典型湖仓架构中,图数据的 source of truth 已经在湖里(如 Paimon 的点表/边表)。今天要用 GraphAr,必须把数据重新物化成 GraphAr 分块(固定大小的顶点分块、带 offset 的邻接布局),湖里管理的 Parquet 文件无法直接被 GraphAr 元数据引用——这意味着同一份图数据会存两份物理副本(外加两套元数据树),似乎与数据湖"一份数据、多种视图"的初衷相悖。
两种可能的整合形态:
GRAPHAR_TABLE——图数据只存一份(GraphAr 布局),注册、schema 演进、compaction、快照时间旅行等由湖引擎负责。个人实践佐证需求:我在自己的 fork(
lipengming/paimon的feature/graph-index分支)上做了paimon-graph原型——Paimon 表之上的邻接索引表、node-map 反查索引和 k-hop 遍历 SDK。实验证明基于湖表的图检索确有需求,但索引布局是 Paimon 私有的;一个系统无关的布局规范(与 GraphAr 已有规范非常接近)能让这类能力跨引擎互通。想请教的问题:
如有兴趣我乐意做 PoC,也欢迎转到 dev 邮件列表讨论。
All reactions