When using Lance as the lakehouse format, does Fileset become redundant for unstructured data management?

[honam-data]

Background
I'm exploring Gravitino as a unified metadata governance layer for an AI/ML pipeline. My data architecture is:
Raw unstructured data (PDFs, images, audio)
→ stored in S3 (MinIO)
→ built into Lance tables (with embeddings, metadata, vector indexes)
→ governed by Gravitino
From what I understand:

1. Fileset is Gravitino's abstraction for managing non-tabular, schema-free data (training datasets, raw files). It manages file locations, credentials, and lifecycle on HDFS/S3/Local.
2. Lance (via lakehouse-generic catalog, format=lance) is Gravitino's integration for the Lance columnar format — managing table schemas, columns, and vector indexes.
   The Confusion
   In practice, Lance itself is a multimodal lakehouse format that already handles the full unstructured data lifecycle:

• Raw files are stored in S3 (e.g., s3://lance-bucket/)
• Lance tables are built on top, providing typed columns, metadata, vector embeddings, and indexes — all self-contained in the same S3 location
• Through Gravitino's Lance integration, I get schema management, permission control, and REST API access
  This means Lance already covers the Fileset use case end-to-end. I can manage raw unstructured data entirely through Lance tables under Gravitino, without ever touching Fileset.
  The Question

1. Does Lance make Fileset redundant? When you're already using Lance as the lakehouse format, what value does Fileset add? Or are they meant for different stages of the data pipeline?
2. What's the intended relationship? Are Fileset and Lance complementary legs of the same journey (raw ingestion → structured table), or are they independent alternatives with overlapping scopes?
3. What's the practical guidance? When should a user choose Fileset vs. Lance Table for managing unstructured data in a real AI/ML pipeline?
   Why This Matters
   As Lance adoption grows (it's increasingly positioned as the standard format for AI/multimodal lakehouses), this architectural ambiguity could confuse newcomers. Clarifying the positioning would help users decide which abstraction to use when.
   I'd love to hear from maintainers and the community — is there a design rationale behind keeping these two abstractions separate, or is this something that may evolve as the Lance integration matures?

[jerryshao]

Hi @honam-data I think Lance and fileset are two separate things.

1. Lance is still a table; it has schema. The typical way of using Lance is still a structured way, like query and search. We put Lance table under the management of the relational catalog, so it still appears as a relational table.
2. The fileset is quite different. It is like a container managing the raw files/objects under this container. We virtualize this container and use a name identifier to represent this container, to separate the understanding of physical location as well as the virtual name.
3. Besides, Lance is used to embed the raw data into a vector and stored in the file; this vector can be used for semantic similarity search. But, a fileset is just a raw container; it doesn't process the files and understand the content of the file, it just unfies the way of accessing the files.

So, they're actually two different concepts and things. They will not replace each other.

It depends on how you use it. Typically, your raw data (text/video/audio file) can be managed by a fileset and accessed by GVFS. And the embedded content can be stored in Lance. They can all be managed by Gravitino to build your own AI lakehouse.

[honam-data]

Hi @jerryshao, thanks a lot for the detailed clarification — the distinction between Lance (structured table with schema) and Fileset (raw file container) makes sense.
I have a follow-up question: you mentioned that Lance is used primarily for query and search, while Fileset doesn't process the files. But in a real AI pipeline, unstructured data carries a lot of metadata along with the raw content — things like file-level metadata (filename, size, format, upload time), business metadata (department, sensitivity level, compliance tags), and administrative metadata (owner, access history, lifecycle status).
Would you recommend storing this metadata in the Lance table alongside the embeddings and content? Or should it stay in the Fileset layer? In other words, can/should a Lance table serve as the single source of truth for both the content AND the metadata of unstructured data, or is metadata management still best kept at the Fileset level?
Curious how the community thinks about this metadata layer in the Lance + Fileset architecture.