Use Mounting Feature for Native Operators

[aicam]

📌 Context & Current State

We recently implemented the LakeFS repository mounting feature for Python UDFs. Currently, mounting a LakeFS repository into a workflow is strictly accessible via Python execution inside UDFs.

While this works well for custom Python scripts, the mounting capability can provide immense value if exposed directly to native operators within Computing Units (CUs) during workflow execution.

---

🎯 Problem Statement & Opportunity

Previously, native operators faced several limitations when dealing with LakeFS or S3 datasets:

• Multi-file traversal: A single operator could not easily traverse directory trees or iterate over multiple files belonging to the same dataset.
• Temp file overhead: Data often had to be explicitly downloaded or copied into temporary local files before processing.

By mounting a LakeFS repository directly inside a Computing Unit (CU), the entire dataset version becomes accessible as a native filesystem path throughout the workflow run.

---

💡 Proposed Use Cases for Native Operators

We want to explore different ways native operators can leverage mounted LakeFS repositories. Here are a few initial ideas:

1. Pattern-Based Scan Operators

Scan operators could accept a dataset version path (or subfolder) along with a glob pattern (e.g., *.csv, data_*.parquet). Depending on configuration, the operator could:

• Concatenate matching files into a single unified stream.
• Emit one row/record per discovered file (useful for file-level metadata processing).

2. "List Files" Source Operator

A dedicated source operator that scans a mounted directory and outputs file paths/metadata downstream. Downstream operators can then process individual files in parallel or sequentially.

3. In-Place File Processing

Operators can read mounted files directly via standard filesystem interfaces (/mnt/lakefs/...) rather than executing expensive pre-copying operations to local /tmp directories.

4. Dynamic Multi-Port Output

Allow a scan operator to dynamically generate output ports based on the number of matching files discovered during workflow setup or runtime execution.

---

⚡ Benefits: Mounting vs. Direct S3 API Calls

Why leverage filesystem mounting instead of standard S3 API client calls?

Feature	Direct S3 API Calls	Mounted LakeFS Repository
Interface	Requires S3 SDK integrations per operator	Standard OS & POSIX filesystem calls (open(), read(), ls)
Data Fetching	Eager download or explicit range queries	Lazy loading: Data blocks are fetched on-demand
Resource Efficiency	Downloads full files even if partially read	Efficient when querying/inspecting large multi-file datasets
Caching	Call multiple times to fetch same data	Cache on first call and become available on disk

---

💬 Questions for the Community

We would love to get your feedback on this proposal:

1. Which native operator patterns would benefit your current workflows the most?
2. Are there performance or schema considerations we should keep in mind for dynamic multi-port scan operators?
3. Are there other use cases for mounted repositories in native workflows that we haven't considered yet?

User Experience

https://github.com/user-attachments/assets/e45ac438-4c42-4cd6-878e-89257c648a9e

[chenlica]

It will be good to include architecture diagrams to show how "datasets" are used, and include screenshots to show how existing operators (e.g., CSV File Scan, and Python UDF) reads a file by giving a specific path.