Add `datasets` namespace to MinIO

[aicam]

Task Summary

Background & Goal
We are introducing a namespace concept for MinIO and LakeFS. The primary goal is to cleanly separate our storage paths for different types of assets. By implementing namespaces now, we can isolate our current datasets and pave the way for adding models (and other assets) in the future.

Current State
Currently, the namespace is defined at the root bucket level without any specific asset prefix:

storageNamespaceURI=s"${StorageConfig.lakefsBlockStorageType}://${StorageConfig.lakefsBucketName}"

Proposed Change
We need to append the datasets prefix to the end of the existing storage path. This new namespace will be used exclusively for dataset storage.

• Target URI Structure: [block-storage-type]://[bucket-name]/datasets

Impact & Migration (Breaking Change)

• ⚠️ Compatibility: Once this change is merged, all existing datasets pointing to the old root namespace will become incompatible.
• Resolution: A data migration will be required to move existing datasets from the old namespace to the new /datasets namespace.

Out of Scope

• The development and execution of the migration script will not be part of the Pull Request for this specific issue. It will be handled as a separate task.

Priority

P2 – Medium

Task Type

• Code Implementation
• Documentation
• Refactor / Cleanup
• Testing / QA
• DevOps / Deployment

[chenlica]

@aicam Also I believe the scope of this sub-issue is not just changing the MinIO. It's about a change to the path of a file in a dataset.

Please use an example to show the change.

[carloea2]

Hi, I dont have much experience with LakeFS, but a suffix approach is possible? or even if I want to access models through an API to use it, can I add as a query param how I want to treat the underlying object as? such as type: "model"?

I think the prefix approach have many advantages for sure, for the long term the prefix seems the solid approach as well.