Refactor existing CLI commands to map dataset and datasource in a less confusing ways

[xinaxu]

We received a few community feedback about the confusing concepts between dataset and datasource

• Week 1 test result #170 @kernelogic was suggesting to have a way to create a dataset with a datasource in the same command, as in singularity V1.
• [Bug]: "One to many", "many to one" or "one to one" for dataset relationship to datasource - zero length temp dag file #204 @distora-w3 was suggesting to take a bottom-up approach that creates datasource first and then attach datasource to dataset

There is also a relevent feature that requires some refactoring of the current mapping

• Use supported Rclone storage backend for output directory #142

Below is a my proposal of how we can refactor our current CLI commands and potentially some database schemas:

Dataset

A dataset is a top level object where you can attach multiple storage sources for data preparation and one optional output storage for exporting CAR files. If no output storage is defined, then inline preparation will be used.

It is also the object that can attach multiple wallets for deal making. There is no other arguments associated with dataset other than an enabled/disabled state. Only enabled dataset is considered for data prep so the user can define source storages and output storage before enabling it for data prep.

singularity dataset create mydataset
singularity dataset enable mydataset
singularity dataset disable mydataset

Storage

A storage is a local or remote storage system. It can be used as a source of a dataset or an output of it to store CAR files.

singularity storage create-source local local_storage /tmp/input
singularity storage create-source s3 --region=us-east-1 s3_storage my_bucket

singularity storage create-output local local_output /tmp/output

As part of defining source storage, you can also define how you'd like to chunk it or encrypt the files.

singularity storage create-source local --piece-size 64GiB local_storage /tmp/input

To browse the list of files of storage, you can use check command.

singularity storage check local_storage # list of actual files inside /tmp/input
singularity storage check local_output # list of CAR files likewise

Attach and detach storage to Dataset

Storages can be attached to dataset as either source or output. One dataset can have multiple sources but up to one output

singularity dataset add-source mydataset local_storage
singularity dataset add-source mydataset s3_storage
singularity dataset add-source mydataset http_site
singularity dataset set-output mydataset local_output

If the dataset is not currently in process of deal making or data preparation, you can detach the source storage

singularity dataset remove-source mydataset local_storage

You can switch to a different output storage anytime, i.e. when your current output storage is running out of space

singularity dataset set-output mydataset local_output_2

If you don't set an output or clear the output storage, inline preparation will be used

singularity dataset clear-output mydataset

Data preparation management

You can start dataset worker anytime

singularity run dataset-worker

However, you will need to enable the dataset before dataset worker can pick them up

singularity dataset enable mydataset

To manage data preparation tasks, it is inside storage menu. The reason why such task is not performed on the dataset level is because the dataset can contain multiple data sources and can be confusing and error prone for users to accidentally start or stop data preparation for all other source storage, i.e. if one source finished prep and another source failed, would retry means retry both or retry failed one. Same issues exist for all other similar commands and involves updating state in the database. See below

singularity storage rescan
singularity storage pause
singularity storage repack
singularity storage daggen
singularity storage status
singularity storage list-chunks
singularity storage list-items
singularity storage list-dags
singularity storage list-path
singularity storage chunkdetail
singularity storage itemdetail
singularity storage list-pieces

For commands that is read only, we can have same command at dataset level, which iterates over all source storages and displays similar info

singularity dataset status
singularity dataset list-chunks
singularity dataset list-items
singularity dataset list-dags
singularity dataset list-pieces

Wallet

Wallet management will not change.

You can associate wallet with a dataset. The dataset needs to have associated wallets before it can be used for deal making

singularity dataset add-wallet <wallet_id>
singularity dataset list-wallets
singularity dataset remove-wallet <wallet_id>

[jyma]

singularity-v2的很多业务涉及到数据库表格，像这样的业务变更可能未来还会有， 可以考虑在进行重大数据库结构变更时增加旧版本数据库文件处理功能，或者可以有一个小工具来完成旧版本数据库到新版本数据库的数据处理。

[distora-w3]

Some thoughts on @jyma comment.

I am not convinced why there is a need to maintain backward compatibility.
Scenario:

1. I use singularity v1 to car up the files and store with SPs
2. Why do I now need singularity V1?
3. Once the files are stored, I only need to know how to make the retrieval, which is a single top level car file. Correct?
4. Is it not the same issue for Singularity MVP?

If we need to know about metadata, encryption and other parameters, perhaps a log file dump would allow other retrieval-related problems. I personally see more value in reaching MVP+ stage than in spending time in backward compatibility.

[jyma]

Hi～ @distora-w3

I agree with you that it's more value in reaching MVP+ stage than in spending time in backward compatibility.

However, the point I'm trying to convey is not that we need to be compatible with V1, but rather that we need to ensure compatibility with versions prior to V2 when there are significant database changes in V2. Of course, we are currently in the testing phase, but after the official version is released, this becomes a necessary issue to address.

[distora-w3]

Makes sense :)

[xinaxu]

There is already a V1->V2 migration tool.
https://data-programs.gitbook.io/singularity/cli-reference/admin/migrate-dataset

However, there is not going to be a V2 pre-MVP -> V2 MVP migration tool

[distora-w3]

@Xinan Xu - PL
Thank you for considering #204. Some feedback on your discussion. To avoid confusion, below, each item has been chunked into a code block to demarcate points made. Each point has a comment below it. (I hope this makes sense)

singularity dataset create mydataset
singularity dataset enable mydataset
singularity dataset disable mydataset

==== Yes to these

singularity storage create-source local local_storage /tmp/input
singularity storage create-source s3 --region=us-east-1 s3_storage my_bucket

singularity storage create-output local local_output /tmp/output

==== Yes to these

singularity storage create-source local --piece-size 64GiB local_storage /tmp/input

==== I have confusion with this
- This command to me says input specifications
- I understand --piece-size as an output item specification related to what the SP can provide
- In my thinking there can be multiple inputs, therefore multiple places to set the parameter and multiple places to make a mistake
- Is this not a "processing" parameter meaning a setting for a worker? Should this not be a "batch job specification"  eg process data source with x, y and z parameters

singularity storage check local_storage # list of actual files inside /tmp/input
singularity storage check local_output # list of CAR files likewise

==== Yes to this
==== Feature request
Can we have progress?
- use case: do I cancel or let finish? Will depend on time remaining

singularity dataset add-source mydataset local_storage
singularity dataset add-source mydataset s3_storage
singularity dataset add-source mydataset http_site
singularity dataset set-output mydataset local_output

==== Yes to this
==== Feature request
Can we have ability to pause?
- use case: detach and then re-attach to preserve non processing could lead to errors and mistake with cli typos
- would need status to be able to determine enabled or disabled in order to toggle production

singularity dataset remove-source mydataset local_storage

==== Similar issue. Can be dangerous to delete or remove with cli. Pause/Disable and then enable would be safer

singularity dataset set-output mydataset local_output_2

==== Very good! This is great for preventing diskfull in production

singularity dataset clear-output mydataset

==== We should use consistent wording "detach-output"

singularity run dataset-worker

==== Can we remove notion of worker and keep to task?
-- singularity run dataset XYZdataset 123dataset ....
-- singularity run dataset all

Comment for ===================
To manage data preparation tasks, it is inside the storage menu. The reason why such task is not performed on the dataset level is because the dataset can contain multiple data sources and can be confusing and error prone for users to accidentally start or stop data preparation for all other source storage, i.e. if one source finished prep and another source failed, would retry means retry both or retry failed one. Same issues exist for all other similar commands and involves updating state in the database. 
=================================

- I understand the reasoning above. However, my logic as a user is as follows:
[1] I make a contract with a customer to store a dataset.
[2] To understand item 1, I need to understand if a dataset has been stored or not to meet the contract.
[3] To understand item 2, I need a status report on the dataset

Example command:

singularity status report XYZdataset

status XYZdataset
--------------------------------------------------------
-- storage1 processed, encrypted, carred, stored with miner 1000,2000
-- storage2 failed, error encryption
--------------------------------------------------------

[4] The above means I trouble shoot or retry the lower levels.

The analogy here is a batch processing job:

XYZdataset job list
- start date
- parameters eg 64GiB piece-size
- process storage 1
- process storage 2
- send confirmation
- log  jobs, errors

If we continue the thinking logically we have a hierarchy:

• job
• dataset
• storage
  Ultimately, the Singularity story does not stop there as we can imagine one more top level.
  contract (or customer)
• multiple jobs
• multiple datasets
• multiple storage

[xinaxu]

I am breaking down above feedback into multiple threads in slack channel.
https://filecoinproject.slack.com/archives/C05JABREATH/p1692282780187149

[xinaxu]

After more thoughts and incorporating some feedback. I think we can simplify this as two steps process.

1. Define storage
2. Start a preparation to pipe from one storage to another storage

Example commands below

# Create storage first
singularity storage create local local_storage /tmp/input
singularity storage create s3 --region=us-east-1 s3_storage my_bucket
singularity storage create local local_output /tmp/output

# Start a preparation task
singularity data-prep create --piece-size 32GiB --source local_storage --output local_output prep1

# Create a new task with different parameters
singularity data-prep create --piece-size 64GiB --source local_storage --output local_output prep2

# Reuse same output with different sources
singularity data-prep create --piece-size 32GiB --source s3_storage --output local_output prep3

# Status command is now only for preparation objects
singualrity data-prep list
singualrity data-prep status prep1

# At storage level, the only command is to show associated preparations
singualrity storage list-preparations local_storage
> local_storage is used as source for
> - prep1
> - prep2

[xinaxu]

Agreed over zoom call to use above commands