new command to list data artifacts in a DVC project

[jorgeorpinel]

Especially useful for "browsing" external DVC projects on Git hosting before using dvc get or dvc import. Looking at the Git repo doesn't show the artifacts because they're only referenced in DVC-files (which can be found anywhere), not tracked by Git.

Perhaps dvc list or dvc artifacts? (Or/and both dvc get list and dvc import list)

As mentioned in treeverse/dvc.org#611 (comment) and other discussions.

---

UPDATE: Proposed spec (from #2509 (comment)):

usage: dvc list [-h] [-q | -v] [--recursive [LEVEL]] [--rev REV | --versions]
                url [target [target ...]]

positional arguments:
  url         URL of Git repository with DVC project to download from.
  target      Paths to DVC-files or directories within the repository to list outputs
              for.

UPDATE: Don't forget to update docs AND tab completion scripts when this is implemented.

[efiop]

+1 for dvc list 🙂

[shcheklein]

@efiop @jorgeorpinel another option is to do dvc ls and it should behave exactly like a regular ls or aws s3 ls. Show all the files (including hidden data) by specifying a Git url. This way you can control the scope to show (by not going into all directories by default) - also you can see your data in the context (other files) with an option to filter them out.

On the other hand it can be good to show just a list of all DVC outputs. It can be done with dvc ls --recursive --outputs-only for example.

What do you think?

In general I'm +100 for dvc list or something similar :)

[efiop]

Clarification: dvc list should work on dvc repositories. E.g. dvc list https://github.com/iterative/dvc should list scripts/innosetup/dvc.ico, etc.

[shcheklein]

@efiop can we change the priority to p1 as we discussed because it's part of the get/import epic story?

[efiop]

@shcheklein Sure, forgot to adjust that. Thanks for the heads up!

[jorgeorpinel]

@shcheklein I'm not sure about the dvc ls name if all we want is to list data artifacts specifically (not all files as you propose).

...you can see your data in the context (other files) with an option to filter them out.

Will most users need this? The problem I think we need to solve is that it's hard to use dvc get and dvc import without a list of available data artifacts. Showing all Git-controlled files may or may not be useful but can be done by existing means.

by specifying a Git url...

What I get from that is: dvc list [url] where url can be a file system path or HTTP/SSH URL to a Git repo containing a DVC project (same as the dvc get and dvc import argument url) – and if omitted tries to default to the local repo (containing pwd).

This way you can control the scope to show (by not going into all directories by default)...

I don't see how just the Git repo URL can control the scope to show. Would need a path argument for this I think (which could only accept directories) and/or a --depth option.

dvc ls --recursive --outputs-only

Too complicated for users to remember if the main purpose of this new command is to list artifacts. Maybe --recursive though... (aws s3 ls for example isn't recursive by default, but has that option.)

In summary I think just dvc list or dvc outs is best.
But I agree we would need to consider the case where there are lots of outputs. Another solution (besides path arg and/or --depth, --recursive options) could be default pagination (and possibly interactive scrolling with ⬆️ ⬇️ – like a stdout pipe to less).
aws s3 ls on the other hand takes a simple approach: It has a hard limit of 1000 objects.

[jorgeorpinel]

Clarification: dvc list should work on dvc repositories. E.g. dvc list https://github.com/iterative/dvc should list scripts/innosetup/dvc.ico/etc.

@efiop yes, exactly. Extended example:

$ dvc list https://github.com/iterative/dvc
scripts/innosetup/dvc.ico
scripts/innosetup/dvc_left.bmp
scripts/innosetup/dvc_up.bmp

This makes me think, maybe an output that combines the DVC-files (similar to dvc pipeline list) with their outputs could be most informative (providing some of the context Ivan was looking for). Something like:

$ dvc list https://github.com/iterative/dvc
scripts/innosetup/dvc.ico	(from scripts/innosetup/dvc.ico.dvc)
scripts/innosetup/dvc_left.bmp	(from scripts/innosetup/dvc_left.bmp.dvc)
scripts/innosetup/dvc_up.bmp	(from scripts/innosetup/dvc_up.bmp.dvc)

UPDATE: Thought of yet another 😓 name for the command above: dvc stage list --outs

[shcheklein]

@jorgeorpinel I think showing the full project in an ls-way is just more natural as opposed to creating our own type of output. There are few benefits:

1. You don't have to use two interface to see the full picture - Github + dvc list. Instead you just use dvc and see the workspace. And can filter it if it's needed.
2. It looks like it's beneficial for dvc get to handle regular Git files. Why not? It can be useful.
3. Like I mentioned - single place in CLI, no need to go to Github to get the full picture.
4. Just easier to understand since people are familiar with ls and intuitively can expect the result.

The idea is that by default it's not recursive, it's not limited to outputs only. You go down on your own if you need by clarifying path - the same way you do with ls, aws ls, etc.

ls, aws ls, etc - they are all not recursive by default for a reason. In a real case the output can be huge and just won't make sense for you. People tend to go down level by level, or use recursive option when it's exactly clear that it's what they need.

I really don't like making piping and less and complex interfaces part of the tool. You can always use less if it's needed.

[jorgeorpinel]

@shcheklein I concur only with benefit #1, so I can agree with showing all files but with an optional flag (which can be developed later, with less priority), not as the default behavior. Thus I wouldn't call the command dvc ls – since it's not that similar to GNU ls. I would vote for dvc outs or dvc list.

2. It looks like it's beneficial for dvc get to handle regular Git files. Why not?

Because it can already be done with git archive (as explained in this SO answer).

3... single place in CLI, no need to go to Github to get the full picture.

The "full picture" could still be achieved from CLI by separately using git ls-tree.

The idea is that by default it's not recursive... You go down on your own if you need by clarifying path...

I can also agree with this: Non-recursive by default sounds easier to implement. So, it would definitely also need an optional dir (path) argument and a --recursive option.

I really don't like making piping and less and complex interfaces part of the tool.

Also agree. I just listed it as an alternative.

---

Anyway, unless anyone else has relevant comments, I suggest Ivan decides the spec for this new command based on all the comments above, so we can get it onto a dev sprint.

[jorgeorpinel]

p.s. my (initial) updated spec proposal is:

usage: dvc list [-h] [-q | -v] [--recursive] [--rev REV | --versions]
                url [target [target ...]]

positional arguments:
  url         URL of Git repository with DVC project to download from.
  target      Paths to DVC-files or directories within the repository to list outputs
              for.