dandi ls with assets filter returns dandiset metadata

[satra]

two things:

this command dandi ls --metadata assets DANDI:000109/0.210914.1904 returns dandiset metadata

also since assets require the recursive flag, perhaps that should be automatically added for all and assets or some indication given to the user to use it.

[jwodder]

@satra Despite what @yarikoptic said in Slack, --metadata isn't really a filter; its only effect is that, if it's set to all or assets (the only other allowed value being api, the default), then, when listing assets, their metadata will be included instead of just showing what the pagination endpoint returns. The original specification was:

So let's go for an explicit --metadata api,all,dandiset,assets ? api -- just return what API returns for a given case, and that would be current behavior. all/dandiset/assets -- would query metadata (for the dandiset or an asset) if no metadata in the corresponding record.

(dandiset was not implemented as it would not make a difference.)

@yarikoptic Given that, exactly what should be changed for this issue?

[satra]

@jwodder - thanks for clarifying.

in that case perhaps we can add a dandi ls --include all, assets, dandiset (default) with --metadata doing the same as before, but reduced to a flag without options.

can you clarify what the lack of --metadata currently means for a dandiset? and whether dandi ls can be used to list an endpoint itself (i couldn't figure out based on trying). if not, then perhaps the above formulation is better, and we could add the ability to query an API endpoint

dandi ls https://api.dandiarchive.org/ or DANDI or https://dandiarchive.org would list all dandisets by default, assets if --recursive or --include assets is added, and their metadata if --metadata is added. wdyt?

[jwodder]

@satra

perhaps we can add a dandi ls --include all, assets, dandiset (default)

What exactly would this option do?

can you clarify what the lack of --metadata currently means for a dandiset?

Nothing. --metadata only affects what gets displayed for assets.

and whether dandi ls can be used to list an endpoint itself

What do you mean by "list an endpoint"?

dandi ls https://api.dandiarchive.org/ or DANDI or https://dandiarchive.org would list all dandisets by default, assets if --recursive or --include assets is added, and their metadata if --metadata is added. wdyt?

The code is really not structured for taking URLs that don't point to specific Dandisets or their assets, and listing every asset for every Dandiset strikes me as a query-intensive operation that we wouldn't want to natively support.

[satra]

--metadata only affects what gets displayed for assets.

since this is the case, then metadata is not really universal, so let's make it universal

dandi ls <DS> could return this info https://api.dandiarchive.org/api/dandisets/<DS>/
dandi ls <DS> --metadata could return the metadata of all versions
dandi ls <DS>/<version> could return the info of the specific version
dandi ls <DS>/<version> --metadata could return the metadata of specific version

if --include was switched to assets then it would return the asset info in the same way.

[jwodder]

@yarikoptic Thoughts?

[yarikoptic]

uff, took me awhile to stop deliberating on this, sorry about that. I see following dimensions we would like to "expose" to the user to tune

• D1. optional /<version>- AFAIK we already support that and obviously only for URLs, not for local files. Version is part of the URL specification. Nothing to be done for that AFAIK.
  • BUT there is a desire to be able to list versions of the dandiset (Make ls https://dandiarchive.org/dandiset/{dandisetid}/ list version records and not a specific (draft or released) one #236)
  • UC1: Also @satra did express (don't remember if there is an issue) desire to be able to list all dandisets for a given location (server, but why not path(s)?)
• D2. target "object"s of interest - dandiset, assets, both of those.
  • "both" is confusing and side-effect of having dandiset.yaml locally (just a file, can extract metadata) and no such thing on the server. So, if we work with a server/URL, there is no "asset" dandiset.yaml.
  • I never saw need in "both", but due to dandiset.yaml being a file, it was included in listing when pointed to
  • D2.1. sub-dimension -- "assets under the prefix" (/assets/ end point) vs "assets and folders in the directory" (/assets/paths/ end point; desire to support - ls without -r on a folder (local or remote) should list only immediate assets and folders #617) .
• D3. "API" response vs metadata for the object"s of interest. I mention "API" because we could do similar responses for local files as well -- would just lack asset_id and blob IDs

ATM the last two are somewhat conflated in -r and --metadata options, but lack expressiveness to distinguish /assets/ vs /assets/paths/ etc. So we indeed should come up with two dedicated for D2 and D3. What about

• D3: --what attrs,metadata,versions -- for "attributes" vs "metadata" querying + if "versions" - list available versions if object is dandiset. Eventually might even list which versions of the dandiset a given asset/path is present in, but for now some kind of Error.

For D2 I am thinking on either it is worth to separate out "assets under the prefix" vs "assets and folders in the directory"....

• --recurse none -- (default) information on specific object (dandiset, folder, asset) -- whatever argument points to. Would not be available for --what metadata or version and pointing to a folder prefix (at least for now, might later e.g. metadata could provide "overlap" of metadata fields for all assets under if --recurse assets).
• --recurse assets -- "assets under the prefix", so no dandiset or folders of any kind -- just listing of all the assets
• --recurse path -- "assets and folders in the directory" -- the /assets/paths as in the web UI.
• --recurse dandisets -- "dandisets" under that path (discover dandisets and display information on them)

Notes:

• If nothing of the desired type found under the specified path (e.g. --recurse dandisets but URL or folder points within dandiset) -- nothing is displayed
• I thought to allow to "mix" multiple --recurse values, e.g. --recurse none,assets while pointing to dandiset to provide current behavior of showing both assets and dandiset, but I think it is not really needed/useful, so let's not bother.

Note: we stop treating dandiset.yaml as an asset locally for the purpose of ls. It will just be skipped a file without any extra metadata.

I think with above --what and --recurse combination we should cover all the desired so far use cases, besides the UC1 but IMHO it is more of the "target" specification aspect, and would just need --recurse dandiset to make it explicit either want to see info about server vs dandisets in it...

Or am I missing any or you see better options/dimensions @satra @jwodder and other @dandi/dandi-cli developers?

edit : Use cases

List dandisets on the instance of the archive

will provide information only about most current version

dandi ls --recurse dandisets dandi://dandi

List assets and folders (without metadata) under the path

dandi ls -f json_pp --recurse path dandi://dandi/000288@draft/sub-H19.03.316/

List all assets under that folder or below (without metadata)

dandi ls -f json_pp --recurse assets dandi://dandi/000288@draft/sub-H19.03.316/

List all assets under that folder or below (metadata records)

dandi ls -f json_pp --recurse assets --what metadata dandi://dandi/000288@draft/sub-H19.03.316/

[satra]

/server/dandisets/versions/assets/ or /dandisets/assetpaths

if we consider that both server and local are essentially realizations of the same api, then ls can simply be routed through the api concepts dandisets, dandisets/versions (for local the version will be fixed), and assets. the assets can be treated either how we do in dandi and how aws s3 treats them, as keys with a separator, regex, etc.,.

i would like to see if we can avoid conflicts where we don't have to inform the user that two options cannot be presented together.

i think it would be nice if we could support globs like s5cmd (https://github.com/peak/s5cmd/blob/master/README.md#beast-mode-s5cmd) and that our api supports them.

[yarikoptic]

i would like to see if we can avoid conflicts where we don't have to inform the user that two options cannot be presented together.

I would like that too, but I also would like to not complicate CLI/Python API with some artificial construct to replace options etc. Could you provide example invocations for what you envision -- I could not quite grasp from your description.

i think it would be nice if we could support globs like s5cmd (https://github.com/peak/s5cmd/blob/master/README.md#beast-mode-s5cmd) and that our api supports them.

server API or this library API? in any case -- probably worth a separate issue with some details/vision in the corresponding (or both) components. But it all might depend on what we come up with here indeed ;)

[yarikoptic]

@jwodder - what CLI/python interface you see to cover all desired use cases?

[jwodder]

Assigning to @yarikoptic for design; assign to me for implementation.

[yarikoptic]

@jwodder, I have adjusted my comment above with desired values for --what and --recurse which would constitute the API/CLI for ls. Positional argument(s) could point to either paths or URLs, but not a mix to avoid messy outputs etc. Prior behavior of -r showing both dandiset and asset(s) doesn't need to be preserved anyhow.

Let me know if anything needs to be clarified.