clarify resource identifier docs #1437

asmacdo · 2024-04-26T20:51:58Z

I pulled the "primer" from https://dandi.readthedocs.io/en/latest/ref/urls.html#resource-ids

Do you think that helps enough?
(I also added to dandi_ls since it reused the same resource id doc)

$ dandi download --help

Usage: dandi download [OPTIONS] [URL]...

  Download files or entire folders from DANDI.

  dandi commands and accept URLs and URL-like identifiers called
  <resource ids> in the following formats for identifying Dandisets, assets, and asset
  collections.

  Text in [brackets] is optional.  A server field is a base API or GUI URL for
  a DANDI Archive instance.  If an optional ``version`` field is omitted from
  a URL, the given Dandiset's most recent published version will be used if it
  has one, and its draft version will be used otherwise.

  Accepted resource identifier patterns:
   - DANDI:<dandiset id>[/<version>]
   - https://gui.dandiarchive.org/...
   - https://identifiers.org/DANDI:<dandiset id>[/<version id>] (<version id> cannot be 'draft')
   - https://<server>[/api]/[#/]dandiset/<dandiset id>[/<version>][/files[?location=<path>]]
   - https://*dandiarchive-org.netflify.app/...
   - https://<server>[/api]/dandisets/<dandiset id>[/versions[/<version>]]
   - https://<server>[/api]/assets/<asset id>[/download]
   - https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/<asset id>[/download]
   - https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/?path=<path>
   - https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/?glob=<glob>
   - dandi://<instance name>/<dandiset id>[@<version>][/<path>]
   - https://<server>/...
      

Options:
  -o, --output-dir DIRECTORY      Directory where to download to (directory
                                  must exist). Files will be downloaded with
                                  paths relative to that directory.
  -e, --existing [error|skip|overwrite|overwrite-different|refresh]
                                  What to do if a file found existing locally.
                                  'refresh': verify that according to the size
                                  and mtime, it is the same file, if not -
                                  download and overwrite.  [default: error]
  -f, --format [pyout|debug]      Choose the format/frontend for output. TODO:
                                  support all of the ls
  --path-type [exact|glob]        Whether to interpret asset paths in URLs as
                                  exact matches or glob patterns  [default:
                                  exact]
  -J, --jobs N[:M]                Number of parallel download jobs and,
                                  optionally number of subjobs per Zarr asset
                                  [default: 6]
  --download [dandiset.yaml,assets,all]
                                  Comma-separated list of elements to download
                                  [default: all]
  --sync                          Delete local assets that do not exist on the
                                  server
  -i, --dandi-instance TEXT       DANDI Archive instance to download from. If
                                  any URLs are provided, they must point to
                                  the given instance. If no URL is provided,
                                  and there is a local dandiset.yaml file, the
                                  Dandiset with the identifier given in the
                                  file will be downloaded from the given
                                  instance.  [env var: DANDI_INSTANCE]
  --help                          Show this message and exit.

Fixes: #dandi#1435

codecov · 2024-04-26T20:54:38Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 62.59%. Comparing base (15196a9) to head (d3d4ec3).

❗ Current head d3d4ec3 differs from pull request most recent head 58c7c4c. Consider uploading reports for the commit 58c7c4c to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #1437       +/-   ##
===========================================
- Coverage   88.54%   62.59%   -25.95%     
===========================================
  Files          77       77               
  Lines       10561    10563        +2     
===========================================
- Hits         9351     6612     -2739     
- Misses       1210     3951     +2741

Flag	Coverage Δ
unittests	`62.59% <100.00%> (-25.95%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

asmacdo · 2024-04-26T20:57:01Z

IMO with the explanation we could now trim that list way back to the common cases and include a linking to https://dandi.readthedocs.io/en/latest/ref/urls.html#resource-ids

dandi/dandiarchive.py

dandi/cli/cmd_download.py

asmacdo · 2024-04-27T00:04:52Z

Added examples and cleaned it up a bit.

Usage: dandi download [OPTIONS] [URL]...

  Download files or entire folders from DANDI.

  RESOURCE ID/URLS:

      dandi commands accept URLs and URL-like identifiers called <resource
      ids> in the following formats for identifying Dandisets, assets, and
      asset collections.

      Text in [brackets] is optional.  A server field is a base API or GUI URL
      for a DANDI Archive instance.  If an optional ``version`` field is
      omitted from a URL, the given Dandiset's most recent published version
      will be used if it has one, and its draft version will be used
      otherwise.

  Accepted resource identifier patterns:
   - DANDI:<dandiset id>[/<version>]
   - https://gui.dandiarchive.org/...
   - https://identifiers.org/DANDI:<dandiset id>[/<version id>] (<version id> cannot be 'draft')
   - https://<server>[/api]/[#/]dandiset/<dandiset id>[/<version>][/files[?location=<path>]]
   - https://*dandiarchive-org.netflify.app/...
   - https://<server>[/api]/dandisets/<dandiset id>[/versions[/<version>]]
   - https://<server>[/api]/assets/<asset id>[/download]
   - https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/<asset id>[/download]
   - https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/?path=<path>
   - https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/?glob=<glob>
   - dandi://<instance name>/<dandiset id>[@<version>][/<path>]
   - https://<server>/...



  EXAMPLES:

  # Download only the dandiset.yaml

  dandi download --download dandiset.yaml DANDI:000027

  # Download only dandiset.yaml if there is a newer version

  dandi download https://identifiers.org/DANDI:000027 --existing refresh

  # Download only the assets

  dandi download --download assets DANDI:000027

  # Download all from a specific version

  dandi download DANDI:000027/0.210831.2033

Options:
  -o, --output-dir DIRECTORY      Directory where to download to (directory
                                  must exist). Files will be downloaded with
                                  paths relative to that directory.
  -e, --existing [error|skip|overwrite|overwrite-different|refresh]
                                  What to do if a file found existing locally.
                                  'refresh': verify that according to the size
                                  and mtime, it is the same file, if not -
                                  download and overwrite.  [default: error]
  -f, --format [pyout|debug]      Choose the format/frontend for output. TODO:
                                  support all of the ls
  --path-type [exact|glob]        Whether to interpret asset paths in URLs as
                                  exact matches or glob patterns  [default:
                                  exact]
  -J, --jobs N[:M]                Number of parallel download jobs and,
                                  optionally number of subjobs per Zarr asset
                                  [default: 6]
  --download [dandiset.yaml,assets,all]
                                  Comma-separated list of elements to download
                                  [default: all]
  --sync                          Delete local assets that do not exist on the
                                  server
  -i, --dandi-instance TEXT       DANDI Archive instance to download from. If
                                  any URLs are provided, they must point to
                                  the given instance. If no URL is provided,
                                  and there is a local dandiset.yaml file, the
                                  Dandiset with the identifier given in the
                                  file will be downloaded from the given
                                  instance.  [env var: DANDI_INSTANCE]
  --help                          Show this message and exit.

Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>

yarikoptic · 2024-05-01T14:30:41Z

dandi/cli/cmd_download.py

+EXAMPLES: \n
+# Download only the dandiset.yaml \n
+dandi download --download dandiset.yaml DANDI:000027 \n
+
+# Download only dandiset.yaml if there is a newer version \n
+dandi download https://identifiers.org/DANDI:000027 --existing refresh
+
+# Download only the assets \n
+dandi download --download assets DANDI:000027
+
+# Download all from a specific version \n
+dandi download DANDI:000027/0.210831.2033
+
+# Download a specific file or directory \n
+dandi download dandi://DANDI/000027@0.210831.2033/sub-RAT123/sub-RAT123.nwb


what could trailing empty spaces would do besides in rare cases possibly negatively effecting wrapping?

also why bother with extra new line?

best to provide specific example for a directory since needs to end with / IIRC (please check that works)

Suggested change

EXAMPLES: \n

# Download only the dandiset.yaml \n

dandi download --download dandiset.yaml DANDI:000027 \n

# Download only dandiset.yaml if there is a newer version \n

dandi download https://identifiers.org/DANDI:000027 --existing refresh

# Download only the assets \n

dandi download --download assets DANDI:000027

# Download all from a specific version \n

dandi download DANDI:000027/0.210831.2033

# Download a specific file or directory \n

dandi download dandi://DANDI/000027@0.210831.2033/sub-RAT123/sub-RAT123.nwb

EXAMPLES:\n

# Download only the dandiset.yaml

dandi download --download dandiset.yaml DANDI:000027

# Download only dandiset.yaml if there is a newer version

dandi download https://identifiers.org/DANDI:000027 --existing refresh

# Download only the assets

dandi download --download assets DANDI:000027

# Download everything for a specific version of a dandiset

dandi download DANDI:000027/0.210831.2033

# Download a specific directory

dandi download dandi://DANDI/000027@0.210831.2033/sub-RAT123/

# Download a file

dandi download dandi://DANDI/000027@0.210831.2033/sub-RAT123/sub-RAT123.nwb

what could trailing empty spaces would do besides in rare cases possibly negatively effecting wrapping?

I just used them to separate the \n for human reading, but I dont feel strongly. Removed in latest commit.

also why bother with extra new line?

I'm not sure why, but a single newline is skipped, maybe something to do with the fstring?

dandi-cli/dandi/cli/cmd_download.py

Lines 15 to 16 in 15196a9

# The use of f-strings apparently makes this not a proper docstring, and so

# click doesn't use it unless we explicitly assign it to `help`:

Your suggestion renders like this:

EXAMPLES: # Download only the dandiset.yaml dandi download --download dandiset.yaml DANDI:000027 # Download only dandiset.yaml if there is a newer version dandi download https://identifiers.org/DANDI:000027 --existing refresh # Download only the assets dandi download --download assets DANDI:000027 # Download everything for a specific version of a dandiset dandi download DANDI:000027/0.210831.2033 # Download a specific directory dandi download dandi://DANDI/000027@0.210831.2033/sub-RAT123/ # Download a file dandi download dandi://DANDI/000027@0.210831.2033/sub-RAT123/sub-RAT123.nwb

Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>

yarikoptic · 2024-05-01T18:09:46Z

Coolio, thank you!

github-actions · 2024-05-03T13:57:32Z

🚀 PR was released in 0.62.0 🚀

clarify resource identifier docs

9637d0e

Fixes: #dandi#1435

yarikoptic reviewed Apr 26, 2024

View reviewed changes

dandi/dandiarchive.py Outdated Show resolved Hide resolved

yarikoptic reviewed Apr 26, 2024

View reviewed changes

dandi/cli/cmd_download.py Show resolved Hide resolved

Add examples

d8bae4d

Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>

asmacdo force-pushed the clarify-download-docs branch from 7702002 to d8bae4d Compare April 27, 2024 00:14

line spacing

16c41b6

asmacdo force-pushed the clarify-download-docs branch from 1197cb0 to 16c41b6 Compare April 27, 2024 00:23

Add specific file example

3afd99b

yarikoptic requested changes May 1, 2024

View reviewed changes

Add example for directory

58c7c4c

Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>

asmacdo force-pushed the clarify-download-docs branch from d3d4ec3 to 58c7c4c Compare May 1, 2024 17:22

yarikoptic merged commit e5ecb8c into dandi:master May 1, 2024
16 of 24 checks passed

github-actions bot added the released label May 3, 2024

yarikoptic mentioned this pull request May 3, 2024

simplify identifier on dandi download cli command #1435

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

clarify resource identifier docs #1437

clarify resource identifier docs #1437

asmacdo commented Apr 26, 2024 •

edited

codecov bot commented Apr 26, 2024 •

edited

asmacdo commented Apr 26, 2024

asmacdo commented Apr 27, 2024 •

edited

yarikoptic May 1, 2024

asmacdo May 1, 2024

yarikoptic commented May 1, 2024

github-actions bot commented May 3, 2024

	# The use of f-strings apparently makes this not a proper docstring, and so
	# click doesn't use it unless we explicitly assign it to `help`:

clarify resource identifier docs #1437

clarify resource identifier docs #1437

Conversation

asmacdo commented Apr 26, 2024 • edited

codecov bot commented Apr 26, 2024 • edited

Codecov Report

asmacdo commented Apr 26, 2024

asmacdo commented Apr 27, 2024 • edited

yarikoptic May 1, 2024

Choose a reason for hiding this comment

asmacdo May 1, 2024

Choose a reason for hiding this comment

yarikoptic commented May 1, 2024

github-actions bot commented May 3, 2024

asmacdo commented Apr 26, 2024 •

edited

codecov bot commented Apr 26, 2024 •

edited

asmacdo commented Apr 27, 2024 •

edited