import-url/update: add --no-download flag #3773

dtrifiro · 2022-07-18T15:39:32Z

Add documentation for new dvc import-url dvc update flags (--no-download)

related: iterative/dvc#8024 (per iterative/dvc#7918)

jorgeorpinel

Let's reorg/rewarp this under 69 cols (to avoid horizontal scrolling) and to match the order in the Options section. Committing:

content/docs/command-reference/import-url.md

content/docs/command-reference/update.md

content/docs/command-reference/import-url.md

content/docs/command-reference/update.md

jorgeorpinel · 2022-07-18T21:22:49Z

content/docs/command-reference/import-url.md

  finish the operation(s)); or if the target data already exist locally and you
  want to "DVCfy" this state of the project (see also `dvc commit`).


Not directly related to the change in question but the description of --no-exec could be better. For starters, maybe say something like "create a value-less import .dvc file"? (for contrast with --no-download which includes *some* values)

This last sentence (seen above) especially is pretty hard to understand. What is "to DVCfy"? 🤔

Also instead of "download everything later" it should be "import the data later" now, I think. Since there's a separate --no-download option now (so let's only use term download there). BUT it could state "implies --no-download".

p.s. this could be done in a separate PR but I think it's related enough to this change to address now.

p.p.s. should also apply to dvc import --no-exec.

I would get rid of the second part ("DVCfy..."). It's not clear what it's supposed to do, and the command will fail if a file with the same of what is being imported already exists locally.

jorgeorpinel

Suggestion on actual change

content/docs/command-reference/import-url.md

content/docs/command-reference/update.md

jorgeorpinel · 2022-07-18T22:54:42Z

Hey @dtrifiro another note for future reference: please open PRs to this repo directly on the upstream (no fork). See https://github.com/iterative/dvc.org/wiki/2.-Pull-requests-for-core-DVC-changes for context. Thanks!

dberenbaum · 2022-08-24T13:15:29Z

@dtrifiro Do you think it's realistic to get this drafted, reviewed, and merged this sprint?

content/docs/command-reference/import-url.md

dberenbaum · 2022-08-26T13:50:28Z

In case @jorgeorpinel didn't already mention it, please open future docs PRs from this repo instead of a fork. That way, the changes will be deployed into an example app when you open the PR and it's easy for people to see what the final docs will look like. 🙏 No need to bother for this PR.

content/docs/command-reference/update.md

dberenbaum

@dtrifiro LGTM except for 2 minor comments.

@jorgeorpinel Do you want to take another look or leave it to me to approve/merge?

related: iterative/dvc#7918

dberenbaum · 2022-08-29T13:37:14Z

@jorgeorpinel I merged, but feel free to leave comments if you have them

jorgeorpinel

feel free to leave comments

A few, minor:

jorgeorpinel · 2022-09-07T06:47:55Z

content/docs/command-reference/import-url.md

+- `--no-exec` - create the import `.dvc` file but don't download `url` or get
+  checksums (assumes that the data source is valid). This is useful if you need


checksums - no context for this term/idea. Not mentioned anywhere in the Description. Not even "hashes" are mentioned.

Idea: could at least l'ink to https://dvc.org/doc/user-guide/project-structure/dvc-files#output-entries. And I'd prefer the term "file hash" but up to you.

jorgeorpinel · 2022-09-07T06:48:06Z

content/docs/command-reference/import-url.md

+  to define the project imports quickly, and import the data later (use
+  `dvc update` to finish the operation(s)).
+
+- `--no-download` - create the import `.dvc` with data checksums but without


import .dvc file

jorgeorpinel · 2022-09-07T06:49:14Z

content/docs/command-reference/import.md

 - `--no-exec` - create the import `.dvc` file but don't download `url` (assumes
  that the data source is valid). This is useful if you need to define the


Checksums not mentioned here (inconsistent?)

jorgeorpinel · 2022-09-07T06:54:02Z

content/docs/command-reference/import-url.md

 usage: dvc import-url [-h] [-q | -v] [-j <number>] [--file <filename>]
-                      [--no-exec] [--to-remote] [-r <name>]
+                      [--no-exec | --no-download] [--to-remote] [-r <name>]
                      [--desc <text>]
                      url [out]


This is now too long when rendered. See https://dvc.org/doc/command-reference/import-url

💅🏼 but let's regroup the args? Same for the other 2 files.

jorgeorpinel · 2022-09-07T06:54:55Z

content/docs/command-reference/import.md

+- `--no-download` - create the import `.dvc` with data checksums but without
+  downloading the associated data. This is useful if you need track changes in
+  remote data but do not (yet) need to download data to the local workspace.
+  Data can be later downloaded using `dvc pull`.


Should we mention update --no-download in these?

jorgeorpinel · 2022-09-07T06:55:41Z

content/docs/command-reference/update.md

-  with `-r`). Use `dvc pull` to get the data locally.
+- `--no-download` - Update data checksums in the `.dvc` file (`md5`, `etag`, or
+  `checksum` fields) without actually downloading the latest data. See
+  `dvc import-url --no-download`/`dvc import --no-download` for more context.


💅🏼

Suggested change

`dvc import-url --no-download`/`dvc import --no-download` for more context.

`dvc import-url --no-download` or `dvc import --no-download` for more context.

dtrifiro · 2022-09-08T09:04:29Z

Thanks for the feedback @jorgeorpinel, here's a new PR: main...import-url-no-download-fixes

we can move the discussion there

jorgeorpinel added A: docs Area: user documentation (gatsby-theme-iterative) C: ref Content of /doc/*-reference labels Jul 18, 2022

jorgeorpinel reviewed Jul 18, 2022

View reviewed changes

content/docs/command-reference/import-url.md Show resolved Hide resolved

content/docs/command-reference/import-url.md Show resolved Hide resolved

jorgeorpinel reviewed Jul 18, 2022

View reviewed changes

content/docs/command-reference/update.md Show resolved Hide resolved

jorgeorpinel reviewed Jul 18, 2022

View reviewed changes

content/docs/command-reference/import-url.md Outdated Show resolved Hide resolved

jorgeorpinel reviewed Jul 18, 2022

View reviewed changes

content/docs/command-reference/update.md Outdated Show resolved Hide resolved

jorgeorpinel reviewed Jul 18, 2022

View reviewed changes

content/docs/command-reference/import-url.md Outdated Show resolved Hide resolved

content/docs/command-reference/update.md Outdated Show resolved Hide resolved

jorgeorpinel added the ⌛ status: wait-core-merge Waiting for related product PR merge/release label Jul 26, 2022

jorgeorpinel removed the ⌛ status: wait-core-merge Waiting for related product PR merge/release label Aug 25, 2022

jorgeorpinel assigned dtrifiro Aug 25, 2022

pmrowla mentioned this pull request Aug 25, 2022

import-url: --no-download and dvc pull iterative/dvc#8172

Closed

dtrifiro force-pushed the import-url-no-download branch from 7e0d255 to 904c474 Compare August 25, 2022 17:05

dtrifiro marked this pull request as ready for review August 25, 2022 17:05

dtrifiro force-pushed the import-url-no-download branch 2 times, most recently from cd68df1 to d146a87 Compare August 25, 2022 17:08

dtrifiro commented Aug 25, 2022

View reviewed changes

content/docs/command-reference/import-url.md Outdated Show resolved Hide resolved

dberenbaum reviewed Aug 25, 2022

View reviewed changes

content/docs/command-reference/import-url.md Outdated Show resolved Hide resolved

dtrifiro force-pushed the import-url-no-download branch from d146a87 to 7faae7c Compare August 25, 2022 21:02

dberenbaum reviewed Aug 26, 2022

View reviewed changes

content/docs/command-reference/update.md Outdated Show resolved Hide resolved

dberenbaum reviewed Aug 26, 2022

View reviewed changes

content/docs/command-reference/update.md Outdated Show resolved Hide resolved

dberenbaum reviewed Aug 26, 2022

View reviewed changes

import/import-url/update: add --no-download flag

6061d63

related: iterative/dvc#7918

dtrifiro force-pushed the import-url-no-download branch from 7faae7c to 6061d63 Compare August 26, 2022 14:25

dberenbaum approved these changes Aug 29, 2022

View reviewed changes

dberenbaum merged commit 00958f7 into iterative:main Aug 29, 2022

dtrifiro deleted the import-url-no-download branch August 30, 2022 21:51

jorgeorpinel reviewed Sep 7, 2022

View reviewed changes

This was referenced Sep 12, 2022

import/import-url/update: --no-download improvements #3930

Merged

ref: reorg and wrap import usage block #3935

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

import-url/update: add --no-download flag #3773

import-url/update: add --no-download flag #3773

dtrifiro commented Jul 18, 2022 •

edited by jorgeorpinel

jorgeorpinel left a comment •

edited

jorgeorpinel Jul 18, 2022 •

edited

jorgeorpinel Jul 18, 2022 •

edited

dtrifiro Jul 21, 2022

jorgeorpinel left a comment

jorgeorpinel commented Jul 18, 2022

dberenbaum commented Aug 24, 2022

dberenbaum commented Aug 26, 2022

dberenbaum left a comment

dberenbaum commented Aug 29, 2022

jorgeorpinel left a comment

jorgeorpinel Sep 7, 2022

jorgeorpinel Sep 7, 2022

jorgeorpinel Sep 7, 2022

jorgeorpinel Sep 7, 2022

jorgeorpinel Sep 7, 2022

jorgeorpinel Sep 7, 2022

jorgeorpinel Sep 7, 2022

dtrifiro commented Sep 8, 2022

		finish the operation(s)); or if the target data already exist locally and you
		want to "DVCfy" this state of the project (see also `dvc commit`).

		- `--no-exec` - create the import `.dvc` file but don't download `url` or get
		checksums (assumes that the data source is valid). This is useful if you need

		- `--no-exec` - create the import `.dvc` file but don't download `url` (assumes
		that the data source is valid). This is useful if you need to define the

	`dvc import-url --no-download`/`dvc import --no-download` for more context.
	`dvc import-url --no-download` or `dvc import --no-download` for more context.

import-url/update: add --no-download flag #3773

import-url/update: add --no-download flag #3773

Conversation

dtrifiro commented Jul 18, 2022 • edited by jorgeorpinel

jorgeorpinel left a comment • edited

Choose a reason for hiding this comment

jorgeorpinel Jul 18, 2022 • edited

Choose a reason for hiding this comment

jorgeorpinel Jul 18, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel left a comment

Choose a reason for hiding this comment

jorgeorpinel commented Jul 18, 2022

dberenbaum commented Aug 24, 2022

dberenbaum commented Aug 26, 2022

dberenbaum left a comment

Choose a reason for hiding this comment

dberenbaum commented Aug 29, 2022

jorgeorpinel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dtrifiro commented Sep 8, 2022

dtrifiro commented Jul 18, 2022 •

edited by jorgeorpinel

jorgeorpinel left a comment •

edited

jorgeorpinel Jul 18, 2022 •

edited

jorgeorpinel Jul 18, 2022 •

edited