Prompt or not to prompt?

[dmpetrov]

EDITED 9/15/19

There are a couple of discussions around prompting for user input from CLI. This issue-question tries to summarize the rules around prompting.

Now, DVC asks y/n questions from the command line. We have to introduce --yes option (which is not a problem) and users should remember to use this option from their scripts all the time otherwise script stuck. This difference (script experience versus user experience) is an important issue for Unix tools.

Traditional Unix-tools do not ask questions (and do not break the scripting experience) which leads to overwriting files or aborting commands. There is an opinion that DVC should behave the same way. The backside of this - it might lead to data loss.

Examples:

1. [Overwriting] cp, tar - will overwrite files with
2. [Aborting] git checkout - fails if there is a conflict
3. [Prompt] there are some exceptions with prompt: unzip, apt-get. However, zip is not a traditional Unix tool, it was ported from DOS/Windows and follows Windows rules.

First, we need to decide if prompting is okay. Second, what are the set of rules for DVC for prompting, overwriting and aborting command?

The first rules summarization from @shcheklein #2497 (comment)

This is my take on abort vs prompt:

1. They both give overhead when you use them in scripts. You have to use some CLI argument to force.
2. If script is not written right (without set -eux) abort is less safer choice.
3. If script is written right, prompt is more annoying since script hangs.
4. The biggest difference for us in terms of workflow and usability comes from some long-running operations (as opposed to Git which is very fast in general). In our case it might be the case that we calculate status and/or md5 before even showing the prompt. Abort does not give a chance to continue - you have to run it second time.

Let's discuss the rules.

We can use conventions as guidelines, but in the end, it is important that we consider the usefulness of prompting in our own terms.

Also, there's no consistency on how to skip those questions (--force vs --yes vs --overwrite-dvcfile).

A quick grep for prompt showed that we ask for confirmation on the following methods:

• safe_remove: "File is going to be removed. Are you sure you want to proceed? [y/n]
• _collect_used_dir_cache: "Missing cache for directory. Cache for files inside will be lost. Would you like to continue? [y/n/]"
• gc: "This command ... (is dangerous). Are you sure you want to proceed? [y/n]"
• _is_outs_only: "Are you sure you want to remove dvcfile with its outputs? [y/n]" (on dvc remove --purge)
• destroy: "This will destroy all info about pipelines... yada yada. Are you sure you want to continue? [y/n]
• check_can_commit: "dependencies changed. Are you sure you commit it? [y/n]" (:warning: there's a typo here)
• stage.create: "dvcfile already exists. Do you wish to run the command and overwrite it? [y/n]"
• stage.reproduce: "Going to reproduce dvcfile. Are you sure you want to continue? [y/n]" (only if --interactive)

From those prompts, I can point out the following:

• All the prompts are "guarding" the user from potential data loss
• Some prompts are on commands that are destructive by itself (I mean, don't expect dvc destroy or dvc remove to be safe)
• I'm not sure if the user can predict when the command will result in some data loss
  • At least for me, it is still not that straight forward to think that dvc run -o foo will remove foo
  • Some of those prompts happen in specific cases (the ones if private methods)
  • What is data loss? Sometimes, files are in the cache but there's no reference (dvcfile) to them, is this data loss?

With that being said:

• If we are going to remove the prompts, we should make the commands more predictable (better documentation, different naming, different levels of verbosity, etc.)
  • Tools were you can predict the behavior are safer than the ones that you can't (at least, you develop trust)
• I like dvc's flexibility, but I can't imagine "designing" a tool for a lot of purposes and environments.
• It shouldn't take us a lot to go through all the commands and discuss them.
• It would help to identify common use cases for it (scripts, files sizes, workflows, etc.)

[shcheklein]

@dmpetrov @MrOutis nice summaries, thanks!!

Just a few comments from the top of my head:

and do not break the scripting experience

To my mind and in my experience aborting is very similar to prompting in terms of scripts. You have to take that into account and remember to provide an option. Aborting can be even more dangerous because people tend to forget to put set -uex and script goes even if there was a problem in the middle.

there are some exceptions

heroku CLI is another example, even if you specify explicitly that you want to do something it will explain what is going on ask you to put a name of your project (repeating their Web UI) before destroying something for you.

[dashohoxha]

Now, DVC asks y/n questions from the command line. We have to introduce --yes option (which is not a problem) and users should remember to use this option from their scripts all the time otherwise script stuck.

For me it is not a problem if a command asks for confirmation (or is interactive), as long as this can be overridden by a --yes or --force option.

For example the command apt install in debian/ubuntu. If it has to download lots of files it asks you for confirmation (showing the list of packages that is going to be installed). But in a script you can called it like apt install --yes ....

It shouldn't take us a lot to go through all the commands and discuss them.

I also think that discussing the commands one by one is better than formulating some general (abstract) principles/rules and trying to follow them strictly.

[dmpetrov]

@dashohoxha I agree that we need to discuss commands one by one to make actionable decisions. However, we need to formulate basic principles to simplify the decision-making process. It should help to each of the team members to make and explain these UX decisions without any holy wars.

Data file importance

The biggest issue related to prompting is data loss. Importance of data files is different. In a high level, we can separate two data types by the importance:

1. critical data or data sources which DVC cannot recover. In some cases, data might be completely unrecoverable for users (no others copies).
2. not critical data or data derivatives/outputs which can be reproduced by DVC. But it might require some time and resources from a user.

In general, it is better not to lose any of the data types. However, it is okay to remove not critical data (data derivatives\outputs) in order to avoid prompting or simplifying UX (not to introduce additional options). Failing is always an option of keeping data.

Examples

DVC aligns a lot to Git and we expect users to have Git experience. It will be a mistake not to reuse the knowledge and behavior patterns that users have built with Git. In general, we should use Git UX as a guiding star when we make decisions about UX. Every time we don't agree on something we should ask ourselves a simple question - how it should look like if it becomes a part of a Git. If there is no clear answer, I'd recommend finding analogies with classical Unix commands such as ls, rm etc which most of our users should be familiar with.

Based on my understanding, Git avoids prompting as much as possible (see discussion in #2497). In some exceptional cases (merge, rebase), dialogs are needed. These exceptions are not just prompted questions, they are full dialogs with multiple questions.

[dmpetrov]

@MrOutis thank you for the list of methods when DVC ask questions. Is it possible to find DVC commands and scenarios/cases when it is happening?

[dashohoxha]

@dmpetrov The principles that you lay out are simple and reasonable, I completely agree with them.

It should help to each of the team members to make and explain these UX decisions without any holy wars.

My opinion is that developers (in general) are a little bit biased towards the software that they are building, and it might be difficult for them to look at the software from the point of view of a user that knows nothing about implementation. Once you know all the implementation details you can't become a novice again.

So, it might be useful if someone is selected as a UX expert, who decides about the details of user interface and user interaction. The others should try to convince him about their point of view, but at the end it is his decision that prevails. This would avoid any holy wars and would bring consistency.

yes, @dmpetrov ! I can come up with some scenarios, I'll try to do it soon

@dmpetrov , it is taking a while 😅 !

⚠️ this is a work in progress ⚠️

---

checkout a file without cache:

dvc init --no-scm
echo 'foo' > foo
dvc add foo
rm -rf .dvc/cache/d3/b07384d113edec49eaa6238ad5ff00
dvc checkout

# WARNING: Cache 'd3b07384d113edec49eaa6238ad5ff00' not found. File 'foo' won't be created.
#   0%|          |Checkout                                                                                                 0/1 [00:00<?,     ?file/s]
# file 'foo' is going to be removed. Are you sure you want to proceed? [y/n]

remove "redundant files" during a directory checkout:

dvc init --no-scm
mkdir directory
echo 'foo' > directory/foo
dvc add directory
echo 'bar' > directory/bar
dvc checkout

# 0%|          |Checkout                                                                                                 0/1 [00:00<?,     ?file/sfile 'directory/bar' is going to be removed. Are you sure you want to proceed? [y/n]

checkout a file that changed:

dvc init --no-scm
echo 'foo' > foo
dvc add foo
echo 'bar' > foo
dvc checkout
#   0%|          |Checkout                                                                                                 0/1 [00:00<?,     ?file/s]
# file 'foo' is going to be removed. Are you sure you want to proceed? [y/n]