executor: setup and configure executors

[dberenbaum]

Meta-issue. Please link related tickets back to this one.

User story

I want to spin up an executor and tear it down.

Requirements

• Define Terraform resources in DVC config files
• Create, update, and destroy resources from within DVC

Tasks

Config-only commands:

• dvc machine add (dvc machine: implement initial terraform support #6462)
• dvc machine remove (dvc machine: implement initial terraform support #6462)
• dvc machine list - (machine: implement list/modify configuration commands #6480)
• dvc machine modify - (machine: implement list/modify configuration commands #6480)

Terraform wrapper commands

• dvc machine create (dvc machine: implement initial terraform support #6462)
• dvc machine destroy (dvc machine: implement initial terraform support #6462)
• dvc machine rename (machine: support renaming machines #6481)
• dvc machine status - (machine: list status of created instances #6482)
• dvc machine ssh - (machine: provide SSH shell access to created instances #6483)

[dberenbaum]

Terraform resources should be configurable in .dvc/config files - including support for system, user, etc. configuration.

[casperdcl]

related: iterative/terraform-provider-iterative#164 (comment)

[dberenbaum]

I'm a little concerned that we are using up the namespaces for machine, especially list and status. We will need some command that outputs table-like info about each machine (similar to docker ps): what machines have been created, how long have they been running, are they idle or running an experiment, etc. Where will that go?

[pmrowla]

I think anything that is aware of whether or not some instance is running experiments or not would go under dvc exp. machine list/status are more like docker machine ls, which just tells you about machines that have been created and/or booted (and the separation between list and status is that list shows .dvc/config entries, and status shows terraform status). machine is currently not aware of anything that is actually running on the instance itself.

docker ps displays the state of deployed docker containers that are running on some machine (that may or may not have been created with docker machine). This is more like the actual "executor" concept in DVC (i.e. keeping track of running/deployed experiments, which is separate from terraform/machine instance status). It seems like we will eventually just want a separate machine ps or exp ps command that would be the equivalent to docker ps and shows the state of running experiments - including whether or not a potential executor is idle.

[pmrowla]

I guess it's possible that we may end up not needing machine status (in favor of the ps command), but IMO there is a distinct difference between "knowing the IP and boot state of a terraform instance" vs "whether or not DVC experiment related processes are idle on some named 'DVC executor' instance" (and I think having 2 separate commands here makes sense)

We could also potentially merge list and status into a single (DVC config) machine list/ls where we would have some CLI output that lists the .dvc/config entries for machines and then nests the terraform boot/IP/etc status information underneath each DVC-config entry

But this all seems like UI decisions that can be made further down the line once we actually have the underlying functionality working

[karajan1001]

docker has docker images to list images and docker ps to list containers. Here machine status only exists when machine was created. So more close to the container than the images?