[RFE] Support multiple instances

[jorgemoralespou]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
FEATURE REQUEST

As a developer I work on many different projects and I have a need to easily switch context from one to the other. Using minikube/minishift is great for testing on top of kubernetes/openshift but does not allow for this context switching.

I'm here requesting that minikube could support multiple instances so I can start/stop different instances depending on the scope of work I'm doing. Also, even the cost of initialization is not too high (with good internet connection), every startup pulls down some containers to work with.
Also, it usually happens that different projects might have different configurations, or applications deployed, and I want to start a minikube instance with just the projects I need (or those that are related to the work I'm doing).

The request would introduce a profile (name not relevant) name to the start process, so that I can have few named machines. If no profile specified, applying default profile.

minikube start # starting default minikube
minikube stop
minikube start project_a
minikube stop
minikube start k8s-1.5-test --config .... # Configuration applied to the VM
minikube stop
minikube start k8s-1.5-test # Configuration already applied, no need to reconfigure
minikube status # Log Active cluster: k8s-1.5-test
minikube stop
minikube delete k8s-1.5-test
minikube list # List of profiles

This is very convenient for developers that tend to switch context very often, but also for many other use cases.

I did a writeup some time back (for a different tool) but the goals are somehow defined there.
http://jorgemoral.es/2016/10/developing-locally-with-openshift/

cc/ @jimmidyson

[dlorenc]

Thanks for the request! I think this is a duplicate of #370, but you have much more of a writeup here so I'll close that one.

[waprin]

I ran into this issue when looking for different minikube contexts. It's true multiple clusters is still a good FR and there's valid use cases for it, but worth nothing for some of those cases Namespaces are probaby good enough.

kubectl create namespace dev
kubectl config set-context minikube --namespace=dev

Now you are in the dev namespace and can have isolated Pods/Services etc in the same cluster.

[jorgemoralespou]

@waprin the main problem with one cluster with multiple namespaces is that you'll be using the resources defined in all the namespaces in the cluster. For development purposes you'll probably prefer to have multiple clusters, which are really fast to start/switch, and keep resource consumption (cpu/memory) to a minimum.
The problem this issue tries to address is not solved by namespacing the resources/deployments. This has always been available and an option.

[waprin]

@jorgemoralespou noted, agreed there are valid use cases for this (another one might be testing cluster federation stuff). Just wanted to note/remind that namespaces existed because I got redirected to this issue, when namespaces were really good enough for what I wanted.

[hferentschik]

@aaron-prindle have you any plans on how this is going to look at? In Minishift we also have the feature request to support multiple VMs - minishift/minishift#126
We should try to align the behavior or even preferably share the implementation. WDYT?

[aaron-prindle]

@hferentschik It definitely would be great to align on our implementation. Right now the simplest way to do this would be to add a name parameter to minikube (minikube start --name foo, minikube logs --name foo), similar to docker-machine and then store a separate .minikube folder for each VM initialized (.minikube-foo) We could also switch to instead of storing flat configuration, stores lists of configurations and then we could keep a single .minikube directory with all the machines. @r2d4 any opinions on this?

[r2d4]

Honestly, my only issue with this is now every command needs a qualified machine name. We already support the MINIKUBE_HOME env. Is that solution not sufficient?

If not, we might want to use something like minikube config set name to set this option for all commands. I think whatever the design is, its important to

1. be able to run all commands without qualifying machine name, i.e. no need for --name on every command
2. be obvious which cluster you're running the command on

[coolbrg]

I liked the idea of @r2d4 for minikube config set name. I am also thinking of minikube switch name with having a flag --name in start command like @aaron-prindle suggested.

1. be able to run all commands without qualifying machine name, i.e. no need for --name on every command
2. be obvious which cluster you're running the command on

👍

[hferentschik]

It definitely would be great to align on our implementation.

+1, this is a quite big change from a ux experience. It would confuse users if they would have to deal with different approaches.

Right now the simplest way to do this would be to add a name parameter to minikube (minikube start --name foo

+1 Right, that would take care of how to create an instance with a given name.

minikube logs --name foo), similar to docker-machine

I think I have the same ideas here as @r2d4. Why not writing the name of the instance you started into the config (transparently). Then all commands executed are against this instance. A bit like you can switch context with kubectl or oc. I think we should, however, have a dedicated [minikube|minishift] select <instance-name> to make this feature more explicit.

Also, I think it might still make sense to have the '--name' option on a global flag name as well. Without specifying anything the command runs against the currently selected instance, but one can also run a once off command against another instance using --name (opposed to having to switch instance forth and back).

We could also switch to instead of storing flat configuration, stores lists of configurations and then we could keep a single .minikube directory with all the machines.

+1 That would be my preference. Even though it might be a bit harder.

With this approach will also have to deal with moving some parts around in the MINIKUBE_HOME directory. Atm, the libmachine VMs are already in their own namespace - ~/.minikube/machines/minikube. It is just that we have only a single VM and we use a hard coded constant for that. If this became configurable, then the VMs are alreday separated. However, one also needs to take care of teh config and log directory in ~/.minikube/. They need to be scoped per instance as well. So it's quite a bit of refactoring required here :-(

[aaron-prindle]

Closing. This functionality was added in #1320 and will be available in the next minikube release. Example use can be seen here: #1320 (comment)

[alahijani]

I can't seem to be able to use it on Windows with hyperv. In order to start the second minikube I have to first stop the first one. Otherwise I get errors like this:

E0524 16:05:15.400121    9068 start.go:117] Error starting host: Temporary Error: Error configuring auth on host: Too
many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded.
 Retrying.
E0524 16:05:17.400669    9068 start.go:123] Error starting host:  Temporary Error: Error configuring auth on host: Too
many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded
Temporary Error: Error configuring auth on host: Too many retries waiting for SSH to be available.  Last error:
Maximum number of retries (60) exceeded

[jujes]

@aaron-prindle v0.20.0 include this feature?
I mean, this script must work?? (two nodes → master and nimion)

minikube start \
    --name=master \
    --profile=master \
    --host-only-cidr=192.168.00.1/24 \
    --dns-domain=master.local \
    --cpus=1 \
    --disk-size="10g" \
    --kubernetes-version="v1.7.0" \
    --memory=1024 \
    --vm-driver=virtualbox

minikube start \
    --profile=nimion-00 \
    --keep-context \
    --cpus=1 \
    --disk-size="10g" \
    --kubernetes-version="v1.7.0" \
    --memory=1024 \
    --vm-driver=virtualbox