Improve checks for existing machine during 'minishift start'

[hferentschik]

When executing 'minishift start' a check for an existing VM occurs. If a machine exists, minishift tries to resume it. This causes problems when the user actually wants to start a new minishift instance using other CLI options, most notably a different driver.

Checks around this should be improved.

[hferentschik]

The error looks somewhat like this:

C:\Users\cdk\Desktop>minishift.exe start
Starting local OpenShift cluster...
E1206 11:59:36.234074     748 start.go:117] Error starting host: Error getting state for host: Failed to find the VM     status. Retrying.
E1206 11:59:36.816459     748 start.go:117] Error starting host: Error getting state for host: Failed to find the VM     status. Retrying.
E1206 11:59:37.355990     748 start.go:117] Error starting host: Error getting state for host: Failed to find the VM     status. Retrying.
E1206 11:59:37.355990     748 start.go:123] Error starting host:  Error getting state for host: Failed to find the     VM status
Error getting state for host: Failed to find the VM status
Error getting state for host: Failed to find the VM status

[hferentschik]

The error looks somewhat like this:

C:\Users\cdk\Desktop>minishift.exe start
Starting local OpenShift cluster...
E1206 11:59:36.234074     748 start.go:117] Error starting host: Error getting state for host: Failed to find the VM     status. Retrying.
E1206 11:59:36.816459     748 start.go:117] Error starting host: Error getting state for host: Failed to find the VM     status. Retrying.
E1206 11:59:37.355990     748 start.go:117] Error starting host: Error getting state for host: Failed to find the VM     status. Retrying.
E1206 11:59:37.355990     748 start.go:123] Error starting host:  Error getting state for host: Failed to find the     VM status
Error getting state for host: Failed to find the VM status
Error getting state for host: Failed to find the VM status

[hferentschik]

A way to resolve this issue, would be to have dedicated 'create' and 'start' commands. This would make it easier to determine the cases and provide better error messages.

[hferentschik]

A way to resolve this issue, would be to have dedicated 'create' and 'start' commands. This would make it easier to determine the cases and provide better error messages.

[LalatenduMohanty]

What about adding a flag in minishift start to do a clean start i.e. it should delete the existing .minishift and start fresh.

[LalatenduMohanty]

What about adding a flag in minishift start to do a clean start i.e. it should delete the existing .minishift and start fresh.

[hferentschik]

What about adding a flag in minishift start to do a clean start i.e. it should delete the existing .minishift and start fresh

Sure, possible. However, in this case I find a dedicated command more explicit and self documenting.

[hferentschik]

What about adding a flag in minishift start to do a clean start i.e. it should delete the existing .minishift and start fresh

Sure, possible. However, in this case I find a dedicated command more explicit and self documenting.

[LalatenduMohanty]

Also it should ask for confirmation from user if he wants to delete existing .minishift and start fresh. The benefit of having it in start is that user does not have to remember another new command. I think from user experience point of view it is more appropriate as user does not want to create but want to start but as a fresh setup.

[LalatenduMohanty]

Also it should ask for confirmation from user if he wants to delete existing .minishift and start fresh. The benefit of having it in start is that user does not have to remember another new command. I think from user experience point of view it is more appropriate as user does not want to create but want to start but as a fresh setup.

[LalatenduMohanty]

I think minishift start --factory-reset or something similar will be more intuitive than create.

[LalatenduMohanty]

I think minishift start --factory-reset or something similar will be more intuitive than create.

[LalatenduMohanty]

@jimmidyson any suggestion?

[LalatenduMohanty]

@jimmidyson any suggestion?

[jimmidyson]

minishift clean sounds good to me, with confirmation from user. I think that start should exit if already started, not try to update running machine.

create is problematic as docker machine doesn't have that in library iirc.

[jimmidyson]

minishift clean sounds good to me, with confirmation from user. I think that start should exit if already started, not try to update running machine.

create is problematic as docker machine doesn't have that in library iirc.

[hferentschik]

This ties very much into another discussion we need to have. Do we want to support multiple OpenShift instances created and managed by minishift? Similar to what docker-machine does. If so, a create makes sense, since now we also need to take things like minishift instance name into account.

... but want to start but as a fresh setup.

which is create right. I think a distinction is very intuitive. IMO the current minishift start tries to mangle two things into one command.

minishift clean sounds good to me, with confirmation from user.

How does that differ from the current delete command?

I think that start should exit if already started, not try to update running machine.

So, if it is running and you run start again, nothing should happen. But what if the machine is stopped (not deleted) and a new start command is issued? I think that is the case which we see most often. The code seems to detect that there is a exiting stopped machine, tries to resume it and gets into trouble (at least I think this is what we are seeing. This needs a bit more investigation as well).

create is problematic as docker machine doesn't have that in library iirc.

We don't need necessarily a create from the underlying libmachine. The distinction between create and start is something we handle within minishift.

[hferentschik]

This ties very much into another discussion we need to have. Do we want to support multiple OpenShift instances created and managed by minishift? Similar to what docker-machine does. If so, a create makes sense, since now we also need to take things like minishift instance name into account.

... but want to start but as a fresh setup.

which is create right. I think a distinction is very intuitive. IMO the current minishift start tries to mangle two things into one command.

minishift clean sounds good to me, with confirmation from user.

How does that differ from the current delete command?

I think that start should exit if already started, not try to update running machine.

So, if it is running and you run start again, nothing should happen. But what if the machine is stopped (not deleted) and a new start command is issued? I think that is the case which we see most often. The code seems to detect that there is a exiting stopped machine, tries to resume it and gets into trouble (at least I think this is what we are seeing. This needs a bit more investigation as well).

create is problematic as docker machine doesn't have that in library iirc.

We don't need necessarily a create from the underlying libmachine. The distinction between create and start is something we handle within minishift.

[LalatenduMohanty]

@hferentschik I have seen some command lines have create and start as verbs and I do not find it intuitive as I always get confused.
Think from this prospective, minishift create will not do anything extra from minishift start apart from deleting the .minishift directory. It is not creating anything but deleting the .minishift directory. So it is not logical to call it create IMO.

I think clean is better than create as @jimmidyson suggested. As it will clean the .minishift directory. delete will delete the VM and clean will clean the directory.

[LalatenduMohanty]

@hferentschik I have seen some command lines have create and start as verbs and I do not find it intuitive as I always get confused.
Think from this prospective, minishift create will not do anything extra from minishift start apart from deleting the .minishift directory. It is not creating anything but deleting the .minishift directory. So it is not logical to call it create IMO.

I think clean is better than create as @jimmidyson suggested. As it will clean the .minishift directory. delete will delete the VM and clean will clean the directory.

[hferentschik]

I have seen some command lines have create and start as verbs and I do not find it intuitive as I always get confused.

How come? Personally I think it makes these two things actually explicit and easier to understand. I think docker-machine does it actually right there. create for me means build something from scratch, and start get something existing to run.

Think from this prospective, minishift create will not do anything extra from minishift start apart from deleting the .minishift directory.

Well, the essence of create is not to delete the existing machine. This is minishift delete. create is about taking the two steps start does and putting the creation part into create and only have the start part in the start command.

I think clean is better than create

clean is also needed and there is an issue for that, but it addresses another problem. Having clean is not going to improve the situation reported in this issue.

[hferentschik]

I have seen some command lines have create and start as verbs and I do not find it intuitive as I always get confused.

How come? Personally I think it makes these two things actually explicit and easier to understand. I think docker-machine does it actually right there. create for me means build something from scratch, and start get something existing to run.

Think from this prospective, minishift create will not do anything extra from minishift start apart from deleting the .minishift directory.

Well, the essence of create is not to delete the existing machine. This is minishift delete. create is about taking the two steps start does and putting the creation part into create and only have the start part in the start command.

I think clean is better than create

clean is also needed and there is an issue for that, but it addresses another problem. Having clean is not going to improve the situation reported in this issue.

[hferentschik]

Another advantage of a separation between clean and start would be that minishift could controls the use of the --use-existing-config flag for cluster up. During create it would be false, but during start you would set it to true. Otherwise it would be up to the user to manage this.

[hferentschik]

Another advantage of a separation between clean and start would be that minishift could controls the use of the --use-existing-config flag for cluster up. During create it would be false, but during start you would set it to true. Otherwise it would be up to the user to manage this.

[LalatenduMohanty]

I was actually talking a different context then what @hferentschik mentioned here. I was actually thinking about completely starting from fresh (i.e. without minishift ISO cached), hence the references to deleting .minishift directory.

[LalatenduMohanty]

I was actually talking a different context then what @hferentschik mentioned here. I was actually thinking about completely starting from fresh (i.e. without minishift ISO cached), hence the references to deleting .minishift directory.

[LalatenduMohanty]

@hferentschik Is there possibility that user can still get in to issues if he runs minishift delete before minishift start with a different driver?

[LalatenduMohanty]

@hferentschik Is there possibility that user can still get in to issues if he runs minishift delete before minishift start with a different driver?

[jorgemoralespou]

@hferentschik @LalatenduMohanty @jimmidyson What is your idea for the lifecycle of minishift vms?
I'm asking as I see a clean/create rather superfluous, of course, depending on how you implement things. I think that simplicity is really a must, and the less commands the better. I've seen this workflow with vagrant which has been very successful over time.

I like the fact that simple start/stop or up/down exist and these command with a delete/destroy should be sufficient. Creation and starting of a VM should be handle at the same time, and if the machine is new, then it should go through all the creation lifecycle (although not providing this as separate step).

In case a VM needs to be recreated, we should aim to be able to destroy/start with the less impact to the user. This ultimately means that:

• the iso needs to be cached (multiple isos can be stored and managed, like vagrant box create |list|delete)
• the images should also be cached on the host (at least provide an option to cache them)

If we can provide these 2 options, destroying and creating should always be easier to implement and safer in resulting configuration than any update you can do to an existing VM.

Also, if this is the case, what would a clean provide? Shouldn't that be something handled by "destroy" that would delete .minishift/

NOTE: I have included a in minishift directory as I think we should aim to have it as there are specific configuration that will be local to a VM (start params, config,...) and other that will be global to all VMs(isos, images,...).

Is there a proposal/design document with all the goals you want minishift to have and how you're thinking on addressing them?

[jorgemoralespou]

@hferentschik @LalatenduMohanty @jimmidyson What is your idea for the lifecycle of minishift vms?
I'm asking as I see a clean/create rather superfluous, of course, depending on how you implement things. I think that simplicity is really a must, and the less commands the better. I've seen this workflow with vagrant which has been very successful over time.

I like the fact that simple start/stop or up/down exist and these command with a delete/destroy should be sufficient. Creation and starting of a VM should be handle at the same time, and if the machine is new, then it should go through all the creation lifecycle (although not providing this as separate step).

In case a VM needs to be recreated, we should aim to be able to destroy/start with the less impact to the user. This ultimately means that:

• the iso needs to be cached (multiple isos can be stored and managed, like vagrant box create |list|delete)
• the images should also be cached on the host (at least provide an option to cache them)

If we can provide these 2 options, destroying and creating should always be easier to implement and safer in resulting configuration than any update you can do to an existing VM.

Also, if this is the case, what would a clean provide? Shouldn't that be something handled by "destroy" that would delete .minishift/

NOTE: I have included a in minishift directory as I think we should aim to have it as there are specific configuration that will be local to a VM (start params, config,...) and other that will be global to all VMs(isos, images,...).

Is there a proposal/design document with all the goals you want minishift to have and how you're thinking on addressing them?

[hferentschik]

I'm asking as I see a clean/create rather superfluous, of course, depending on how you implement things.

Agreed, one alternative for example is to store some meta information about the start parameters and on a successive start act accordingly:

• Same parameters -> no issue
• Parameters differ -> warning, user interaction needed, potentially abort
• No parameters -> things get a bit interesting here.
  • No parameters and there is no machine -> create machine and bootstrap.
  • Machine exists, apply the stored parameters and give feedback to the user (otherwise it is magic why sometimes parameters are needed and sometimes not)

We will have to work more with metadata sooner or later, since we will also need to at some stage find ways to make sure that for example custom configurations only get applied once.

I think that simplicity is really a must, and the less commands the better.

+1, however sometimes more commands can be better as well. There is a trade-off here, between simplicity and clarity to the user. All I was saying was so far, that to me an additional separation in commands could make sense. I am partly looking at docker-machine behavior here.

I like the fact that simple start/stop or up/down exist and these command with a delete/destroy should be sufficient

As you say, it depends.

the iso needs to be cached (multiple isos can be stored and managed, like vagrant box create |list|delete)

+1 and partly already working this way.

the images should also be cached on the host (at least provide an option to cache them)

+1 There is an issue for this - #143. Even though the plan for now is to just cache the main OpenShift image to improve startup time. There is another idea to keep all state on the host allowing even to throw away the VM and re-creating it w/o loosing OpenShift configuration state. This can be done already and I use it regularly, but it depends on the ability to consistently mount host directories into the VM.

And +1 to make this optional.

Also, if this is the case, what would a clean provide? Shouldn't that be something handled by "destroy" that would delete .minishift/

No, delete is currently just deleting the machine sub directory. This is the VM and its state effectively. What we need and want is a full clean, roughly equivalent to deleting the whole ~/.minishift directory. In this context I am leaning atm towards an additional flag 'minishift delete --all'. I also want to provide a command which gives the user an indication on which data is stored on the host, what it means and how much memory it takes up.

I have included a in minishift directory as I think we should aim to have it as there are specific configuration that will be local to a VM (start params, config,...) and other that will be global to all VMs(isos, images,...).

I am not quite sure what you mean here. Are you missing parts of a sentence?

Is there a proposal/design document with all the goals you want minishift to have and how you're thinking on addressing them?

ATM things would be spread out across the issues here. I guess one way of getting a better feel for things is to look at all issues marked as feature. That said, it might make sense to create a roadmap of some form and make it part of the docs.

[hferentschik]

I'm asking as I see a clean/create rather superfluous, of course, depending on how you implement things.

Agreed, one alternative for example is to store some meta information about the start parameters and on a successive start act accordingly:

• Same parameters -> no issue
• Parameters differ -> warning, user interaction needed, potentially abort
• No parameters -> things get a bit interesting here.
  • No parameters and there is no machine -> create machine and bootstrap.
  • Machine exists, apply the stored parameters and give feedback to the user (otherwise it is magic why sometimes parameters are needed and sometimes not)

We will have to work more with metadata sooner or later, since we will also need to at some stage find ways to make sure that for example custom configurations only get applied once.

I think that simplicity is really a must, and the less commands the better.

+1, however sometimes more commands can be better as well. There is a trade-off here, between simplicity and clarity to the user. All I was saying was so far, that to me an additional separation in commands could make sense. I am partly looking at docker-machine behavior here.

I like the fact that simple start/stop or up/down exist and these command with a delete/destroy should be sufficient

As you say, it depends.

the iso needs to be cached (multiple isos can be stored and managed, like vagrant box create |list|delete)

+1 and partly already working this way.

the images should also be cached on the host (at least provide an option to cache them)

+1 There is an issue for this - #143. Even though the plan for now is to just cache the main OpenShift image to improve startup time. There is another idea to keep all state on the host allowing even to throw away the VM and re-creating it w/o loosing OpenShift configuration state. This can be done already and I use it regularly, but it depends on the ability to consistently mount host directories into the VM.

And +1 to make this optional.

Also, if this is the case, what would a clean provide? Shouldn't that be something handled by "destroy" that would delete .minishift/

No, delete is currently just deleting the machine sub directory. This is the VM and its state effectively. What we need and want is a full clean, roughly equivalent to deleting the whole ~/.minishift directory. In this context I am leaning atm towards an additional flag 'minishift delete --all'. I also want to provide a command which gives the user an indication on which data is stored on the host, what it means and how much memory it takes up.

I have included a in minishift directory as I think we should aim to have it as there are specific configuration that will be local to a VM (start params, config,...) and other that will be global to all VMs(isos, images,...).

I am not quite sure what you mean here. Are you missing parts of a sentence?

Is there a proposal/design document with all the goals you want minishift to have and how you're thinking on addressing them?

ATM things would be spread out across the issues here. I guess one way of getting a better feel for things is to look at all issues marked as feature. That said, it might make sense to create a roadmap of some form and make it part of the docs.

[hferentschik]

@jorgemoralespou - fyi #207

[hferentschik]

@jorgemoralespou - fyi #207

[jorgemoralespou]

Agreed, one alternative for example is to store some meta information about the start parameters and on a successive start act accordingly:

• Same parameters -> no issue
• Parameters differ -> warning, user interaction needed, potentially abort
• No parameters -> things get a bit interesting here.
  • No parameters and there is no machine -> create machine and bootstrap.
  • Machine exists, apply the stored parameters and give feedback to the user (otherwise it is magic why sometimes parameters are needed and sometimes not)
    We will have to work more with metadata sooner or later, since we will also need to at some stage find ways to make sure that for example custom configurations only get applied once.

Once a machine is created, why should a user want to change some configuration? I would only support that configuration that can be applied via a plugin/config command, and not the configuration needed to bootstrap, so that a user would implicitly change configuration. And to be honest, the only option I can think of is to support proxies as you move from a location into another, I can not think of another change that should occur with a configured cluster. Hence, I would try to simplify these scenarios to the maximum. This is very much what many developers are already used to.

I think that simplicity is really a must, and the less commands the better.

+1, however sometimes more commands can be better as well. There is a trade-off here, between simplicity and clarity to the user. All I was saying was so far, that to me an additional separation in commands could make sense. I am partly looking at docker-machine behavior here.

Agree

the images should also be cached on the host (at least provide an option to cache them)

+1 There is an issue for this - #143. Even though the plan for now is to just cache the main OpenShift image to improve startup time. There is another idea to keep all state on the host allowing even to throw away the VM and re-creating it w/o loosing OpenShift configuration state. This can be done already and I use it regularly, but it depends on the ability to consistently mount host directories into the VM.

I think state should not be on the host and mainly as depends on mounting host filesystems. Only things that should live in the host, IMHO are:

• isos
• images (and possibly not all, definitely not the ones built as part of s2i)
• startup configuration to minishift, bootstrap log,
  This would provide the benefit that:
• size on host can be easily controlled
• deleting a machine can delete anything related to that cluster
  That would just add some additional management to isos, cached images and profiles.
• A destroy/delete would delete the profile and the machine but keep the isos and cached images
• A "nuke" command will delete the complete minishift. Although I don't see this useful, and maybe could be handled by an uninstall process.

No, delete is currently just deleting the machine sub directory. This is the VM and its state effectively. What we need and want is a full clean, roughly equivalent to deleting the whole ~/.minishift directory. In this context I am leaning atm towards an additional flag 'minishift delete --all'. I also want to provide a command which gives the user an indication on which data is stored on the host, what it means and how much memory it takes up.

I see this so dangerous as a command, that I would not implement it, or call it "uninstall". Deleting .minishift if you have multiple clusters will delete each one of them, and I don't think a user would want to do that regularly enough to provide a command, and less with that name.

I have included a in minishift directory as I think we should aim to have it as there are specific configuration that will be local to a VM (start params, config,...) and other that will be global to all VMs(isos, images,...).

I am not quite sure what you mean here. Are you missing parts of a sentence?

This was linked to previous comment, that markdown messed up:

Also, if this is the case, what would a clean provide? Shouldn't that be something handled by "destroy" that would delete .minishift/PROFILE

I was referring to the fact that the profiles need to be centralized under a .minishift, but every profile should have their own folder that can be safely deleted, but again, I think that providing a command to delete .minishift would not be a priority for me, and if implemented, definitely with a more clear name: nuke/uninstall/...

I saw the issue for the Roadmap #207. Waiting for your comments there.

[jorgemoralespou]

Agreed, one alternative for example is to store some meta information about the start parameters and on a successive start act accordingly:

• Same parameters -> no issue
• Parameters differ -> warning, user interaction needed, potentially abort
• No parameters -> things get a bit interesting here.
  • No parameters and there is no machine -> create machine and bootstrap.
  • Machine exists, apply the stored parameters and give feedback to the user (otherwise it is magic why sometimes parameters are needed and sometimes not)
    We will have to work more with metadata sooner or later, since we will also need to at some stage find ways to make sure that for example custom configurations only get applied once.

Once a machine is created, why should a user want to change some configuration? I would only support that configuration that can be applied via a plugin/config command, and not the configuration needed to bootstrap, so that a user would implicitly change configuration. And to be honest, the only option I can think of is to support proxies as you move from a location into another, I can not think of another change that should occur with a configured cluster. Hence, I would try to simplify these scenarios to the maximum. This is very much what many developers are already used to.

I think that simplicity is really a must, and the less commands the better.

+1, however sometimes more commands can be better as well. There is a trade-off here, between simplicity and clarity to the user. All I was saying was so far, that to me an additional separation in commands could make sense. I am partly looking at docker-machine behavior here.

Agree

the images should also be cached on the host (at least provide an option to cache them)

+1 There is an issue for this - #143. Even though the plan for now is to just cache the main OpenShift image to improve startup time. There is another idea to keep all state on the host allowing even to throw away the VM and re-creating it w/o loosing OpenShift configuration state. This can be done already and I use it regularly, but it depends on the ability to consistently mount host directories into the VM.

I think state should not be on the host and mainly as depends on mounting host filesystems. Only things that should live in the host, IMHO are:

• isos
• images (and possibly not all, definitely not the ones built as part of s2i)
• startup configuration to minishift, bootstrap log,
  This would provide the benefit that:
• size on host can be easily controlled
• deleting a machine can delete anything related to that cluster
  That would just add some additional management to isos, cached images and profiles.
• A destroy/delete would delete the profile and the machine but keep the isos and cached images
• A "nuke" command will delete the complete minishift. Although I don't see this useful, and maybe could be handled by an uninstall process.

No, delete is currently just deleting the machine sub directory. This is the VM and its state effectively. What we need and want is a full clean, roughly equivalent to deleting the whole ~/.minishift directory. In this context I am leaning atm towards an additional flag 'minishift delete --all'. I also want to provide a command which gives the user an indication on which data is stored on the host, what it means and how much memory it takes up.

I see this so dangerous as a command, that I would not implement it, or call it "uninstall". Deleting .minishift if you have multiple clusters will delete each one of them, and I don't think a user would want to do that regularly enough to provide a command, and less with that name.

I have included a in minishift directory as I think we should aim to have it as there are specific configuration that will be local to a VM (start params, config,...) and other that will be global to all VMs(isos, images,...).

I am not quite sure what you mean here. Are you missing parts of a sentence?

This was linked to previous comment, that markdown messed up:

Also, if this is the case, what would a clean provide? Shouldn't that be something handled by "destroy" that would delete .minishift/PROFILE

I was referring to the fact that the profiles need to be centralized under a .minishift, but every profile should have their own folder that can be safely deleted, but again, I think that providing a command to delete .minishift would not be a priority for me, and if implemented, definitely with a more clear name: nuke/uninstall/...

I saw the issue for the Roadmap #207. Waiting for your comments there.

[budhram]

It looks like things would be more clear once #126 is fixed. Then, we can decide how to approach it.

[budhram]

It looks like things would be more clear once #126 is fixed. Then, we can decide how to approach it.

[gbraad]

I think we need to break down this task in several steps:

• provide feedback that the instance is restarting
• provide feedback which settings are used (mostly implemented as part of #485)
  • 'iso-url' -> cached?
• provide feedback for ignored settings
  • disk-size
  • vm-driver
• provide feedback how to make ignored settings works (m delete)
• ...

[gbraad]

I think we need to break down this task in several steps:

• provide feedback that the instance is restarting
• provide feedback which settings are used (mostly implemented as part of #485)
  • 'iso-url' -> cached?
• provide feedback for ignored settings
  • disk-size
  • vm-driver
• provide feedback how to make ignored settings works (m delete)
• ...

[gunnarmorling]

I encountered problems in that area, too (#1647), specifically about the "provide feedback for ignored settings" and "provide feedback how to make ignored settings works". A message like this would be nice:

Specified vm-driver "abc" will be ignored as the Minishift VM already exists and has been created using driver "xyz". To recreate the VM using another driver, delete the existing one using "minishift delete".

[gunnarmorling]

I encountered problems in that area, too (#1647), specifically about the "provide feedback for ignored settings" and "provide feedback how to make ignored settings works". A message like this would be nice:

Specified vm-driver "abc" will be ignored as the Minishift VM already exists and has been created using driver "xyz". To recreate the VM using another driver, delete the existing one using "minishift delete".

[LalatenduMohanty]

As discussed with @minishift/minishift-dev in face2face we will fix #1622 but this issue is not relevant now as we expect user to do minishift config set for persistent configuration.

[LalatenduMohanty]

As discussed with @minishift/minishift-dev in face2face we will fix #1622 but this issue is not relevant now as we expect user to do minishift config set for persistent configuration.