feature request: Take a snapshot of a cluster

[owenthereal]

Steps to reproduce the issue:

This is a feature request not an issue.

I would like to be able to take a snapshot of the cluster and reuse it for CI for a faster boot time. I have functional tests that run on a minikube cluster but the tests require installation of a bunch of software, e.g. ingress-nginx, cert-manager etc. I'm wondering whether there is a way to "pre-burn" a minikube image by taking a snapshot of a running cluster. The image includes a version of the minikube k8s and software that I install on it. The image is then used in CI to avoid reinstalling all the basic software every time. This would immensely shorten the CI time.

Full output of failed command:

N/A

Full output of minikube start command used, if not already included:

N/A

Optional: Full output of minikube logs command:

N/A

[afbjorklund]

Did you check out the minikube cache command already ?

[owenthereal]

minikube cache adds/deletes/loads an image to/from minikube. What I wanted is to burn the whole cluster with software that I install into the cluster as an image so that it can be reused elsewhere, e.g. CI.

[afbjorklund]

You did not mention what platform and driver this was for. I'm not sure the "pause" feature will survive such a "burn"...

We talked about adding the suspend/resume feature earlier, but ended up adding the pause/unpause feature instead.
In theory the VirtualBox driver should support it, but I have only done it locally - and there is no minikube support for it

There might be other "kubernetes backup" software, to take snapshots of running clusters. Maybe they can be integrated.

[owenthereal]

You did not mention what platform and driver this was for

I'm looking for this feature for Linux with the Docker driver mainly. A quick google shows that it is possible and easy to do: https://stackoverflow.com/questions/26331651/how-can-i-backup-a-docker-container-with-its-data-volumes.

Regarding other drivers, most of them should have the feature of snapshotting or suspending/resuming as you mentioned.

There might be other "kubernetes backup" software, to take snapshots of running clusters. Maybe they can be integrated.

I would love to see the integration of such software. Any pointers?

[afbjorklund]

Something like https://github.com/vmware-tanzu/velero perhaps ?

[owenthereal]

I checked https://github.com/vmware-tanzu/velero and it's cloud provider specific. I'm unclear how that would apply for minikube.

[medyagh]

this is a good idea ! we have plans to explore the idea of freezing a cluster to potentially start minikube faster with a frozen cluster.

this is on our roadmap to explore.
both
@ilya-zuyev @priyawadhwa are looking into this.

[priyawadhwa]

Yup, we are aiming to have a design for this this quarter. I will update this issue as we know more!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[owenthereal]

Hello, I'm curious that while we are waiting for this feature to be ready, is there anything you could share we do in the meantime, e.g. commands to run to take a snapshot of the running cluster, export it, and run the snapshot at another place.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[sharifelgamal]

This unfortunately fell by the wayside as priorities shifted in our roadmap. I still think it's a great idea, but a design was never fully finished.

[mholtzhausen]

Is there a chance this is coming back?

[sharifelgamal]

The minikube maintainers don't currently have the bandwidth to work on this feature, but it's still a cool idea! I'll add the help wanted label and would be happy to look at a PR.

[nirs]

I can see something like this:

Creating cluster:

minikube start ...
play with cluster ...
minikube stop

minikube snapshot snap1

Using cluster form last persisted state:

minikube start
Play ...
minikube stop

Reverting to snapshot, dropping all changes since the snapshot:

minikube revert snap1

Taking a snapshot is easy with vm based drivers (e.g. kvm2). Not sure about other drivers,
but such feature can be useful even if it works only with some drivers.

For kvm2/qemu based drivers, creating a snapshot can be simple as:

mv disk.qcow2 snap1.qcow2
qemu-img create -f qcow2 -b snap1.qcow2 -F qcow2 disk.qcow2

Reverting a snapshot:

mv snap1.qcow2 disk.qcow2 

If the original disk was raw, we need to change the disk in the vm xml.
If there are multiple snapshot, the qcow2 chain may need to be modified.

Creating and deleting live snapshot can be nice but is not really needed
to make this useful.

This can awesome feature of CI. In ramen we have a test environment creating 3 clusters
with complicated deployment, taking 7-8 minutes to create:
https://github.com/RamenDR/ramen/blob/main/test/regional-dr.yaml

If we can re-create the environment periodically and starting from a snapshot,
we can get a running test environment in 1-2 minutes. Of course if the tests run
with the environment take 60 minutes, saving 6 minutes creating the environment
is not huge saving, but still important.

Is it possible to implement such feature as a 3rd party addon? Do we have APIs to query
and modify the vm disks, so such addon will not break when minikube changes the way
disks are stored?