Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for predictable IP's across restarts #951

Closed
RaeesBhatti opened this issue Dec 28, 2016 · 72 comments
Closed

Support for predictable IP's across restarts #951

RaeesBhatti opened this issue Dec 28, 2016 · 72 comments

Comments

@RaeesBhatti
Copy link
Contributor

@RaeesBhatti RaeesBhatti commented Dec 28, 2016

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Minikube version (use minikube version): v0.14.0

Environment:

  • OS : macOS Sierra 1.12.1
  • VM Driver : xhyve
  • Docker version: 1.11.1
  • Install tools:
  • Others:

What happened:
Minikube changes IP address when recreated.

What you expected to happen:
Minikube should retain the IP address when deleted and created again.

How to reproduce it (as minimally and precisely as possible):

  • minikube start --vm-driver=xhyve
  • minikube ip (note the IP address)
  • minikube stop
  • minikube delete
  • minikube start --vm-driver=xhyve
  • minikube ip (IP address has changed)

Anything else do we need to know:
I have several things configured to Minikube's IP address and have to recreate it occasionally. They all have to be reconfigured each time. It would be convenient to have Minikube not change its IP address after deleting and recreating.

@ankon
Copy link
Contributor

@ankon ankon commented Dec 28, 2016

This also happens with the kvm driver, but not with the virtualbox driver.

Would be great if one could force a fixed IP address when using minikube start, independent of the driver.

@r2d4
Copy link
Contributor

@r2d4 r2d4 commented Dec 28, 2016

There is an option with the xhyve driver to set a static IP. By setting a static UUID here [0], you can ensure that you get the same IP across deletes/starts. We had this feature enabled for awhile but disabled it again because there were some issues. The discussion of the issues starts with this comment and continues in this issue. [1]

I think that this is a great feature to have, we just need to figure out the networking issues.

[0] https://github.com/zchee/docker-machine-driver-xhyve/blob/master/xhyve/xhyve.go#L70
[1] #708 (comment)

@dongsupark
Copy link
Contributor

@dongsupark dongsupark commented Jan 26, 2017

Same here. I have been seeing the issue that the IP address of VM changes across restarts.
It happens only with --vm-driver=kvm, not with --vm-driver=virtualbox. I don't know about the xhyve driver.

@georgecrawford
Copy link

@georgecrawford georgecrawford commented Mar 27, 2017

I'm seeing this with the xhyve driver as well. Any updates on a fix?

@tmirks
Copy link

@tmirks tmirks commented Mar 27, 2017

@georgecrawford There's a workaround by manually (or programmatically) updating /var/db/dhcpd_leases. I believe you can just delete the contents (make a backup just in case).

@ankon
Copy link
Contributor

@ankon ankon commented Apr 6, 2017

I've by now seen this also with the virtualbox driver: typically my VM is 192.168.99.100, but sometimes it ends up as .101 or similar. Removing the minikube VM in those cases seems to help.

@gheibia
Copy link

@gheibia gheibia commented Apr 15, 2017

Same issue here. I'm using KVM and I don't even delete the cluster. I simply stop and start it again. I even tried to change dnsmasq's configuration to limit the DHCP​'s address range. Didn't make a difference.

@n-k
Copy link

@n-k n-k commented May 17, 2017

Same issue with vmwarefusion driver on OSX.

@jstangroome
Copy link
Contributor

@jstangroome jstangroome commented May 21, 2017

Also occurring for minikube v0.19.0 with Virtualbox 5.1.18 on Windows 10 x64 Build 14393.1198.

  1. minikube start
  2. minikube ip (note the IP address)
  3. minikube stop
  4. minikube delete
  5. minikube start
  6. minikube ip (IP address has changed)

Assigned IP address has so far incremented from 192.168.99.100 to 192.168.99.105

@andziak
Copy link

@andziak andziak commented Jun 21, 2017

The IP address for Minikube changes on Max OS with xhyve driver. It would be great to use static IP address as we're developing apps in Pycharm remotely on Minikube pods. This way we have a full working copy of the whole environment on developer's machine. When IP changes, we need to reconfigure dozen projects in Pycharm which takes lots of time.

@zainul-ma
Copy link

@zainul-ma zainul-ma commented Jun 22, 2017

its, would be great to have feature set static ip in minikube, any temp solution for this ?

@garethr
Copy link

@garethr garethr commented Aug 10, 2017

I've also seen this with the Hyper-V driver. Stopping and starting appears to fix the issue until you connect to a new network.

@squillace
Copy link

@squillace squillace commented Aug 20, 2017

Yeah, this isn't a feature, as it makes dnsmasq harder to configure. Now you can't map domain names to a single IP. Looking for any workaround here.

@dgoldssfo
Copy link

@dgoldssfo dgoldssfo commented Aug 23, 2017

@squillace On Mac:

minikube start
minikube ip (for me, 192,168.99.100)
minikube stop; minikube delete
Stop/quiesce all running VMs
sudo /Library/Application\ Support/VirtualBox/LaunchDaemons/VirtualBoxStartup.sh restart
minikube start
minikube ip (192,168.99.100 like I want and not 192.168.99.101)

@yoshiwaan
Copy link

@yoshiwaan yoshiwaan commented Aug 23, 2017

Having the DHCP addresses reset after a login/logout or some other trick isn't a good solution. What if I want to work on minishift for a while before I start up minikube? What if I have a test kitchen run running that uses VirtualBox?
You need a clear way to set a static address for minikube (and thus you can create a hosts entry for it as well). Something --ip-address 192.168.100.52.

The solution that micheletedeschi posts (there's 2 comments from him) in docker/machine#1709 works pretty well for docker-machine. If minikube had a similar way to run arbitrary scripts at boot up then that would be one quick way to solve the problem in a way that doesn't remove the option for a command line flag in the future.

@dgoldssfo
Copy link

@dgoldssfo dgoldssfo commented Aug 23, 2017

I agree, it's a workaround/hack and it should only be used in situations where you don't have a bunch of other VMs that you need to worry about running on VirtualBox. I should have indicated that in my previous post, thanks for posting this.

And yes, minikube definitely needs to have a way to use a static IP.

@gbraad
Copy link
Contributor

@gbraad gbraad commented Aug 24, 2017

Although I am now experimenting with adding an option --ip-address to minishift: minishift/minishift#1293 this is not easy for each hypervisor. The workarounds used allow the VM to get an IP address from the hypervisors DHCP server, and then change it in-place with the given option.

In my case, I am experimenting with Hyper-V as it allows to pass data to the VM, but so far I none of the other hypervisors do this. This means that the value can be passed into the VM before it has even tried to get an IP address (before the docker-machine provisioning started). Since it would several problems we have with Hyper-V: minishift/minishift#418 #1627 it is my focus for the moment.

If you have other suggestions, please let me know... especially when it concerns the other hypervisors.

@gbraad
Copy link
Contributor

@gbraad gbraad commented Sep 1, 2017

Static IP address option for Minishift using HyperV is being worked on as minishift/minishift#1316. I am coordinating with @aaron-prindle to see if this can also work for minikube.

@gbraad
Copy link
Contributor

@gbraad gbraad commented Sep 27, 2017

PR with a possible solution is available at Minishift's repo for testing. Instructions are here: minishift/minishift#1316 (comment)
If results are usable, we will also integrate this in minikube and the iso.

@coryasilva
Copy link

@coryasilva coryasilva commented Nov 7, 2017

@JavaJung
Copy link

@JavaJung JavaJung commented Jan 5, 2018

New user of minikube and kubernetes-cli here. I have this issue of changing minikube ip too, on mac. The driver is Hyperkit commit 3e3161, minikube v0.24.1, kubectl 1.8.0 (client and server).

If it will be possible to integrate setting a static ip for the VM in minikube, would that work for every driver?

@pfk-beta
Copy link

@pfk-beta pfk-beta commented Feb 19, 2018

Same issue, doesn't matter which vm driver used.

@DennisBayer
Copy link

@DennisBayer DennisBayer commented Feb 26, 2018

Hi @gbraad,
any news on this issue, e.g. an ETA?

Best regards!

@styk-tv
Copy link

@styk-tv styk-tv commented Mar 6, 2018

So --vm-driver=virtualbox has no effect on my Mac. Start-Stop-Start will still rotate. It seems they go from 100 to 103 and then repeat.

I was experimenting with export HOSTALIASES (you can have your userspace personal hosts file. Works on Ubuntu, HOWEVER not working on Mac) Would be nice to just map minikube.local and then output that local hosts file.

Well i still decided to go that path with sudo to update /etc/hosts. (after minikube start) Not ideal but it works. All provisioning scripts after that reference minikube.local hostname.

EDIT: use below at your own risk. test on a copy of hosts first before you set against your system file. Tested on Macbook. It detects the line with minikube and replaces entire line with new line and new ip - puts a tab in between. Yes it asks for root pass, yes its bad, yes it works.

sudo sed -i -e 's/.*minikube.*/'$(minikube ip)$'\tminikube.local/' /etc/hosts
@alphabt
Copy link

@alphabt alphabt commented Feb 28, 2019

I'm also having this issue on minikube 0.34.1 with hyperkit in macOS. PR #3572 didn't fix it for me. I had to resort to deleting entries in /private/var/db/dhcpd_leases.

  • minikube start --vm-driver=hyperkit
  • minikube ip (note the IP address)
  • minikube stop
  • minikube delete
  • minikube start --vm-driver=hyperkit
  • minikube ip (IP address has changed)
@maicohjf
Copy link

@maicohjf maicohjf commented Mar 17, 2019

  1. Set configuration context $ kubectl config use-context k8s
    List all PVs sorted by name, saving the full kubectl output to /opt/KUCC0010/my_volumes . Use
    kubectl s own functionally for sorting the output, and do not manipulate it any further.
    Question weight 3%
    Answer:
    $kubectl get pv --all-namespaces --sort-by="metadata.name" >>
    /opt/KUCC0010/my_volumes or
    $kubectl get pv --all-namespaces --sort-by="spec.capacity.storage" >>
    /opt/KUCC0010/my_volumes
@marxangels
Copy link

@marxangels marxangels commented Mar 30, 2019

C:\Users\trump\.VirtualBox\HostInterfaceNetworking-VirtualBox Host-Only Ethernet Adapter-Dhcpd.leases

using vbox under win10, just edit such file, change the ip and restart minikube.

@matti
Copy link

@matti matti commented Jun 9, 2019

hyperkit has uuid which is used in MAC address generation, when you start the machine with UUID like

minikube start -p makkara-1 --vm-driver hyperkit --uuid 00000000-0000-0000-0000-000000000001 --cpus 12 --memory 2048

the MAC address stays the same even if you delete the machine - the IP addresses can be changed in /var/db/dhcpd_leases the address must be in 192.168.64.xx because /Library/Preferences/SystemConfiguration/com.apple.vmnet.plist sets the address range. I haven't tried changing that range.

@stafot
Copy link

@stafot stafot commented Jun 19, 2019

Based on comments #951 (comment) , #951 (comment) I created a multi OS support variant of the "workaround" script because I needed to support both MacOS and Linux systems.
Is here for anyone might find it useful.

@esbol-moldrakhmetov
Copy link

@esbol-moldrakhmetov esbol-moldrakhmetov commented Jul 20, 2019

Maybe I can help someone else. First I put VBox 6.* and then I had a problem with changing the ip address, then I made downgrade on VBox 5.2.

@pietervogelaar
Copy link

@pietervogelaar pietervogelaar commented Aug 19, 2019

A bit heavy handed way to guarantee minikube gets 192.168.99.100 everytime with virtualbox.

function minikube_reset_vbox_dhcp_leases() {
  # # Reset Virtualbox DHCP Lease Info
  common_message_print "Resetting Virtualbox DHCP Lease Info..."
  kill -9 $(ps aux |grep -i "vboxsvc\|vboxnetdhcp" | awk '{print $2}') 2>/dev/null

  if [[ -f ~/Library/VirtualBox/HostInterfaceNetworking-vboxnet0-Dhcpd.leases ]] ; then
    rm  ~/Library/VirtualBox/HostInterfaceNetworking-vboxnet0-Dhcpd.leases
  fi
}

minikube_reset_vbox_dhcp_leases

I experienced a bit unstable VirtualBox Manager (GUI) with this.

@strowi
Copy link

@strowi strowi commented Aug 19, 2019

hi,

for anyone interested, i worked around this with a simple static ip. Disadvantage is, its not returned with "minikube ip". But since i can configure it thats not a big problem..
minikube ssh "sudo ifconfig eth0:0 192.168.99.200"

regards,
strowi

@tstromberg
Copy link
Contributor

@tstromberg tstromberg commented Dec 19, 2019

Marking as a backlog item, because I don't believe this will be likely possible to do consistently across hypervisors until we migrate away from libmachine to something like terraform.

@fredalbino
Copy link

@fredalbino fredalbino commented Mar 3, 2020

This is still an ongoing thing for me as well. Do we have any additional information in regards to this?

@pwhissell
Copy link

@pwhissell pwhissell commented Mar 6, 2020

The lease deletion workaround implemented as a bash script for ubuntu users:

minikube stop
mkdir ~/.config/VirtualBox/leasebackup/
mv ~/.config/VirtualBox/*.leases ~/.config/VirtualBox/leasebackup/
mv ~/.config/VirtualBox/*.leases-prev ~/.config/VirtualBox/leasebackup/
VBoxManage hostonlyif remove vboxnet0

-- edit:
All vms and vbox services must be shutdown before restarting minikube

@medyagh
Copy link
Member

@medyagh medyagh commented Apr 20, 2020

on docker driver we could do this : #7756

I would be happy for anyone who like to pick that issue first for docker driver, and then later we could do same logic for KVM and virtualbox

@tstromberg
Copy link
Contributor

@tstromberg tstromberg commented Apr 23, 2020

I'm closing this admittedly popular, but overly broad issue, because it is effectively unsolvable for all drivers we interact with.

It is not currently a priority for the maintainers to address this issue, but PR's are certainly welcome to introduce predictable IP support for a particular driver. I caution that doing so may be tricky, particularly when running multi-cluster and/or multi-node configurations.

@matti
Copy link

@matti matti commented Apr 24, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet