Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to get endpoints from store: failed to decode endpoint IPv4 address (<nil>) after json unmarshal: invalid CIDR address: <nil> #806

Closed
alefesta opened this issue Jun 9, 2017 · 30 comments

Comments

Projects
None yet
@alefesta
Copy link

commented Jun 9, 2017

Expected behavior

Start docker on windows switched to windows containers

Actual behavior

If I do start docker on win switched to linux containers everything works fine, if I switch back to windows container it crash with the error below.

Docker daemon failed with message:
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to get endpoints from store: failed to decode endpoint IPv4 address () after json unmarshal: invalid CIDR address:

at Docker.Backend.ContainerEngine.Windows.DoStart(Settings settings) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Backend\ContainerEngine\Windows.cs:line 198
at Docker.Backend.ContainerEngine.Windows.Restart(Settings settings) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Backend\ContainerEngine\Windows.cs:line 135
at Docker.Core.Pipe.NamedPipeServer.<>c__DisplayClass9_0.b__0(Object[] parameters) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Core\pipe\NamedPipeServer.cs:line 47
at Docker.Core.Pipe.NamedPipeServer.RunAction(String action, Object[] parameters) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Core\pipe\NamedPipeServer.cs:line 145

Information

  • Diagnostic ID from "Diagnose & Feedback" in the menu.
    57D97238-7C4B-4C5C-9C78-B31DA0191A5E/2017-06-09_09-10-24

Steps to reproduce the behavior

1.Upgrade to docker to version; Docker version 17.06.0-ce-rc2, build 402dd4a
2. ...

@jasonbivins

This comment has been minimized.

Copy link

commented Jun 9, 2017

Hi @alefesta

Do you have a CIDR address specified or mentioned in your daemon.json file? %programdata%\docker\config\daemon.json

https://docs.docker.com/engine/reference/commandline/dockerd/#windows-configuration-file

@alefesta

This comment has been minimized.

Copy link
Author

commented Jun 9, 2017

@jasonbivins

This comment has been minimized.

Copy link

commented Jun 29, 2017

Hi @alefesta
Are you still having issues with starting Docker?

@alefesta

This comment has been minimized.

Copy link
Author

commented Jun 30, 2017

@techmike2kx

This comment has been minimized.

Copy link

commented Jul 19, 2017

I have the same issue, I tried stable and edge releases, but still no Windows Containers ....

@dirething

This comment has been minimized.

Copy link

commented Jul 22, 2017

I dont know if more examples are useful, but I have this as well now

E19D45B3-4CD6-4CA4-A29B-3E9EEFD3DA62/2017-07-22_13-32-35

Docker daemon failed with message:
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to get endpoints from store: failed to decode endpoint IPv4 address () after json unmarshal: invalid CIDR address:

at Docker.Backend.ContainerEngine.Windows.DoStart(Settings settings) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Backend\ContainerEngine\Windows.cs:line 198
at Docker.Backend.ContainerEngine.Windows.Start(Settings settings) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Backend\ContainerEngine\Windows.cs:line 78
at Docker.Core.Pipe.NamedPipeServer.<>c__DisplayClass9_0.b__0(Object[] parameters) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Core\pipe\NamedPipeServer.cs:line 47
at Docker.Core.Pipe.NamedPipeServer.RunAction(String action, Object[] parameters) in C:\gopath\src\github.com\docker\pinata\win\src\Docker.Core\pipe\NamedPipeServer.cs:line 145

Version: 17.06.1-ce-rc1-win20 (12818)
Channel: edge
Sha1: 28a0bb200dda0ffc5d70b0d5df4b77efacfa82fe
Started on: 2017/07/22 13:47:28.884
Resources: C:\Program Files\Docker\Docker\Resources
OS: Windows 10 Pro
Edition: Professional
Id: 1703
Build: 15063
BuildLabName: 15063.0.amd64fre.rs2_release.170317-1834
log.txt

rolling back to a previous release appears to fix it, upgrading again breaks it

Edit: rolling back doesn't fix it, as soon as I set a transparent network and set a container to use it and restart while that container is running and restart twice this happens.

@dirething

This comment has been minimized.

Copy link

commented Jul 28, 2017

Additional Details:

How I have been able to recover:

  • uninstal Docker
  • run ci-zap c:\users\wlanders\downloads\docker-ci-zap.exe -folder c:\programdata\docker (uninstall/reinstall is not enough it seems)
  • reinstall Dcoker
  • make all needed changes to config before enabling network or downloading any containers

Fastest way to reproduce the issue:

  • Install Docker
  • make needed changes to daemon.json & apply
  • make your first network & container:
PS C:\WINDOWS\system32> docker network create --driver=transparent real
8c28f92f1d9f8d7bd9b2312e5b110f12f9d3e05bdc35e51765b491c77553c44e
PS C:\WINDOWS\system32> docker run -d --network=real --hostname=dregistry -v C:\registry:C:\registry  --name dregistry stefanscherer/registry-windows
Unable to find image 'stefanscherer/registry-windows:latest' locally
latest: Pulling from stefanscherer/registry-windows
bce2fbc256ea: Pull complete
baa0507b781f: Pull complete
706a568059a2: Pull complete
174bbd1d6140: Pull complete
146e2c3e621e: Pull complete
6ef504f2e82b: Pull complete
683b428db034: Pull complete
570bce496428: Pull complete
Digest: sha256:1dd0d752715cc66651bab588cf368f47f6c630dc1e34fa5c6835784a199cde56
Status: Downloaded newer image for stefanscherer/registry-windows:latest
afe0bc764e95d66de6ce5d8f5dfbf80c7f764e1bbdb07d4c3cc7e06cc457f850
  • docker tray icon --> settings --> reset --> restart docker
Docker daemon failed with message:
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to get endpoints from store: failed to decode endpoint IPv4 address (<nil>) after json unmarshal: invalid CIDR address: <nil>


   at Docker.Backend.ContainerEngine.Windows.DoStart(Settings settings)
   at Docker.Backend.ContainerEngine.Windows.Restart(Settings settings)
   at Docker.Core.Pipe.NamedPipeServer.<>c__DisplayClass9_0.<Register>b__0(Object[] parameters)
   at Docker.Core.Pipe.NamedPipeServer.RunAction(String action, Object[] parameters)

Most recent crash report id: F0E3EC16-989B-4B2D-9542-76C99F9BE72B/2017-07-28_16-23-05

daemon.json:

{
	"insecure-registries": ["dregistry:5000"],
	"registry-mirrors": [],
	"allow-nondistributable-artifacts": ["dregistry:5000"],
	"debug": true,
	"experimental": true
}
@dirething

This comment has been minimized.

Copy link

commented Jul 31, 2017

@jasonbivins please see above and let me know if having a repeatable example is useful before I try more severe measures than uninstall/zap/reinstall to resolve this

@benze

This comment has been minimized.

Copy link

commented Aug 10, 2017

I have the same issue, but my C:\ProgramData\Docker\config\daemon.json file is empty

The only change I last made to my Docker for Windows Container was to add a transparent lan before the last Docker restart. I don't know if that causes the issue or not. Furthermore, I'm not sure how to revert that change given that I can't start Docker Engine for Windows Containers.

Is there anything that can be done? I don't want to Factory Reset and lose all my local containers & images. Crash report Id: E334AF44-A213-437F-9AD6-B5433D504D83/2017-08-10_11-27-19

I've noticed that Linux Containers work fine.

@benze

This comment has been minimized.

Copy link

commented Aug 10, 2017

@dirething Thanks for the instructions. I gave up trying to "fix" it, and followed your list:

  • uninstal Docker
  • run docker-ci-zap .exe: docker-ci-zap.exe -folder c:\programdata\docker (uninstall/reinstall is not enough it seems)
  • reinstall Docker

I'm convinced there is a problem with the transparent lan I tried to create. Lesson learnt. Not creating one of those again.

@dirething

This comment has been minimized.

Copy link

commented Aug 12, 2017

After not getting any feedback for a week I allowed docker to update itself and I haven't had to destroy my setup so far. I am running 17.07.0-ce-rc2 now.

@benze : The container in my reproduction steps is a local registry. if you make the appropriate adjustments to your docker config it will allow you to tag the images and push them into the local registry which stores outside what gets destroyed by destroying docker. This saved me 12hrs of downloads every time I had to reboot (and therefore corrupt docker) for the last few weeks. If you are still experiencing this issue after the update it may be worth looking into. One tip if following the author's instructions and you are having this issue: you must fix the docker config BEFORE you start the registry container or you will end up wrecking docker with the restart that is required to get the daemon config to load.

@nikoo56

This comment has been minimized.

Copy link

commented Sep 7, 2017

Please help. I am getting this error when I switch to Windows Container:

Docker daemon failed with message:
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to get endpoints from store: failed to decode endpoint IPv4 address () after json unmarshal: invalid CIDR address:

I have created transparent network as well and I have deleted them but I think I mess something with my networking. I tried all the above but I am still getting this error.

@tfenster

This comment has been minimized.

Copy link

commented Nov 9, 2017

I could get it up and running again by deleting C:\ProgramData\docker\network\files\local-kv.db. Afterwards the service starts fine, I just have two unnamed transparent networks. I can remove those two and re-create my named one and it works again.

Not really a solution but a quicker workaround than removing and re-installing everything

@guidoffm

This comment has been minimized.

Copy link

commented Nov 20, 2017

I have the same problem using "17.06.2-ee-5" on Windows 2016. We use a transparent network with DHCP.

@benze

This comment has been minimized.

Copy link

commented Nov 20, 2017

@bojanrajkovic

This comment has been minimized.

Copy link

commented Nov 26, 2017

Ran into this same issue--the workaround in #806 (comment) solved it for me.

@jasonbivins

This comment has been minimized.

Copy link

commented Nov 28, 2017

Hi @benze

It sounds like there may have been a stuck network - this script will reset all of your windows networks to factory default
https://github.com/MicrosoftDocs/Virtualization-Documentation/tree/live/windows-server-container-tools/CleanupContainerHostNetworking

@tfenster

This comment has been minimized.

Copy link

commented Nov 28, 2017

Hi @jasonbivins

that script only works if the docker daemon can start or am I missing something? Cause at least in my case the daemon didn't start at all, so I couldn't use anything depending on a running daemon

@ByronWrightGeek

This comment has been minimized.

Copy link

commented Dec 13, 2017

I just ran into the same issue as I was trying out transparent networks. I was able to get the service to start after removing C:\ProgramData\docker\network\files\local-kv.db. This file was recreated when I started the service.

After restarting I was able to list the networks and it appeared that the transparent network was corrupt because the name was changed to a long random string. I was able to remove it at that point. However, at this point, each time I restart the container host or the docker service a randomly named transparent network comes back.

Final piece was running .\WindowsContainerNetworking-LoggingAndCleanupAide.ps1 -Cleanup as mentioned above. Now the host seems fine and the network is not reappearing.

@fasihrana

This comment has been minimized.

Copy link

commented Feb 8, 2018

I ran into this issue as well. I'm running Docker 17.06.2 on Win Server 2016.

Everything was running fine until the VM restarted overnight and since then can't run any docker engine commands.

If I run dockerd from powershell, here is what I see:

PS C:\ProgramData\docker\config> dockerd
time="2018-02-08T10:32:15.350729200Z" level=info msg="Windows default isolation mode: process"
time="2018-02-08T10:32:15.357700700Z" level=info msg="[graphdriver] using prior storage driver: windowsfilter"
time="2018-02-08T10:32:15.435542100Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2018-02-08T10:32:15.437615000Z" level=info msg="Loading containers: start."
time="2018-02-08T10:32:15.456671100Z" level=info msg="Restoring existing overlay networks from HNS into docker"
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to get endpoints from store: failed to decode endpoint IPv4 address (<nil>) after json unmarshal: invalid CIDR address: <nil>

More importantly, my daemon.json has disappeared from the config folder:

PS C:\ProgramData\docker\config> ls


    Directory: C:\ProgramData\docker\config


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----         2/7/2018   7:31 PM            244 key.json


PS C:\ProgramData\docker\config>
@eknraw

This comment has been minimized.

Copy link

commented Feb 21, 2018

I am also running into the same issue on two windows 2016 servers. This only occurs when I add a transparent network. The solution provided by @tfenster to delete C:\ProgramData\docker\network\files\local-kv.db seems to be the fastest workaround.

@kine

This comment has been minimized.

Copy link

commented Mar 6, 2018

Same issue on my VMWare Windows 2016 server, when moved from one VMWare host to another. Two-times. Each time I removed local-kv.db to be able to start the service and cleared unnamed networks and recreated them.

@MatthiasKammerlohr

This comment has been minimized.

Copy link

commented Mar 12, 2018

Same issue on our VMWare Windows Server 2016 after every reboot!

@haiduc32

This comment has been minimized.

Copy link

commented Jul 3, 2018

Windows Server 2016 Docker 17.06.2-ee-14 - exact same problem.

Is anyone looking into the problem?

@rn

This comment has been minimized.

Copy link
Contributor

commented Jul 9, 2018

@haiduc32 if you can repro with ee on windows server, can you file an issue with microsoft? See http://success.docker.com/article/where-to-get-help-with-windows for details

@jan11011977

This comment has been minimized.

Copy link

commented Jul 12, 2018

I also have this problem on Windows Server Core 1803. Add a transparent network, run some containers using the transparent network, reboot the server, and about 40-50% of the time the Docker service will fail to start on reboot and we manually need to delete the local-kv.db file and recreate the network.

This issue is very easy to reproduce for me, so if you need any more information I would be happy to help.

@docker-desktop-robot

This comment has been minimized.

Copy link
Collaborator

commented Oct 10, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

@heaths

This comment has been minimized.

Copy link

commented Oct 10, 2018

Could a maintainer have a look at this? Seems a number of people are able to reproduce this and willing to share any details necessary.

@heaths

This comment has been minimized.

Copy link

commented Oct 10, 2018

/remove-lifecycle stale

@docker-desktop-robot

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.