Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--security-opt flag times out and --credentialspec doesn't do anything #587

Open
jhiller opened this Issue Mar 31, 2017 · 44 comments

Comments

Projects
None yet
@jhiller
Copy link

jhiller commented Mar 31, 2017

--security-opt flag does not work

Windows 10 v1607
OS Build 14393.953
Domain Joined
Docker Version 17.03.1-ce-win5 (10743)
32 GB RAM
Core i7-6820HQ

Link to doc: https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/manage-serviceaccounts

works
docker run -d --credentialspec gMSA.json microsoft/nanoserver ping -t 127.0.0.1

doesn't work
docker run -h docker_msa_o -d --security-opt "credentialspec=file://gMSA.json" microsoft/nanoserver ping -t 127.0.0.1

docker : C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon:
container 9f01aa3c25df4247fa2320c2fe071268564fa21a3eab4279a1ba72dce2801b37 encountered an error
during Start: failure in a Windows system call: The operation timed out because a response was
not received from the Virtual Machine hosting the Container. (0xc0370109).
At line:1 char:1

@jhiller jhiller changed the title --security-opt flag times out but --credentialspec works --security-opt flag times out and --credentialspec doesn't do anything Mar 31, 2017

@jhiller

This comment has been minimized.

Copy link
Author

jhiller commented Mar 31, 2017

Looking at the daemon logs, --credentialspec no longer appears to do anything because the credentials are not passed along to the container. This is using --credentialspec

HCSShim::CreateContainer id=020d7d3c3fa6396c480f87808b15c24150bb8c74fe21ad73762b179335cbac09 config={"SystemType":"Container","Name":"020d7d3c3fa6396c480f87808b15c24150bb8c74fe21ad73762b179335cbac09","Owner":"docker","IsDummy":false,"IgnoreFlushesDuringBoot":true,"LayerFolderPath":"D:\\docker\\windowsfilter\\020d7d3c3fa6396c480f87808b15c24150bb8c74fe21ad73762b179335cbac09","Layers":[{"ID":"022a722b-a693-5ea1-81cd-f93c715957be","Path":"D:\\docker\\windowsfilter\\535d6a6f565a5f656e9e114f3e9c69bc0e11cbc5944a9c2d5ccd8c26e4229fcd"},{"ID":"e6b57c65-46ca-5fe8-8bfc-6139125f3af7","Path":"D:\\docker\\windowsfilter\\b2f12ac2484f79347b7ebb7549ec69895774587f6e55626d47be1994cfc91b23"}],"HostName":"020d7d3c3fa6","MappedDirectories":[],"SandboxPath":"D:\\docker\\windowsfilter","HvPartition":true,"EndpointList":["3dbc3320-70c0-4fbb-ba06-71f86115e64c"],"HvRuntime":{"ImagePath":"D:\\docker\\windowsfilter\\535d6a6f565a5f656e9e114f3e9c69bc0e11cbc5944a9c2d5ccd8c26e4229fcd\\UtilityVM"},"Servicing":false,"AllowUnqualifiedDNSQuery":true}"

@PatrickLang

This comment has been minimized.

Copy link
Contributor

PatrickLang commented Mar 31, 2017

Have you tried this on Windows Server 2016 on the same domain? I'm curious if there's a difference between using client & server builds.

--security-opt="credentialspec=file://..." is the right syntax

@jhiller

This comment has been minimized.

Copy link
Author

jhiller commented Apr 4, 2017

Seems to run in Server 2016, though I can't access domain resources from inside the container. I'm looking at that now. It doesn't time out like it does in Windows 10. Have you tested with Windows 10?

Client:
Version: 17.03.0-ee-1
API version: 1.26
Go version: go1.7.5
Git commit: 9094a76
Built: Wed Mar 1 00:49:51 2017
OS/Arch: windows/amd64

Server:
Version: 17.03.0-ee-1
API version: 1.26 (minimum version 1.24)
Go version: go1.7.5
Git commit: 9094a76
Built: Wed Mar 1 00:49:51 2017
OS/Arch: windows/amd64
Experimental: false

@PatrickLang

This comment has been minimized.

Copy link
Contributor

PatrickLang commented Apr 4, 2017

Thanks for checking. I need to try it on Windows 10 again since I haven't in several months. This was intended for Windows Server 2016 and it is primarily tested there.

@jhiller

This comment has been minimized.

Copy link
Author

jhiller commented Apr 6, 2017

I had $ at the end of the managed service account name in the credentialspec file. Once I removed that I was able to access domain resources from the container on server 2016. I ran the identical config/command on Windows 10 and it still times out. Should I open an issue over on the Windows Containers forum?

@mylesbkeating1993

This comment has been minimized.

Copy link
Contributor

mylesbkeating1993 commented Apr 7, 2017

Yes, the forums are a better place for this :)

Microsoft Containers forum: https://social.msdn.microsoft.com/Forums/en-US/home?category=virtualization&forum=windowscontainers&filter=alltypes&sort=lastpostdesc&brandIgnore=true&page=2

Docker forum: forums.docker.com

@mylesbkeating1993 mylesbkeating1993 self-assigned this Apr 7, 2017

@mylesbkeating1993

This comment has been minimized.

Copy link
Contributor

mylesbkeating1993 commented Apr 10, 2017

@jhiller could you let me know when you've posted in the forum so I can close this issue?

Thanks!

And of course let me know if there's anything I can help with: myles.keating [at] Microsoft.com

@seanturner83

This comment has been minimized.

Copy link
Contributor

seanturner83 commented May 25, 2017

Definitely able to repeat this. This may not be absolutely the right place for this issue, but it needs to be tracked somewhere - please advise. Does anyone have any sort of workaround?

@seanturner83

This comment has been minimized.

Copy link
Contributor

seanturner83 commented May 25, 2017

Also tried with run-containerimage:

Run-ContainerImage : Docker API responded with status code=InternalServerError, response={"message":"container bc2be6162d0725618ac599941dbfd2712e889a70984e7650826d1f3c8a2fe95f encountered an error during Start: failure in a Windows system
call: The operation timed out because a response was not received from the Virtual Machine hosting the Container. (0xc0370109)"}
At line:1 char:1

  • Run-ContainerImage -ImageIdOrName "31b" -HostConfiguration $hostConfi ...
  •   + CategoryInfo          : NotSpecified: (:) [Invoke-ContainerImage], DockerApiException
      + FullyQualifiedErrorId : Docker Client Exception,Docker.PowerShell.Cmdlets.InvokeContainerImage
    
@PatrickLang

This comment has been minimized.

Copy link
Contributor

PatrickLang commented May 25, 2017

@seanturner83 Windows 10 or Windows Server 2016?

@seanturner83

This comment has been minimized.

Copy link
Contributor

seanturner83 commented May 25, 2017

Windows 10 - am I right in thinking this is because I'm stuck with a Hyper-V layer between my container and my domain-joined workstation?

@PatrickLang

This comment has been minimized.

Copy link
Contributor

PatrickLang commented May 25, 2017

@seanturner83 try it on Windows Server 2016. Based on the above reports it appears to be broken on Win10

@seanturner83

This comment has been minimized.

Copy link
Contributor

seanturner83 commented May 25, 2017

that's not so useful for the thirty or so developers who want it to run the same on their workstation as it does in the cloud though...

@mstephano

This comment has been minimized.

Copy link

mstephano commented May 26, 2017

  • Anyone knows when this will be fixed for Windows 10? It doesn't make sense to run a workstation with Visual Studio 2017 in Windows Server 2016 because of licensing (too expensive).. Win10 is cheaper.
  • Also, Docker-compose is not installed by default on Windows Server 2016.. you have to download it.
@Alexander-Bartosh

This comment has been minimized.

Copy link

Alexander-Bartosh commented Jul 26, 2017

Still does not work for Windows 10.
Was there an issue created for this topic ?

FYI: It does work for Windows Server 2016 hosted container.

@vikmn

This comment has been minimized.

Copy link

vikmn commented Oct 25, 2017

Anyone have an outcome on this?

@seanturner83

This comment has been minimized.

Copy link
Contributor

seanturner83 commented Oct 25, 2017

Yeah I went back to Linux :)

@PatrickLang

This comment has been minimized.

Copy link
Contributor

PatrickLang commented Nov 16, 2017

This should be fixed on Windows 10 Fall Creators Update (aka 1709, build 16299) and later. It worked on the Windows 10 anniversary update 1609, broke in 1703, and was fixed for 1709. Can you let us know if it's still happening?

@PatrickLang

This comment has been minimized.

Copy link
Contributor

PatrickLang commented Dec 15, 2017

Ping - anyone still having problems with Windows 10 build 16299? If not, I'll close this next week

@alekslt

This comment has been minimized.

Copy link

alekslt commented Dec 19, 2017

@PatrickLang I still experience this error. I'm on Win10 1709 (16299.125).
I recently upgraded to 1709 and was hit with the Hyper-V issues (in addition to not having gmsa/credspec working). As a part of the troubleshooting and fixing-process I also removed and reinstalled the windows features for hyper-v and windows containers.
I've reset Docker, but have not tried to completely remove docker, yet. Have run updates though.

I've run Debug-ContainerHost and the only issue I get is regarding Windows-NAT,
but I believe this to be due to recent Hyper-V changes.
Containers without sec.opt/credspec can be reached fine, with the resulting log entries attached.

The only antivirus installed is Windows Defender with an exemption for the docker prog.file and progdata areas.

Version info:

Engine:
Version: 17.12.0-ce-rc3
API version: 1.35 (minimum version 1.24)
Go version: go1.9.2
Git commit: 80c8033
Built: Thu Dec 14 00:49:52 2017
OS/Arch: windows/amd64
Experimental: true

Same results using experimental false btw.

Inspecting the Microsoft-Windows-Hyper-V-Compute Event Logs I hopefully see something of importance.

First a run of the "microsoft/nanoserver" container without security-ops credspec settings.

[37beabaef4fd79d82e2da81913edfded4da403e92809b1ead38de6d4ccca07e5] Create Container, type 'Hyper-V Container', settings '{"SystemType":"Container","Name":"37beabaef ....

Then the run of the same container with the credentialspecs set.

[73c4a69ba54c1d8c82a0ee4db81d4537d5fdd0de431fe266a914b8ed02d6b4a3] Create Container, type 'Hyper-V Container', settings ''

Attached is a file containing the log entries referenced above from the Hyper-V-Compute log,
as well as the docker services.log with debug enabled.

debug-log-win-hyperv.txt

@jrottmann

This comment has been minimized.

Copy link

jrottmann commented Feb 16, 2018

I'm experiencing the same problem on Windows 10 Version 1709 (Build 16299.125). Is anyone working on this issue? Are any workarounds known to access domain resources from inside a container?

@smasherprog

This comment has been minimized.

Copy link

smasherprog commented Mar 23, 2018

I am having the same issue... I am on 1709 as well

@mateuszdrab

This comment has been minimized.

Copy link

mateuszdrab commented Apr 7, 2018

I justed experienced the same issue on Server 1709 with a Docker image running under --isolation hyperv however the issue ceased as soon as I rebuilt my image with the matching base image of windowsservercore 1709 and stopped using the isolation switch. Hopefully this gets resolved in the next build.

@tparduejr

This comment has been minimized.

Copy link

tparduejr commented Apr 18, 2018

@mateuszdrab , you said Server 1709....was this on Windows 10? If so, can you give more details?

@mateuszdrab

This comment has been minimized.

Copy link

mateuszdrab commented Apr 18, 2018

@tparduejr That was on Windows Server, 1709 with Windows Server 2016 image running in hyper-v isolation mode. After rebuilding the container to 1709 base, the issue was no longer the case (as there was no need to use hyper-v isolation). I'd assume that the issue here is Hyper-V isolation. Can try to reproduce this in Win 10 or maybe someone else can.

@smasherprog

This comment has been minimized.

Copy link

smasherprog commented Apr 18, 2018

I get this on windows 10 1709 fully patched when trying to use credentials starting a container. It should be a simple repro: just try to use domain credentials when starting a container-- it will time out

@jrottmann

This comment has been minimized.

Copy link

jrottmann commented Apr 18, 2018

@PatrickLang do you know if there are any plans from the hyper-v devs to get this fixed?

@PatrickLang

This comment has been minimized.

Copy link
Contributor

PatrickLang commented Apr 18, 2018

Looks like it broke again. I know they were looking at it recently so I'll ping them. If you can get support cases open directly with Microsoft that will help expedite it since a Windows hotfix is needed. This affects both Windows 10 version 1709 and 1803.

Breadcrumb for myself - internal issue # is 15707011

@erwabo

This comment has been minimized.

Copy link

erwabo commented Apr 26, 2018

I was told from the premiere support team: "Microsoft does not engage on any Docker issues where the configuration is not WS2016 with Docker Enterprise Edition installed. That means no Design Change Request will be accepted. The bottom line is that this is NOT supported by Microsoft--customer needs to engage Docker support at their website: https://www.docker.com/get-docker (Docker Website with Product information), click on the link that says learn more about EE and it should direct you to the right option" Im not sure if one hand talks to the other over there, but hopefully someone is working on this.

@erwabo

This comment has been minimized.

Copy link

erwabo commented May 16, 2018

Has anyone heard anything else regarding this issue?

@josephderonde

This comment has been minimized.

Copy link

josephderonde commented Jun 4, 2018

This is basically stopping us from adopting docker at the moment, as we can not get windows authentication working for development purposes on an intranet solution. Has anyone found a way around this, like building locally to a server 2016 docker enterprise box?

@PatrickLang

This comment has been minimized.

Copy link
Contributor

PatrickLang commented Jun 4, 2018

It's still not fixed yet. I've had a hard time getting an ETA.

The best workarounds for now are to either:

  • Build on Windows Server 2016 - you can use process or hyperv isolation there
  • Build on Windows Server version 1709 or version 1803 - process isolation only
@josephderonde

This comment has been minimized.

Copy link

josephderonde commented Jun 5, 2018

So is there any way to get Visual Studio to build on a remote docker host? I am struggling to find the information on it. I don't think it would be ideal to have VS installed on the Server 2016 image.

@erwabo

This comment has been minimized.

Copy link

erwabo commented Aug 7, 2018

I have not really found a way to do that, however, I am just building in VS then pushing the image to a registry and then pulling it to the docker host and building it that way, That seems to work well and have not really run into any issues....but still cant do the credential_spec on Windows 10

@erwabo

This comment has been minimized.

Copy link

erwabo commented Sep 14, 2018

Has anyone heard any updates on this issue or had any success with the release of Windows 10 1803?

@LeEmo86

This comment has been minimized.

Copy link

LeEmo86 commented Sep 14, 2018

I've been hitting my head against this issue in Win 10 v1803 recently. I'd love to get my hands on a fix or work around.

@jds80021

This comment has been minimized.

Copy link

jds80021 commented Oct 29, 2018

So the facts are: a windows container won't run on a windows host with windows active directory authentication? I am starting to wonder what all the hype around docker is given that most organizations that use windows (file shares, sql server, etc.) use active directory authentication for access control. No one stores user names and passwords in config files anymore. Other than changing your security architecture, how are developers getting around this problem? Unfortunately my organization has abandoned any effort moving forward with docker because of this.

@carlfischer1

This comment has been minimized.

Copy link
Contributor

carlfischer1 commented Nov 9, 2018

There are fixes in Windows Server 2019 / Windows 10 1809 that allow credential specs to work with hyper-v isolated containers. Windows 10 1809 is also the first desktop version to support process isolated containers, which should also enable credential specs to work.

@Alexander-Bartosh

This comment has been minimized.

Copy link

Alexander-Bartosh commented Nov 9, 2018

Should than the issue be verified and closed?

@LeEmo86

This comment has been minimized.

Copy link

LeEmo86 commented Nov 11, 2018

AFAIA the Windows 10 1809 update is still pulled from the Windows Update servers, so I think we'll have to wait for it to safely roll out to most people before we can validate this.

@Alexander-Bartosh

This comment has been minimized.

Copy link

Alexander-Bartosh commented Nov 20, 2018

Works for me on Windows 10: Version 1809 Build 17763.134

@carlfischer1 Could you please let us know how to enable process isolation in Windows 10 1809 ?

@iofacture

This comment has been minimized.

Copy link

iofacture commented Jan 10, 2019

I believe Windows 10: Version 1809 went out via windows update on 01/08/2019 - I've tested it on build 17763.253 and can confirm the credential spec file does work, or at least does not timeout

@jds80021

This comment has been minimized.

Copy link

jds80021 commented Feb 12, 2019

This issue seems to be back. Running Windows 10, version 10.0.17763.0. Everything was working fine after update Version 1809 Build 17763.134. After installing the latest update windows-10-update-kb4476976 the problem is now back. Anyone else seeing this ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.