[Windows] windowsfilter folder impossible to delete

[charlessolar]

I've been running TP5 for several months on 3 machines - I noticed the hard drive filling up due to the amount of images laying around. Following some other posts I used docker rm -v $(docker ps -q) and docker rmi $(docker images -q) and similar to clean the disk. But I ended up breaking docker.

When I try to start a container now I only get Win32: Access is denied

{"failed": true, "item": "1", "rc": 1, "stderr": "C:\\Program Files\\Docker\\docker.exe: Error response from daemon: container 5113bc8f16fa56787a4495f56206304fbead47fb227a015970efb345baab3e20 encountered an error during CreateProcess failed in Win32: Access is denied. (0x5) extra info: {\"ApplicationName\":\"\",\"CommandLine\":\"cmd /S /C Domain.exe\",\"WorkingDirectory\":\"C:\\\\bin\",\"Environment\":{\"CONT_IMG_VER\":\"v1.0.0\"},\"EmulateConsole\":false,\"CreateStdInPipe\":true,\"CreateStdOutPipe\":true,\"CreateStdErrPipe\":true,\"ConsoleSize\":[0,0]}.\n", "stdout": "5113bc8f16fa56787a4495f56206304fbead47fb227a015970efb345baab3e20\n", "stdout_lines": ["5113bc8f16fa56787a4495f56206304fbead47fb227a015970efb345baab3e20"]}

I believe this happened because I broke some sym links in the docker store aka C:/programdata/docker/windowsfilter - perhaps these commands accidentally deleted a chunk of files that are still needed thereby screwing everything up.

Thats my theory. But the bug I am reporting is - I can't reset docker. So this happened, I just want to delete everything from docker - everything under C:/programdata/docker and restart the service in a fresh state. But I can't because windowsfilter is a sea of "Access denied" errors.

I've tried taking control of all the files in several ways, using several tools including rimraf and fsutil, powershell, cmd - all of it fails.

Which leaves the only option to "fix" docker to reformat the machine...

So tldr: Users should be able to "uninstall" docker and restore the host to a "clean" state - which is not possible solely because of windowsfilter permissions

[thaJeztah]

ping @jstarks

[jstarks]

We have a tool for this... @jhowardmsft can you help?

[friism]

It's here: https://github.com/jhowardmsft/docker-ci-zap

You have to compile it, then run: .\docker-ci-zap.exe -folder "C:\ProgramData\docker"

[charlessolar]

Thanks! Ill have to setup golang on my server host - but thats better than having to reformat.
Any chance of getting this included as part of #18601 or some docker --nuke option .. haha

[lowenna]

Note it is a REALLY dangerous utility. It can do some serious damage. Use with caution, no warranty on misuse or otherwise etc. No plans at all to include it in docker.

[charlessolar]

Gotcha - I'll keep an eye out for what command exactly caused this error state then

[justincormack]

I a going to close this as it seems resolved, hopefully other people needing the dangerous powertools will find it. Feel free to continue discussion if there are further problems

[micdenny]

I had the same problem, I resolved by simply take the ownership of the folder, then add myself with full control and flag to do the same on all the child with inheritance enabled. Then I simply deleted the folder as usual.

[mapitman]

Using WIndows 10...
I'm having some problems using this tool. I've taken ownership of the folder and all files recursively and I get this when I try to delete a layer:

> docker-ci-zap.exe -folder C:\ProgramData\Docker\windowsfilter\10c92b07d00274425bba2dc4ccad7f080f3290d3e9fd9fddd21575db50ea6175
time="2016-11-04T16:30:34-07:00" level=error msg="hcsshim::DestroyLayer - Win32 API call returned error r1=0x80070514 err=Not all privileges or groups referenced are assigned to the caller.id=10c92b07d00274425bba2dc4ccad7f080f3290d3e9fd9fddd21575db50ea6175 flavour=0"
ERROR:  hcsshim::DestroyLayer - Win32 API call returned error r1=0x80070514 err=Not all privileges or groups referenced are assigned to the caller.id=10c92b07d00274425bba2dc4ccad7f080f3290d3e9fd9fddd21575db50ea6175 flavour=0

[charlessolar]

In my experience you have to nuke the entire folder you can't delete a
single folder inside windows filter. Point the program to
c:/programdata/docker

On Nov 4, 2016 18:34, "Mark Pitman" notifications@github.com wrote:

I'm having some problems using this tool. I've taken ownership of the
folder and all files recursively and I get this when I try to delete a
layer:

docker-ci-zap.exe -folder C:\ProgramData\Docker\windowsfilter\10c92b07d00274425bba2dc4ccad7f080f3290d3e9fd9fddd21575db50ea6175
time="2016-11-04T16:30:34-07:00" level=error msg="hcsshim::DestroyLayer - Win32 API call returned error r1=0x80070514 err=Not all privileges or groups referenced are assigned to the caller.id=10c92b07d00274425bba2dc4ccad7f080f3290d3e9fd9fddd21575db50ea6175 flavour=0"
ERROR: hcsshim::DestroyLayer - Win32 API call returned error r1=0x80070514 err=Not all privileges or groups referenced are assigned to the caller.id=10c92b07d00274425bba2dc4ccad7f080f3290d3e9fd9fddd21575db50ea6175 flavour=0

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#26873 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABq00gjWAGf-TyCSZ45RvY50eQvWlM6-ks5q68DwgaJpZM4KFXuY
.

[mapitman]

Thanks for the tip, but I get the same error when I do that. For now I am renaming the directory and reinstalling docker. I'd still like to free up the space though, so if anyone else has any ideas, I'd appreciate it!

[charlessolar]

Also make sure the docker service isn't running - and sometimes i've had to switch it to manual and restart the computer. (Manual so that it wont start on boot)

[mapitman]

Docker has been uninstalled. I set chkdsk to run on next boot and rebooted my system. Then I ran the docker-ci-zap tool against the c:\ProgramData\Docker directory and it has deleted everything. Hopefully this thread will help someone else!

The reason I started trying to delete these folders in the first place was because I was having problems pulling the microsoft/windowsservercore image and it was complaining about files in that path. I uninstalled docker, deleted the whole c:\ProgramData\Docker directory, re-installed Docker and everything was peachy. Then I started running into problems again, so I thought I could do that whole process again, but this time I couldn't just delete the directories.

Hopefully some of this stuff gets worked out over time.

[lowenna]

You probably had a container still running, or a VHD still mounted as a result. From PowerShell, get-computeprocess | stop-computeprocess might have helped. Or going into diskmgmt.msc and unmounting any possible left over mounted VHDs.

[clarity99]

trying this as well:
C:\ProgramData>docker-ci-zap.exe -folder c:\ProgramData\Docker
time="2016-11-17T20:22:41+01:00" level=error msg="hcsshim::DestroyLayer - Win32 API call returned error r1=0x80070003 err=The system cannot find the path specified.id=Docker flavour=0"
ERROR: hcsshim::DestroyLayer - Win32 API call returned error r1=0x80070003 err=The system cannot find the path specified.id=Docker flavour=0

is it because I deleted some files manually from windowsfilter?

[clarity99]

well, it left just an empty windowsfilter dir which I could then delete from explorer. weird.

[lowenna]

@clarity99 Very unlikely. Are you elevated - you would need to be. Could be locked files

[clarity99]

yes, i was in an elevated prompt. docker was already uninstalled.

[artisticcheese]

Don't delete that folder! It has links to files outside of that folder. You will end up destroying your OS. Just end up restoring OS from backup.

[lowenna]

It has links to files outside of that folder.

No, it doesn't.

[artisticcheese]

Yes it does.

 Directory of C:\ProgramData\Docker\windowsfilter\c948d00a1becae853147fd0c58bb75fe715b5af2d5f19365df20143387461f09\Files\Users\All Users\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Oracle\Java\javapath

09/20/2016  11:14 AM    <SYMLINK>      java.exe [C:\Program Files\Java\jre8\bin\java.exe]
09/20/2016  11:14 AM    <SYMLINK>      javaw.exe [C:\Program Files\Java\jre8\bin\javaw.exe]
09/20/2016  11:14 AM    <SYMLINK>      javaws.exe [C:\Program Files\Java\jre8\bin\javaws.exe]

Directory of C:\ProgramData\Docker\windowsfilter\c948d00a1becae853147fd0c58bb75fe715b5af2d5f19365df20143387461f09\Files\Users\All Users

11/07/2016  07:42 AM    <JUNCTION>     Application Data [C:\ProgramData]
11/07/2016  07:42 AM    <JUNCTION>     Desktop [C:\Users\Public\Desktop]
11/07/2016  07:42 AM    <JUNCTION>     Documents [C:\Users\Public\Documents]
11/07/2016  07:42 AM    <JUNCTION>     Start Menu [C:\ProgramData\Microsoft\Windows\Start Menu]
11/07/2016  07:42 AM    <JUNCTION>     Templates [C:\ProgramData\Microsoft\Windows\Templates]
               0 File(s)              0 bytes

[lowenna]

Did you manually create that in some way?

Edit: To add, the CI servers docker-ci-zap this all the time, have been since day 1 and I've never seen any issue of OS corruption.

[artisticcheese]

Of course not

…

On Dec 1, 2016 7:01 PM, "John Howard" ***@***.***> wrote: Did you manually create that in some way? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#26873 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMkQNLpwJeKj7-zNQWpBFrJEEWjm1LLEks5rD23jgaJpZM4KFXuY> .

[jstarks]

docker-ci-zap will not follow symlinks, so this is fine.

[artisticcheese]

Some people in this thread and others on internet recommended to use explorer or rmdir to delete that folder which is what I tried and end up deleting almost entirely my profile off Windows. I was wondering why I keep getting access denied in deleting some folders while now I understand because it was actively used by user processes. Not sure how I end up with those but I did not do anything funky.

[artisticcheese]

There is some cyclic recurrance looks like happening as well in those links as well

Users\All Users\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data

[charlessolar]

Unfortunately general suggested solution for "Access denied" while deleting folders is rimraf or explorer taking forceful ownership. Which IS a definite bad idea for this folder because of the symlinks.
I had to reinstall windows server 3-4 times before opening this issue and getting docker-ci-zap - since then however everything has been fine

[artisticcheese]

I'm concerned that those links per docker/MS shall not be there, more then accidentally deleting them. Unless I misunderstood comment from @jhowardmsft

[jstarks]

I see. Yes, unfortunately, if you do delete via something that follows symlinks/junction points, you're going to have a bad time. I don't have a great solution to that problem other than "don't do that"... We are working on ensuring that it is much harder to get into a situation where you need to delete this folder.

[stooboo]

doh!

...if anyone else gets the msg ..
"This version of (blah blah blah)\docker-ci-zap.exe is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher."

I got it when I put docker-ci-zap.exe on my E: drive :-(

... so ..put it on C: before executing ... and it works ! ;-)

[segilbert]

This worked for me! I reclaimed 99.6 gigs of space.

Download
https://github.com/jhowardmsft/docker-ci-zap/blob/master/docker-ci-zap.exe

Stop Docker

Run
.\docker-ci-zap.exe -folder "C:\ProgramData\docker

Start Docker

[rs38]

so there is no other way yet to delete those folders?

 icacls "C:\ProgramData\Docker" /T /C /grant Administrators:F

runs indeed also into errors

[friism]

@rs38 please see the above comment: #26873 (comment)

(but be careful when using that tool)

[rs38]

the link does not work.
my comment was a bit misleading. I meant:

"so there is no other way yet than docker-ci-zap.exe to delete those folders?"

[lowenna]

The problem is that you need backup privilege to delete those files - similar to the windows.old folder you see when you upgrade Windows. That tool (heed the warning thrice over, you can do some catastrophic damage with it) is one of very few that can do it.

[artisticcheese]

Issue is not with permissions really but due to the hard links to other files in filesystem and normal explorer/cmd/powershell delete operation will follow a link and try to delete OS system files.

[rs38]

great answers with plausible background info. thanks guys!

[bzier]

It's worth pointing out that the program returns a somewhat cryptic error if you have a trailing backslash on the folder path. With or without quotes didn't matter (presumably it would if you had spaces in the path, but clearly most of us aren't dealing with that here).

Error:

PS C:\> .\docker-ci-zap.exe -folder C:\ProgramData\docker\
time="2017-09-29T11:49:04-07:00" level=error msg="hcsshim::DestroyLayer - Win32 API call returned error r1=0x8007007b err=The filename, directory name, or volume label syntax is incorrect.id= flavour=0"
ERROR:  hcsshim::DestroyLayer - Win32 API call returned error r1=0x8007007b err=The filename, directory name, or volume label syntax is incorrect.id= flavour=0

PS C:\> .\docker-ci-zap.exe -folder "C:\ProgramData\docker\"
time="2017-09-29T11:50:24-07:00" level=error msg="hcsshim::DestroyLayer - Win32 API call returned error r1=0x8007007b err=The filename, directory name, or volume label syntax is incorrect.id= flavour=0"
ERROR:  hcsshim::DestroyLayer - Win32 API call returned error r1=0x8007007b err=The filename, directory name, or volume label syntax is incorrect.id= flavour=0

Success:

PS C:\> .\docker-ci-zap.exe -folder C:\ProgramData\docker
INFO: Zapped successfully

PS C:\> .\docker-ci-zap.exe -folder "C:\ProgramData\docker"
INFO: Zapped successfully

[johnnyoshika]

Downloading docker-ci-zap.exe from https://github.com/jhowardmsft/docker-ci-zap and then running this worked:

.\docker-ci-zap.exe -folder "C:\ProgramData\docker"

I had to run cmd as administrator, otherwise it didn't work for me.

[kae36]

I uninstalled docker and tried to remove C:\ProgramData\Docker with explorer and couldn't remove it,. I rebooted thinking that would clear any running processes up, but I still couldn't remove the Docker directory. So I ran into this thread and then tried running

docker-ci-zap.exe -folder "C:\ProgramData\docker"

in an administrator cmd tool window and got this error message.

time="2017-12-12T15:24:41-05:00" level=error msg="hcsshim::DestroyLayer - Win32 API call returned error r1=0x80070020 eror=The process cannot access the file because it is being used by another process.id=Docker flavour=0" ERROR: hcsshim::DestroyLayer - Win32 API call returned error r1=0x80070020 err=The process cannot access the file because it is being used by another process.id=Docker flavour=0

I don't see any docker processes running (and I rebooted since I've removed docker).

I ran the PowerShell command get-computeprocess as administrator and it returns nothing and I can't see any virtual mounts and there doesn't seem to be any docker service processes.

Any Ideas? I just can't get rid of the C:\ProgramData\Docker folder.

Edited to add (to my shame) the following...
Okay, Ignore this. I found an explorer window that had the C:\ProgramData\Docker\windowsfilter folder open. I feel quite embarrassed now. My Docker folder is now gone.

[ghost]

Point of curiousity...Why not start in safe mode or repair mode with cmd as admin. Then rmdir?

[sstaszkiewicz-copperleaf]

Because #26873 (comment)

[sparrowt]

I eventually realised my problem was that C:\ProgramData\Docker\windowsfilter contents were for Windows containers but I was in Linux containers mode. When using Linux containers, Docker commands (e.g. docker system prune) only affect or 'see' linux containers (stored inside MobyLinuxVM.vhdx) so you need to switch to Windows containers mode before you can diagnose disk usage in windowsfilter folder.

Right-click on the Docker tray icon and Switch to Windows containers... after which docker system df will now show you which windows images are responsible for the disk usage.

If you've checked docker system df and tried docker system prune in both Windows and Linux mode and still have unexplained space usage, then I suggest rather than trying to delete files in windowsfilter manually (risky), instead try this script Find-OrphanDockerLayers.ps1 with the -RenameOrphanLayers argument (from an elevated powershell prompt, when in 'Windows containers' mode). Then when the docker service next shuts down it cleans up the windowsfilter/<HASH>-removing folders. Note: I think it has a timeout because it only cleans up a couple of GB per service shutdown, so you may have to cycle through this a few times until it's all gone.

[Mario-Hofstaetter]

so there is no other way yet to delete those folders?

 icacls "C:\ProgramData\Docker" /T /C /grant Administrators:F

For anyone google ing and landing here, this enabled me to delete C:\ProgramData\Docker\windowsfilter after moving to D:\
At least on Windows Server 2019 Standard Evaluation Edition, Build 10.0.17763.379
Thanks @rs38

[Cloudmersive]

How is this still not fixed?

[masters3d]

The only reliable way I found is to rename each of the folders with -removing at the end.

Get-ChildItem -Path C:\ProgramData\Docker\windowsfilter -Directory | % {Rename-Item $_.FullName "$($_.FullName)-removing" -ErrorAction:SilentlyContinue}

Source https://stackoverflow.com/a/66590524

Then restart docker-desktop a few times. Only a couple get deleted upon restart of docker-desktop

[mjfos2r]

still isn't fixed.

[elenzil]

my Win10 process for deleting the windowsFilter folder is to format the drive:

1. be fortunate enough to have sufficient free space to copy everything except windowsFilter.
2. 'shrink' the partition which windowsFilter is on.
3. make a new partition.
4. copy everything except windowsFilter to the new partition.
5. format the old partition.
6. copy everything back.

obviously this is super cumbersome, but for me beats running a tool that comes with huge CaveatEmptor warnings, worrying about the behavior of hard-links, etc.

[dazinator]

The zap tool is not working for me. My docker folder is on E:/ drive, not c:/. This is the output of running the tool.

.\docker-ci-zap.exe -folder "E:\Docker"
time="2021-11-17T22:38:32Z" level=error msg="hcsshim::DestroyLayer - Win32 API call returned error r1=0x80004005 err=Failed to load vmcompute.dll: The specified module could not be found.id=Docker flavour=0"
ERROR:  hcsshim::DestroyLayer - Win32 API call returned error r1=0x80004005 err=Failed to load vmcompute.dll: The specified module could not be found.id=Docker flavour=0

I do not have docker installed any longer as I uninstalled it.

UPDATE: Looks like you need to make sure Hyper-V platform windows feature is enabled, before you can execute the zap tool - it runs after I did that.

[bzuillsmith]

TLDR: Docker Desktop needs a direct in-program way to change the storage location of images and, at the very least, warnings and guidance about clearing old images when when switching where these images are stored. Probably similar guidance when uninstalling. If not remove the images for us, it should give us guidance about how to do it right.

My experience:

I wanted to clear the images from my primary drive and move them to a secondary. I set the "graph" path and restarted Docker. Then Cut/Paste of the Docker program data failed so I just decided to delete it. That failed with permission issues.

I then used the takeown + icacls commands as some have recommended here and elsewhere (danger, I do not recommend it!). Unfortunately I did not know about the symlinks yet. Had no idea this was a possibility. I just assumed the docker image files were isolated. The logs were so huge I didn't take a close enough look.

I then ran a few deletes from the explorer and then realized it would be faster using rmdir /s. After it was all done, I skipped back through some of the logs and happened to notice ownership changes for files that were definitely not part of a Docker image...

My OS doesn't seem to be bricked and I haven't noticed problems yet, but I have clearly altered permissions and ownership on a ton of files unintentionally and I cannot find documentation where all these hard links go and why. My terminal buffer was not long enough to go back and see all the damage. Hard for me to know what potential security holes it has opened up. I'm wondering if I need to reinstall windows just to be safe or if it only affected user files like ProgramData files (it definitely changed ownership of some ProgramData files not in Docker).