-
-
Notifications
You must be signed in to change notification settings - Fork 981
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Packer 1.4.0 causes Ultra Defrag to hang indefinitely #259
Comments
SDelete can take an extremely long time to finish. How long did it run before exiting? |
It's been running 2 days. No disk activity on the virtual drive. |
Oh...yeah, that's not expected :-/ Let me see if I can repro on a Windows machine this weekend. |
Hey @jringold - I wasn't able to reproduce this yesterday. I rebuilt both win10 and win2016 on Virtualbox and VMware running on Ubuntu 16.04 and both completed without issue. I'll see if I can find a Windows host to test this on as it may be windows specific. |
I'm using the powershell script on windows, not the shell script on Ubuntu. Is there a difference in what's being called? |
@clong @jringold I don't believe it matters which OS/script you run, as I ran into the same issue when I was building DetectionLab on Debian/Kali Linux using the shell script and same versions (packer version was 1.4.0 as well). Interestingly, my attempt to build DetectionLab on Kali Linux failed at the same location and hangs after SDelete is "finished". I am doing two test builds on my Windows 10 x64 system; one test with packer 1.3.5, and the other with 1.4.0 and will see if it is anyway related to some new stderr / stdout "forwarding" that packer 1.4.0 does now (specifically from https://www.hashicorp.com/blog/announcing-packer-v-1-4-0 - "Shell communicator now forwards Stderr logs to the error output"). Apologies for any speculation and will see if I can upload my build logs to further diagnose the issue. |
Ah! I am using an older version of Packer - I didn't even realize 1.4 was out. That may explain it. |
Which version of Packer are you using to get it to build? |
1.3.4 |
@clong , I just realized that the underlying issue might be related to the below messages which were sent to stdout previously, but as of packer 1.4.0, get redirected to stderr. IIRC, any stderr will cause packer to stop by default, regardless of the error. From @jringold 's build log, but I got this as well when packer tried to "net stop / start wuauserv"... ==> virtualbox-iso: The Windows Update service is not started. I can create a separate issue to track this "failed service starting", but I think that's the underlying issue why packer 1.4.0 is complaining now that this is getting sent to stderr rather than 1.3.x simply pushing it to stdout and proceeding regardless... Will see if I can debug this as well while I am at work. :/ |
@jaweesh , if you absolutely need to build DetectionLab ASAP, I'd recommend downgrading to packer 1.3.5 temporarily while me and Chris attempt to pin down this issue. As per my above comment, I think I've pinned down the issue to stderr logging with packer 1.4.0 that is getting tripped by the Windows update service failing to shutdown/start up near the SDelete/UltraDefrag step. Also, in your packer build logs, do you see something similar to the above comment where it complains that wuauserv could not be started or stopped? If so, then that pretty much confirms my suspicions that component (failing) is the cause of packer 1.4.0 throwing up when it hits a stderr. |
@ProtoDroidBot switching from Powershell ISE to Powershell.exe built windows_server2016 successfully without downgrading packer |
Interesting, when you finish the build; could you share the .\DetectionLab\Packer"packer_build.log" file here? Kind of curious as to how this is building correctly on your end with Packer 1.4.0. Unless the issue is elsewhere, which might be the case as well... Also for the WinRM issue, you might want to create a separate issue in GitHub for it and share the steps you took that required enable_winrm to be manually enabled. Thanks @jaweesh ! :) |
Actually, @clong, does packer generate a build log for Windows systems? After checking the build.ps1 PowerShell file, it is not clear where the logging is going to as the PACKER_LOG environment variable does not appear to be present in the .ps1 version of the file... |
Here are the logs from the stdout of powershell.exe, i couldn't find packer_build.log.
successful build logs from powershell.exe failed build logs from powershell_ise.exe |
Thanks for the info @jaweesh ! I might need to file an issue for Packer logging on Windows/PowerShell so the PowerShell build.ps1 is in line with the Shell script In other news, I am testing one modification to the ./DetectionLab/Packer/scripts/compact.bat file which will ignore stopping/(re)starting the service if it isn't running by the time the compact.bat file gets to SDelete.exe. Should band-aid fix this issue in the interim while everyone figures out the underlying issues with starting/stopping Windows Update. :( So far, the build on my Ubuntu system is progressing just nicely without throwing the usual update "error(s)". |
Downgrading Packer has resolved the issue and the build now completes with packer 1.3.4 |
Yes Indeed, Packer 1.3.5 solved the build problems. here is a gist with build stdout logs |
So the lead I was pursuing with regards to wuauserv "failing to start" didn't actually work and I still got a hanged packer install at the UltraDefrag location. Interestingly, in most (if not all cases) where Ultra Defrag hangs, a message similar to the below appears in the packer logs:
I was tinkering around with the batch file and changing the cmd /c udefrag.exe to start /wait udefrag.exe somehow suppressed that message to packer in (interim) testing. I had a minor hunch which could be incorrect that UltraDefrag was stepping on SDelete with the new packer update BUT I need to do further testing before confirming this. Build is still running and I have udefrag logging turned on in case something else "breaks". No promises that this seemingly minor change didn't cause additional issues or build times though. :( Apologies to @clong in advance in case it does. |
I got packer 1.4.0 to successfully complete the WindowsServer2016 build and its generating the .ova export at ~12 PM EST. What I do not understand currently is why "cmd /c (udefrag.exe)" suddenly didn't work with Packer 1.4.0 but "START /wait (udefrag.exe)" was able to complete just fine. 🤷♂ @clong as promised, below are the (partial) successful packer build and udefrag log files: udefrag.log file from WindowsServer2016 machine - udebug.log |
@ProtoDroidBot thanks for all the debugging you've been doing! Yeah, it's weird that |
I made some small updates to |
Hi folks - is anyone still running into this issue since #270 got merged? |
I downgraded and it works for me, i will try to rebuild a new one tomorrow and feedback |
Just tried to build again today and it hung. I'll try the |
@clong , which OS did you build off of? I built it on Debian/Kali Linux successfully using the latest repo and using Packer 1.4.0. Trying now on Windows 10 with Packer 1.4.0 and without the START /wait fix. Will upload build logs as I get them. Edit 1: Kali/Debian Build logs as promised: Additional errors that it couldn't DL ultradefrag for some reason. :/ Packer 1.4.0 Build logs for Debian Edit 2: Hit the sysprep stage for WindowsServer2016 (packer box), so unless Windows10 doesn't build, I cannot reproduce this on my end with the latest repo. |
@ProtoDroidBot this issue seems to be occurring intermittently for me. I've had it hang while building using VMware's provider on MacOS, but I've also had it succeed while building using the virtualbox provider on Ubuntu 16.04. I'm still not entirely sure what is causing the hang and why it seems to be intermittent |
Hey @clong ? In some testing I did, it looks like UltraDefrag opens a new window when executing the commandline variant of the program. I am not sure if/how this is interacting with the OS' that are building from Packer, but it would seem that its throwing other errors as well and might not be running at all. See my build logs above ( #259 (comment) ) Also, not sure if #283 is related to the recent changes done to compact.bat. I am wondering if it would be better to ditch UltraDefrag altogether in favor of a different program or the stock Windows Defrag. IIRC, UltraDefrag went for the commercial route so support for older versions might be limited and with all of these issues, it might be harder to justify keeping it. Alternatively we could have a switch or an option to defrag/compact the VM images as well. Your call. :) |
I think this issue has been resolved via the |
Attempting to address issue clong#259
I REALLY hate to reopen an old thread especially since the issue is "fixed", but I am building a separate Packer project from here https://github.com/StefanScherer/packer-windows and it appears that the "cmd /c" actually works now when I tested it with Packer version 1.5.1? This makes me think it was a bit of a regression, but IDK the full story and more light testing is needed. :| |
Please verify that you are building from an updated Master branch before filing an issue.
Description of the issue:
While building the Windows 2016 hosts host, the build process stops after running sdelete.
Link to Gist Containing Build Logs:
https://gist.github.com/jringold/c32d4ac83d1e7820319c364d68a965ca
The text was updated successfully, but these errors were encountered: