Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Packer 1.4.0 causes Ultra Defrag to hang indefinitely #259

Closed
jringold opened this issue Apr 27, 2019 · 30 comments
Closed

Packer 1.4.0 causes Ultra Defrag to hang indefinitely #259

jringold opened this issue Apr 27, 2019 · 30 comments
Labels

Comments

@jringold
Copy link

jringold commented Apr 27, 2019

  • Operating System Version: Windows 10
  • Provider (VirtualBox/VMWare): Virtualbox
  • Vagrant Version: 2.2.4
  • Packer Version: 1.4.0
  • Are you using stock boxes (downloaded) or were they built from scratch using Packer? build from Packer
  • Is the issue reproducible or intermittent? reproducible, eveytime

Please verify that you are building from an updated Master branch before filing an issue.

Description of the issue:

While building the Windows 2016 hosts host, the build process stops after running sdelete.

virtualbox-iso:  cmd /c C:\Windows\Temp\sdelete.exe -q -z C:
virtualbox-iso: )
virtualbox-iso: UltraDefrag 6.1.0, Copyright (c) UltraDefrag Development Team, 2007-2013.
virtualbox-iso: UltraDefrag comes with ABSOLUTELY NO WARRANTY. This is free software,
virtualbox-iso: and you are welcome to redistribute it under certain conditions.

Link to Gist Containing Build Logs:

https://gist.github.com/jringold/c32d4ac83d1e7820319c364d68a965ca

@clong
Copy link
Owner

clong commented Apr 27, 2019

SDelete can take an extremely long time to finish. How long did it run before exiting?

@jringold
Copy link
Author

jringold commented Apr 27, 2019

SDelete can take an extremely long time to finish. How long did it run before exiting?

It's been running 2 days. No disk activity on the virtual drive.

@clong
Copy link
Owner

clong commented Apr 27, 2019

Oh...yeah, that's not expected :-/

Let me see if I can repro on a Windows machine this weekend.

@clong
Copy link
Owner

clong commented Apr 28, 2019

Hey @jringold - I wasn't able to reproduce this yesterday. I rebuilt both win10 and win2016 on Virtualbox and VMware running on Ubuntu 16.04 and both completed without issue. I'll see if I can find a Windows host to test this on as it may be windows specific.

@clong clong added the bug label Apr 28, 2019
@jringold
Copy link
Author

I'm using the powershell script on windows, not the shell script on Ubuntu. Is there a difference in what's being called?

@ProtoDroidBot
Copy link

@clong @jringold I don't believe it matters which OS/script you run, as I ran into the same issue when I was building DetectionLab on Debian/Kali Linux using the shell script and same versions (packer version was 1.4.0 as well).

Interestingly, my attempt to build DetectionLab on Kali Linux failed at the same location and hangs after SDelete is "finished". I am doing two test builds on my Windows 10 x64 system; one test with packer 1.3.5, and the other with 1.4.0 and will see if it is anyway related to some new stderr / stdout "forwarding" that packer 1.4.0 does now (specifically from https://www.hashicorp.com/blog/announcing-packer-v-1-4-0 - "Shell communicator now forwards Stderr logs to the error output").

Apologies for any speculation and will see if I can upload my build logs to further diagnose the issue.

@clong
Copy link
Owner

clong commented Apr 29, 2019

Ah! I am using an older version of Packer - I didn't even realize 1.4 was out. That may explain it.

@jringold
Copy link
Author

Which version of Packer are you using to get it to build?

@clong
Copy link
Owner

clong commented Apr 29, 2019

Which version of Packer are you using to get it to build?

1.3.4

@ProtoDroidBot
Copy link

@clong , I just realized that the underlying issue might be related to the below messages which were sent to stdout previously, but as of packer 1.4.0, get redirected to stderr. IIRC, any stderr will cause packer to stop by default, regardless of the error.

From @jringold 's build log, but I got this as well when packer tried to "net stop / start wuauserv"...

==> virtualbox-iso: The Windows Update service is not started.
virtualbox-iso: C:\Users\vagrant>rmdir /S /Q C:\Windows\SoftwareDistribution\Download
==> virtualbox-iso: More help is available by typing NET HELPMSG 3521.
virtualbox-iso: C:\Users\vagrant>mkdir C:\Windows\SoftwareDistribution\Download
==> virtualbox-iso: System error 1058 has occurred.
==> virtualbox-iso: The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.
virtualbox-iso: C:\Users\vagrant>net start wuauserv

I can create a separate issue to track this "failed service starting", but I think that's the underlying issue why packer 1.4.0 is complaining now that this is getting sent to stderr rather than 1.3.x simply pushing it to stdout and proceeding regardless...

Will see if I can debug this as well while I am at work. :/

@ProtoDroidBot
Copy link

@jaweesh , if you absolutely need to build DetectionLab ASAP, I'd recommend downgrading to packer 1.3.5 temporarily while me and Chris attempt to pin down this issue. As per my above comment, I think I've pinned down the issue to stderr logging with packer 1.4.0 that is getting tripped by the Windows update service failing to shutdown/start up near the SDelete/UltraDefrag step.

Also, in your packer build logs, do you see something similar to the above comment where it complains that wuauserv could not be started or stopped? If so, then that pretty much confirms my suspicions that component (failing) is the cause of packer 1.4.0 throwing up when it hits a stderr.

@jaweesh
Copy link

jaweesh commented May 1, 2019

@ProtoDroidBot switching from Powershell ISE to Powershell.exe built windows_server2016 successfully without downgrading packer
will try and win10 build and see if it solve that without downgrading
:sorry i deleted the post:
also i had to enable WinRM manually by running the enable_winrm.ps1 from Floppy disk

@ProtoDroidBot
Copy link

Interesting, when you finish the build; could you share the .\DetectionLab\Packer"packer_build.log" file here? Kind of curious as to how this is building correctly on your end with Packer 1.4.0. Unless the issue is elsewhere, which might be the case as well...

Also for the WinRM issue, you might want to create a separate issue in GitHub for it and share the steps you took that required enable_winrm to be manually enabled.

Thanks @jaweesh ! :)

@ProtoDroidBot
Copy link

Actually, @clong, does packer generate a build log for Windows systems? After checking the build.ps1 PowerShell file, it is not clear where the logging is going to as the PACKER_LOG environment variable does not appear to be present in the .ps1 version of the file...

@jaweesh
Copy link

jaweesh commented May 1, 2019

Here are the logs from the stdout of powershell.exe, i couldn't find packer_build.log.
note I removed bellow lines from windows_10.json and windows_2016.json to speed up the build process

"./scripts/microsoft-updates.bat",
"./scripts/win-updates.ps1"

successful build logs from powershell.exe
https://gist.github.com/jaweesh/90ac96168e93298b625b53c82046f5fa

failed build logs from powershell_ise.exe
https://gist.github.com/jaweesh/321b3aa20eb94da40e9e356d240c5e9a

@ProtoDroidBot
Copy link

Thanks for the info @jaweesh ! I might need to file an issue for Packer logging on Windows/PowerShell so the PowerShell build.ps1 is in line with the Shell script

In other news, I am testing one modification to the ./DetectionLab/Packer/scripts/compact.bat file which will ignore stopping/(re)starting the service if it isn't running by the time the compact.bat file gets to SDelete.exe. Should band-aid fix this issue in the interim while everyone figures out the underlying issues with starting/stopping Windows Update. :(

So far, the build on my Ubuntu system is progressing just nicely without throwing the usual update "error(s)".

@jringold
Copy link
Author

jringold commented May 1, 2019

Downgrading Packer has resolved the issue and the build now completes with packer 1.3.4

@jaweesh
Copy link

jaweesh commented May 2, 2019

Yes Indeed, Packer 1.3.5 solved the build problems.

here is a gist with build stdout logs
https://gist.github.com/jaweesh/5dde756bcdd75c79db460c012d415a68

@clong clong changed the title Windows2016 build failure Packer 1.4.0 causes Ultra Defrag to hang indefinitely May 3, 2019
@ProtoDroidBot
Copy link

So the lead I was pursuing with regards to wuauserv "failing to start" didn't actually work and I still got a hanged packer install at the UltraDefrag location. Interestingly, in most (if not all cases) where Ultra Defrag hangs, a message similar to the below appears in the packer logs:

    virtualbox-iso: UltraDefrag 6.1.0, Copyright (c) UltraDefrag Development Team, 2007-2013.
    virtualbox-iso: UltraDefrag comes with ABSOLUTELY NO WARRANTY. This is free software,
    virtualbox-iso: and you are welcome to redistribute it under certain conditions.

I was tinkering around with the batch file and changing the cmd /c udefrag.exe to start /wait udefrag.exe somehow suppressed that message to packer in (interim) testing. I had a minor hunch which could be incorrect that UltraDefrag was stepping on SDelete with the new packer update BUT I need to do further testing before confirming this.

Build is still running and I have udefrag logging turned on in case something else "breaks". No promises that this seemingly minor change didn't cause additional issues or build times though. :( Apologies to @clong in advance in case it does.

@ProtoDroidBot
Copy link

I got packer 1.4.0 to successfully complete the WindowsServer2016 build and its generating the .ova export at ~12 PM EST.

What I do not understand currently is why "cmd /c (udefrag.exe)" suddenly didn't work with Packer 1.4.0 but "START /wait (udefrag.exe)" was able to complete just fine. 🤷‍♂

@clong as promised, below are the (partial) successful packer build and udefrag log files:

udefrag.log file from WindowsServer2016 machine - udebug.log
(packer_build.log) PowerShell output from packer with the successful completion of udefrag.exe; note this is a partial build log at the moment as its still churning away at the Windows10 box now. - packer_build.log / PowerShell output from host

@clong
Copy link
Owner

clong commented May 4, 2019

@ProtoDroidBot thanks for all the debugging you've been doing! Yeah, it's weird that START /wait would work -- I'll give it a shot in my lab here and see if that's a reliable fix.

@clong
Copy link
Owner

clong commented May 5, 2019

I made some small updates to compact.bat and vm-guest-tools.bat (but none that should affect UltraDefrag) last night and the build actually went fine on 1.4.0. I'm going to run one more tonight and see how things go to make sure it wasn't a fluke. If all is well, I'll push those changes and see if this fixes it for others.

@clong
Copy link
Owner

clong commented May 10, 2019

Hi folks - is anyone still running into this issue since #270 got merged?

@jaweesh
Copy link

jaweesh commented May 10, 2019

I downgraded and it works for me, i will try to rebuild a new one tomorrow and feedback

@clong
Copy link
Owner

clong commented May 11, 2019

Just tried to build again today and it hung. I'll try the START /wait fix and see if that takes care of it.

@ProtoDroidBot
Copy link

ProtoDroidBot commented May 11, 2019

@clong , which OS did you build off of? I built it on Debian/Kali Linux successfully using the latest repo and using Packer 1.4.0.

Trying now on Windows 10 with Packer 1.4.0 and without the START /wait fix.

Will upload build logs as I get them.

Edit 1: Kali/Debian Build logs as promised: Additional errors that it couldn't DL ultradefrag for some reason. :/ Packer 1.4.0 Build logs for Debian

Edit 2: Hit the sysprep stage for WindowsServer2016 (packer box), so unless Windows10 doesn't build, I cannot reproduce this on my end with the latest repo.

@clong
Copy link
Owner

clong commented May 12, 2019

@ProtoDroidBot this issue seems to be occurring intermittently for me. I've had it hang while building using VMware's provider on MacOS, but I've also had it succeed while building using the virtualbox provider on Ubuntu 16.04. I'm still not entirely sure what is causing the hang and why it seems to be intermittent

clong added a commit that referenced this issue May 13, 2019
@ProtoDroidBot
Copy link

Hey @clong ? In some testing I did, it looks like UltraDefrag opens a new window when executing the commandline variant of the program. I am not sure if/how this is interacting with the OS' that are building from Packer, but it would seem that its throwing other errors as well and might not be running at all. See my build logs above ( #259 (comment) )

Also, not sure if #283 is related to the recent changes done to compact.bat.

I am wondering if it would be better to ditch UltraDefrag altogether in favor of a different program or the stock Windows Defrag. IIRC, UltraDefrag went for the commercial route so support for older versions might be limited and with all of these issues, it might be harder to justify keeping it. Alternatively we could have a switch or an option to defrag/compact the VM images as well. Your call. :)

@clong
Copy link
Owner

clong commented May 27, 2019

I think this issue has been resolved via the start /wait workaround (thanks @ProtoDroidBot !). I was able to build the 2016 box yesterday without any issues using Packer v1.4. I tend to agree though, I'm not even sure defragging these VMs provides any noticeable performance benefits. It might be better to either use the native defrag utility or scrap defragging all together. I'm going to leave it for now, but if it causes issues down the road I'll probably remove it entirely.

@clong clong closed this as completed May 27, 2019
Selora pushed a commit to Selora/DetectionLab that referenced this issue Dec 3, 2019
@ProtoDroidBot
Copy link

I REALLY hate to reopen an old thread especially since the issue is "fixed", but I am building a separate Packer project from here https://github.com/StefanScherer/packer-windows and it appears that the "cmd /c" actually works now when I tested it with Packer version 1.5.1? This makes me think it was a bit of a regression, but IDK the full story and more light testing is needed.

:|

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants