Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with hyperv-ubuntu-16.04.json #2

Closed
evancox10 opened this issue Nov 21, 2016 · 20 comments
Closed

Issues with hyperv-ubuntu-16.04.json #2

evancox10 opened this issue Nov 21, 2016 · 20 comments

Comments

@evancox10
Copy link

evancox10 commented Nov 21, 2016

Hi, I'm trying to test these with this version of Packer w/ Hyper-V support, but getting the following errors:

PS C:\Users\Evan\vagrant> packer validate .\hyperv-ubuntu-16.04.json
Template validation failed. Errors are shown below.

Errors validating build 'hyperv-iso'. 2 error(s) occurred:

* unknown configuration key: "headless"
* unknown configuration key: "ram_size_mb"

Any ideas?

@taliesins
Copy link
Owner

@evancox10 remove headless and rename ram_size_mb to ram_size. My bad, I have updated this due to feedback from Packer team. I have not applied this change to the base box scripts yet.

@evancox10
Copy link
Author

Thanks, will give it a shot

@evancox10 evancox10 changed the title unknown configuration keys: headless and ram_size_mb Issues with hyperv-ubuntu-16.04.json Nov 22, 2016
@evancox10
Copy link
Author

evancox10 commented Nov 22, 2016

Alright, that got me past that. Ran into a few more issues:

  1. I have a symmetric 1 Gbps connection, so downloading the 700 MiB ISO should take 5 minutes max. But having packer download the ISO from the default url location was extremely slow, not sure of exact speed but would have taken >1 hour to finish. HTTP download via Chrome will do 150-250 Mbps, and running curl from within the W10 Bash + Ubuntu environment is similarly fast. I've noticed that Invoke-WebRequest is already slow, and tops out around 10 Mbps. If that is being used then it would help explain why.

  2. The first time I ran packer build (after providing the .iso myself), I received the following error:

PS C:\Users\Evan\vagrant\packer-baseboxes> packer build -var 'iso_url=./ubuntu-16.04.1-server-amd64.iso' hyperv-ubuntu-16.04.json
hyperv-iso output will be in this color.

==> hyperv-iso: Creating temporary directory...
==> hyperv-iso: Downloading or copying ISO
    hyperv-iso: Downloading or copying: file:///C:/Users/Evan/vagrant/packer-baseboxes/ubuntu-16.04.1-server-amd64.iso
==> hyperv-iso: Starting HTTP server on port 8298
==> hyperv-iso: Creating switch 'EVAN-HYPERV-VSWITCH' if required...
==> hyperv-iso:     switch 'EVAN-HYPERV-VSWITCH' already exists. Will not delete on cleanup...
==> hyperv-iso: Creating virtual machine...
==> hyperv-iso: Enabling Integration Service...
==> hyperv-iso: strconv.ParseUint: parsing "0 0": invalid syntax
==> hyperv-iso: Unregistering and deleting virtual machine...
==> hyperv-iso: Deleting output directory...
==> hyperv-iso: Deleting temporary directory...
Build 'hyperv-iso' errored: strconv.ParseUint: parsing "0 0": invalid syntax

==> Some builds didn't complete successfully and had errors:
--> hyperv-iso: strconv.ParseUint: parsing "0 0": invalid syntax

==> Builds finished but no artifacts were created.

Not sure where the ParseUint is coming from, kind of strange.

However, after rerunning the command in debug mode, this error did not show up. It also didn't show up upon running the command again in regular mode.

  1. Installing the server ISO seems to get stuck at a blank purple screen, not sure what exact step this is in the install process, but Packer is at "Waiting for SSH to become available..."
PS C:\Users\Evan\vagrant\packer-baseboxes> packer build -var 'iso_url=./ubuntu-16.04.1-server-amd64.iso' hyperv-ubuntu-16.04.json
hyperv-iso output will be in this color.


==> hyperv-iso: Creating temporary directory...
==> hyperv-iso: Downloading or copying ISO
    hyperv-iso: Downloading or copying: file:///C:/Users/Evan/vagrant/packer-baseboxes/ubuntu-16.04.1-server-amd64.iso
==> hyperv-iso: Starting HTTP server on port 8874
==> hyperv-iso: Creating switch 'EVAN-HYPERV-VSWITCH' if required...
==> hyperv-iso:     switch 'EVAN-HYPERV-VSWITCH' already exists. Will not delete on cleanup...
==> hyperv-iso: Creating virtual machine...
==> hyperv-iso: Enabling Integration Service...
==> hyperv-iso: Setting boot drive to os dvd drive C:/Users/Evan/vagrant/packer-baseboxes/ubuntu-16.04.1-server-amd64.iso ...
==> hyperv-iso: Mounting os dvd drive C:/Users/Evan/vagrant/packer-baseboxes/ubuntu-16.04.1-server-amd64.iso ...
==> hyperv-iso: Skipping mounting Integration Services Setup Disk...
==> hyperv-iso: Mounting secondary DVD images...
==> hyperv-iso: Configuring vlan...
==> hyperv-iso: Starting the virtual machine...
==> hyperv-iso: Waiting 5s for boot...
==> hyperv-iso: Host IP for the HyperV machine: 192.168.1.172
==> hyperv-iso: Typing the boot command...
==> hyperv-iso: Waiting for SSH to become available...

Still stuck there right now after some time. I've read reports of this clearing up on its own eventually. so we'll see where it's at tomorrow.

Update: Just saw that it errored out with "!!! Download debconf preconfiguration file: The file needed for preconfiguration could not be retreieved from http://192.168.1.172:8874/preseed.cfg. The installation will proceed in non-automated mode." However, checking in Chrome, curl, etc., I can see that this file is reachable via HTTP.

I'm not sure if these are issues with the packer baseboxes in this repo, or the packer-hyperv .exe itself. Would love to see this get included into the main packer branch, so just want to help squash bugs =) If you think I should raise these issues in your other repo, or in the PR to the master packer, let me know.

Thanks,
Evan

@evancox10
Copy link
Author

evancox10 commented Nov 23, 2016

UPDATE: Figured it out, Windows Firewall had inbound access to packer.exe blocked. Changed this to allow and it starts to proceed. Issue 2 from my post above is still a problem. After rebooting computer today, the parse UInt error happened on the first run of packer build. Subsequent runs did not encounter the error. Have not rechecked issue 1 today, presumably still slow but maybe not.


Some more info on the inability to access the preseed.cfg file from within the VM:

  1. I can ping the Hyper-V host from within the packer VM just fine, but running wget http://IP:PORT/preseed.cfg results in an indefinite hang at Connecting to IP:PORT.... I'll see if it eventually times out or does anything else, but it stays here for at least a few minutes.

  2. From within another, unrelated Hyper-V Ubuntu VM, I am also unable to access http://IP:PORT/preseed.cfg` through either FireFox or wget. Other HTTP services are normal. Can still access this from the Hyper-V host itself, however.

  3. Checking the system console log of the packer VM(ctrl+alt+F4), there don't seem to be any issues with the VM getting an IP address via DHCP, everything looks normal there. I can ping the VM from the host.

  4. I'm at a total as to how to debug network issues from within the Busy Box-based installer. No ifconfig, no netstat, etc.

  5. Host OS is Windows 10 Pro build 14971 (latest insider fast ring build).

  6. After about 15-20 minutes while I was actively debugging the above, the HTTP server seems to have completely dropped off. Can no longer see it from anywhere. This is despite no change in the status reported by packer (still at "Waiting for SSH to become available..."). I would expect the HTTP server to stay up until Packer itself timed out and gave up.


IMO, all of the above, and especially #2, point to a problem in Hyper-V and the networking it is providing to guests. Is it trying to filter out HTTP over non-standard ports? (I do not have the vswitch's filtering driver even enabled.) Some other issue? Maybe someone at @microsoft can help?

@taliesins
Copy link
Owner

@evancox10 for the firewall I think it was the same issue as #1

I am not sure what to do in this regard, other then possible adding some documentation. Any suggestions?

@taliesins
Copy link
Owner

@evancox10 for the slow ISO download speed. The download portion of the code is shared between all builders. So either we have an issue with GO downloading on Windows or the Packer implementation of downloading.

More then likely there is something on your host limiting the Packer.exe download speeds.

I don't notice the issue, as I have Teamcity agents using xcopy from a network share before running a build. I then reference the iso path locally, by overriding environment variable "IsoFileName"

@evancox10
Copy link
Author

evancox10 commented Nov 24, 2016

Re. firewall issues. Hard to go wrong with documentation for commonly encountered issues! Kind of funny that I ran into the same problem as the only other filed bug to the project. Should have checked that... my fault there.

Re. download speeds, must have been a server side issue, I just tried again and averaged around 100 Mbps with peaks to 200 Mbps. Some transient issue with Ubuntu release servers I suppose.

Also, any comments re. the strange "parse UInt error on initial run bug"? Let me know if there's any way I can get more debug information to try to pin down where this is coming from.

Lastly, I let a full run continue last night and I don't think it completed successfully. It said it finished, but didn't produce any artifacts. The full (really long) log is below.

PS C:\Users\Evan\vagrant\packer-baseboxes> packer build hyperv-ubuntu-16.04.json
hyperv-iso output will be in this color.

==> hyperv-iso: Creating temporary directory...
==> hyperv-iso: Downloading or copying ISO
    hyperv-iso: Downloading or copying: http://releases.ubuntu.com/16.04/ubuntu-16.04.1-server-amd64.iso
==> hyperv-iso: Starting HTTP server on port 8602
==> hyperv-iso: Creating switch 'EVAN-HYPERV-VSWITCH' if required...
==> hyperv-iso:     switch 'EVAN-HYPERV-VSWITCH' already exists. Will not delete on cleanup...
==> hyperv-iso: Creating virtual machine...
==> hyperv-iso: Enabling Integration Service...
==> hyperv-iso: Setting boot drive to os dvd drive C:\Users\Evan\vagrant\packer-baseboxes\packer_cache\f39e83ca80919c0a45010f5df4054d8237da20b99d0277c93c91
cbaaa12e7792.iso ...
==> hyperv-iso: Mounting os dvd drive C:\Users\Evan\vagrant\packer-baseboxes\packer_cache\f39e83ca80919c0a45010f5df4054d8237da20b99d0277c93c91cbaaa12e7792.
iso ...
==> hyperv-iso: Skipping mounting Integration Services Setup Disk...
==> hyperv-iso: Mounting secondary DVD images...
==> hyperv-iso: Configuring vlan...
==> hyperv-iso: Starting the virtual machine...
==> hyperv-iso: Waiting 5s for boot...
==> hyperv-iso: Host IP for the HyperV machine: 192.168.1.172
==> hyperv-iso: Typing the boot command...
==> hyperv-iso: Waiting for SSH to become available...
==> hyperv-iso: Connected to SSH!
==> hyperv-iso: Provisioning with shell script: ./linux/ubuntu/update.sh
<snip>
    hyperv-iso: Removing linux-libc-dev:amd64 (4.4.0-47.68) ...
    hyperv-iso: Removing manpages-dev (4.04-2) ...
    hyperv-iso: Removing patch (2.7.5-1) ...
    hyperv-iso: Processing triggers for man-db (2.7.5-1) ...
    hyperv-iso: Processing triggers for libc-bin (2.23-0ubuntu4) ...
    hyperv-iso: dd: error writing '/EMPTY': No space left on device
    hyperv-iso: 16577+0 records in
    hyperv-iso: 16576+0 records out
    hyperv-iso: 17381478400 bytes (17 GB, 16 GiB) copied, 26.9703 s, 644 MB/s
==> hyperv-iso: Gracefully halting virtual machine...
==> hyperv-iso: Clean up os dvd drive...
==> hyperv-iso: Unregistering and deleting virtual machine...
==> hyperv-iso: Deleting output directory...
==> hyperv-iso: Deleting temporary directory...
Build 'hyperv-iso' errored: Shutdown command has not successful.

ExitStatus: 2300218

Stdout:

Stderr:

==> Some builds didn't complete successfully and had errors:
--> hyperv-iso: Shutdown command has not successful.

ExitStatus: 2300218

Stdout:

Stderr:

==> Builds finished but no artifacts were created.

@evancox10
Copy link
Author

And after that, I've proceeded to get the strange EOF errors below. Could this have something to do with line-ending differences? I noticed that the Ubuntu .json files in your repo all use LF (Unix) endings, while the rest of the files use CR-LF. Should this match the host OS or the guest OS?? Does this even matter??? (It shouldn't IMHO).

PS C:\Users\Evan\vagrant\packer-baseboxes> rm -Recurse -Force .\packer_cache\
PS C:\Users\Evan\vagrant\packer-baseboxes> packer build hyperv-ubuntu-16.04.json
hyperv-iso output will be in this color.

==> hyperv-iso: Creating temporary directory...
==> hyperv-iso: Downloading or copying ISO
    hyperv-iso: Downloading or copying: http://releases.ubuntu.com/16.04/ubuntu-16.04.1-server-amd64.iso
    hyperv-iso: Download progress: 1%
    hyperv-iso: Download progress: 6%
    hyperv-iso: Download progress: 18%
    hyperv-iso: Download progress: 25%
    hyperv-iso: Download progress: 41%
    hyperv-iso: Download progress: 52%
    hyperv-iso: Download progress: 66%
    hyperv-iso: Download progress: 81%
    hyperv-iso: Download progress: 97%
    hyperv-iso: Download progress: 100%
==> hyperv-iso: Starting HTTP server on port 8140
==> hyperv-iso: Creating switch 'EVAN-HYPERV-VSWITCH' if required...
==> hyperv-iso:     switch 'EVAN-HYPERV-VSWITCH' already exists. Will not delete on cleanup...
==> hyperv-iso: Creating virtual machine...
==> hyperv-iso: Enabling Integration Service...
==> hyperv-iso: Setting boot drive to os dvd drive C:\Users\Evan\vagrant\packer-baseboxes\packer_cache\f39e83ca80919c0a45010f5df4054d8237da20b99d0277c93c91
cbaaa12e7792.iso ...
==> hyperv-iso: Mounting os dvd drive C:\Users\Evan\vagrant\packer-baseboxes\packer_cache\f39e83ca80919c0a45010f5df4054d8237da20b99d0277c93c91cbaaa12e7792.
iso ...
==> hyperv-iso: Skipping mounting Integration Services Setup Disk...
==> hyperv-iso: Mounting secondary DVD images...
==> hyperv-iso: Configuring vlan...
==> hyperv-iso: Starting the virtual machine...
==> hyperv-iso: Waiting 5s for boot...
==> hyperv-iso: Host IP for the HyperV machine: 192.168.1.172
==> hyperv-iso: Typing the boot command...
==> hyperv-iso: Waiting for SSH to become available...
Build 'hyperv-iso' errored: unexpected EOF

==> Some builds didn't complete successfully and had errors:
--> hyperv-iso: unexpected EOF

==> Builds finished but no artifacts were created.
PS C:\Users\Evan\vagrant\packer-baseboxes> packer build hyperv-ubuntu-16.04.json
Failed to initialize build 'hyperv-iso': error initializing builder 'hyperv-iso': EOF
hyperv-iso output will be in this color.


==> Builds finished but no artifacts were created.
PS C:\Users\Evan\vagrant\packer-baseboxes> packer build hyperv-ubuntu-16.04.json
Failed to initialize build 'hyperv-iso': error initializing builder 'hyperv-iso': EOF
hyperv-iso output will be in this color.


==> Builds finished but no artifacts were created.


@evancox10
Copy link
Author

evancox10 commented Dec 2, 2016

Hey, any ideas on the failed shutdown cmd issue? I tried your new 145 build and got the same result.

Also, once it errors out, should I expect it to clean up all the VMs/whatever packer created so that it starts from a clean slate? i.e. Is box creation supposed atomic, in that it either succeeds completely or leaves no trace that it happened? If so, it appears not to be cleaning up everything correctly, as the xenial-ubuntu VM is still present and running. I would expect this not to be the case, since you can see in this in the packer output: ==> hyperv-iso: Unregistering and deleting virtual machine... I believe that's where the EOF errors are coming from, because it was trying to recreate the box on top of the existing VM.

Edit: Accidentally hit "Close and comment" instead of just "comment"

Edit 2: Shutdown failure possibly related to hashicorp/packer#4134 ?

@evancox10 evancox10 reopened this Dec 2, 2016
@PatrickLang
Copy link

I'm hitting the same issue here too. I fetched the latest code today, built it, and have been debugging things. I think it's possible that sysprep could cause a firewall rule reset on Windows, but I really think the bigger problem is the Wait() statement is waiting on a successful result when the connection is about to fail. That would happen with both winrm and ssh.

@PatrickLang
Copy link

@taliesins
Copy link
Owner

From looking at the gist, I think it may be due to improvements in the ssh communicator. When we are emptying the hdd we write 0 to file system till there is no space left.

I think previously the error didn't bubble up. I will try to spin up an instance with the latest Packer and I will make sure error is swallowed when writing 0s

@PatrickLang
Copy link

I removed ./linux/ubuntu/cleanup.sh and it still fails. I don't think it's related to zeroing the HDD.

@PatrickLang
Copy link

Also happens in my fork of boxcutter/Ubuntu
https://github.com/PatrickLang/ubuntu/blob/hyperv2/hyperv_builder.md

I agree that it's probably due to changes in the ssh communicator. Windows doesn't use it and I have built several images correctly with clean shutdown through the winrm communicator.

@PatrickLang
Copy link

PatrickLang commented Dec 12, 2016

This looks like it could be the problem. I applied the fix from hashicorp/packer#4054 to hyperv-iso in my repo PatrickLang/packer@098abe9

Testing in progress. Got two different Ubuntu 16.04 builds going - generation 1 from boxcutter, generation 2 from your packer-baseboxes repo

@PatrickLang
Copy link

It worked - PR submitted

@evancox10
Copy link
Author

Thanks for the help with this Patrick!

Link to updated PR:
taliesins/packer#11

@taliesins
Copy link
Owner

Closing issue as it was resolved in #11

@marcofranssen
Copy link

marcofranssen commented May 30, 2017

Looks like this is also occuring for me. It creates a internal network switch and in powershell you see the host ip show up. However the ubuntu machine does seem to be able to resolve an IP address which basically causes this issue.

Is there any kind a network switch which has to be configured and provided as a param?

@taliesins
Copy link
Owner

@marcofranssen how are you assigning an ip to ubuntu machine. Do the virtual machine and the host have an ip assigned on the internal interfaces?

If you are using dhcp, make sure your dhcp server is configured on the internal network.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants