-
Notifications
You must be signed in to change notification settings - Fork 21.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
azure-arm WinRM Time out #31188
Comments
@Anbukugan please provide me with the link to the document you are following so I can better assist. |
@MicahMcKittrick-MSFT I followed the guide form these websites below https://docs.microsoft.com/en-us/azure/virtual-machines/windows/build-image-with-packer |
@cynthn Looks like I am seeing issues as well running through the document as is
@TomArcherMsft are you still/ also working on packer stuff? Any suggestions? |
@Anbukugan can you try increasing the WinRM timeout to 10m? "winrm_timeout": "10m", This fixed the issue for me. |
@Anbukugan I updated the doc and increased the default timeout period from 3 minutes to 5 which seems to have fixed the error in my tests. I will close this out but if you are still having issues after increasing the timeout period just let me know. |
i am using same code with |
I have the same issue, with the same configuration. On the hashicorp site I'm seeing others have the same issue, and indeed if I change from "2016-datacenter" to "2019-datacenter" the build runs just fine. This is happening for me both in Azure Devops with hosted agents as on my own laptop. |
I had the same time-out issue. I got this working by increasing the WinRM time-out from 5 mins to 10 mins. But, I continued to use image_sku -'2016-datacenter', didn't change this one! |
@harsh-vm when did you try with latest 2016-Datacenter ? currently in 2016-datacenter WinRM disabled so how would you enable it ? |
@sanchetanparmar I am not sure what's the source of your comment of WinRm being disabled in 2016-datacenter as I have used it yesterday and it still works as expected. For reference, this is what I have used under my 'builders' section of packer template:
where, image_sku is 2016-datacenter. Hope this helps.. |
@harsh-vm Created a VM with 2016 and checked WinRM status which is disabled. Here my packer template Packer - 1.4.5 and 1.5.0
just tried above code working for 2016-Datacenter ==> azure-arm: Timeout waiting for WinRM. |
Thank you for posting this I was updating powershell and the winrm is timing out - when I attempt to connect using MAC Remote Desktop I get connection refused. going to revert to 2012 for now just to get some Powershell testing done. |
I am experiencing the exact same issue. It was working before 17th of Nov (that was the last Sunday my pipeline was able to generate an image from the Marketplace 2016-Datacenter) |
@sanchetanparmar and @Dilergore, This seems to be really an issue for most people. I wonder how it worked for me! I had the image built just once as mentioned earlier by increasing the time out and the image was created with no further issues. What I hadn’t tried was creating VM with that image! Probably there might be an issue even there. I have currently moved on to 2019 core server from 2016 as per my work requirements. But, will try to build image with 2016 data centre sku and will feed you guys if it works! If yes, will share the template I tried if not, Microsoft might have to take a look at it! |
@harsh-vm My Packer pipeline looks like, that I use the Marketplace "2016-Datacenter" to create a Bronze image. Then I push it to a Shared Image Gallery, then I use it again in Packer during a second build (second pipeline) to create a Golden Image which I trigger right after the Bronze Pipeline. If I use the Bronze Image from 17th of Nov, that builds okay as a Golden Image. I think there is a new 2016-Datacenter image on the Marketplace since 17th of Nov and in that there was a change not to allow WinRM by default. I tried to look into the versions but it is really hard to determine which is the latest and also hard to determine when they released it (at least for some): I would be grateful if you would have the time to test it again with the latest 2016-Datacenter Marketplace image, as I am currently very busy and I won't have time to pick this up in the next two weeks. p.s.: I am still using Packer 1.4.5. If you can confirm that it is not working for you either I can fast track it with Microsoft. |
Greetings - I have been using the dame Packer code and confirmed today the winrm timeout is still persistent - have tried with winrm ssl set to true and false user and password and with user name and password errors with connection reset by peer, the other ssl method just winrm times cancelling build. windows 2012 works but the power shell levels are ot compatible with code scripts. |
@Mcphejoh Can you check what is the difference between the WinRM / Windows Firewall (default) settings between 2012 and 2016 (or 2019 and 2016)? |
bump - so glad i found this thread. I was stuck on this for ages. I tried 2016-datacenter with 10m timeout and it failed with winRM error. Switched to 2012-R2-Datacenter with 3m timeout and it worked. Must have wasted 3 hours trying with 2016-datacenter. This really needs to get fixed on the documentation, im new to packer and i thought for a long time i had done something wrong. At least make the default server 2012 or 2019 until the issue gets addressed. |
@MicahMcKittrick-MSFT can you re-open this? |
@akingscote This is definitely not a documentation issue. This should work by default. We need to find the root cause, then engage Microsoft to fix (probably) the Marketplace Image for 2016-Datacenter. |
@Dilergore sure thing, but as someone who is trying to learn this stuff, its incredibly frustrating losing hours that on something that has a workaround. This issue has been open since May, so i doubt its getting fixed any time soon. My first reference is this documentation. I understand it should work, but the issue is that it dosent work. |
I don't know anything about why this issue was created. I think back when this was created the problem was with the timeout. When the users started to increase it, it started working. There can be many reasons to that, one could be I can think of is that using a small VM may increase the configuration time for WinRM... If you check the config on the beginning of this issue, it is mentioned that DS2v2 was the VM used. That is 2 cores and 7 Gigs of memory. That combined with Standard HDD... I can imagine that booting up and configuring WinRM takes longer than the default timeout. Anyway I can confirm since early Q3 it was working for me and it stopped working mid-November. |
I have been fighting this issue for a couple of days both with Windows 2016 and Windows 2019 DataCentre images. I read some Microsoft documentation ([https://support.microsoft.com/en-gb/help/4494462/events-not-forwarded-if-the-collector-runs-windows-server](Microsoft support article)) that stated the winrm process runs differently on a Windows 2019 machine when the machine has less than 3.5GB. So I thought I would change vm_size: "Standard_B1ms" which has 2GB ram. |
@SimonDoy using a small VM definitely can be a problem. I am using the following fairly big and fast VMs for my builds:
...And I have the timeout set to 20 mins currently. For both 2016 and 2019 I am using the same specs. 2019 is working, 2016 is not. This problem is unique to the 2016 Marketplace Image / WinRM process. |
Been running into the same problem, switching from Windows 2016 to 2019 solved the issue |
Hello, Use this: With that image version WinRM connects. This is from 12th of November. Update1: Update2: |
I cannot reproduce the issue anymore, so I closed the issue in the hashicorp queue along with the MS case. I tested today multiple times and with all of the images since November and even with the latest one it is working for me. Don't know what changed. |
I still can reproduce the issue, used image
From initial troubleshooting it looks to me a certificate issue, trying to run
And when trying to connect using openssl to retrieve the certificate i'm getting
Trying to re-generate self-signed certificate and reconfigure WinRM causing packer to immediately respond to the connection
and from openssl showcerts i'm getting a correct answer
I see that packer is using azure osProfile.windowsConfiguration.winRM value in the template to configure winRM on the VM, So here i would assume that either there is an issue with creating the certificate from packer side before uploading it to azure vault, or and issue with azure that prevents the VM from configuring winRM correctly using the values from the template, this may needs more troubleshooting.
|
@AliAllomani reopened the other issue, let's continue there. |
Same issue for me for "image_sku": "2016-Datacenter" from few days, but somehow it worked with couple to retry's. my "winrm_timeout": "10m" |
Hi All, I'm a Program Manager for the Azure VM image builder (which uses Packer under the hood), we have seen this too, and we have engaged the Windows PG to investigate. Initially there was a low memory condition which could cause problems, when using Standard D1_v2, this was due to Windows Update, and has been mitigated. However, there is still an issue, Windows PG is investigating, and I will report back when I hear. In the meantime one really kind member reached out with this workaround: danielsollondon/azvmimagebuilder#14 (comment) |
Quick update, I spoke to the Windows team, they have identified an issue with the Windows Server 2016 image (November onwards), that impacts the time to initiate a WinRM connection with Packer, they are still working on this, and will update again mid next week. In the meantime please try increasing the Packer timeout to 30mins, and try a larger VM size. |
I am using |
Update from the Windows team: Windows Server 2016 images since November 2019 can have a post first boot performance issue related to an OS code integrity operation. This issue is more pronounced on small Azure VM sizes (with lower throughput and IO) rendering the VM not immediately usable after first boot. The performance issue is mitigated in February 2020 images and forward. Please use the latest February Windows Server 2016 image once it is available from the Marketplace (ETA 2/17). |
Hey @danielsollondon , I'm also seeing this issue pretty regularly with Win10 images, specifically I'm using offer EDIT: I found hashicorp/packer#8658, I'll continue the discussion over there instead if the switch from |
Still occurring with a Standard_D4s_v3 on latest 2016 image |
Packer azure-arm WinRM Issue
I have included my JSON file below
I have used the existing resource group, VNET and subnet. But it times out while connecting to the VM. It shows the following error
azure-arm output will be in this color.
==> azure-arm: Running builder ...
==> azure-arm: Getting tokens using client secret
azure-arm: Creating Azure Resource Manager (ARM) client ...
==> azure-arm: Using existing resource group ...
==> azure-arm: -> ResourceGroupName : 'test'
==> azure-arm: -> Location : 'uksouth'
==> azure-arm: Validating deployment template ...
==> azure-arm: -> ResourceGroupName : 'test'
==> azure-arm: -> DeploymentName : 'pkrdpn19ib1zopk'
==> azure-arm: Deploying deployment template ...
==> azure-arm: -> ResourceGroupName : 'test'
==> azure-arm: -> DeploymentName : 'kvpkrdpn19ib1zopk'
==> azure-arm: Getting the certificate's URL ...
==> azure-arm: -> Key Vault Name : 'pkrkvn19ib1zopk'
==> azure-arm: -> Key Vault Secret Name : 'packerKeyVaultSecret'
==> azure-arm: -> Certificate URL : 'https://pkrkvn19ib1zopk.vault.azure.net/secrets/packerKeyVaultSecret/37221457a5a3459fa0da7f3745e1c9ee'
==> azure-arm: Setting the certificate's URL ...
==> azure-arm: Validating deployment template ...
==> azure-arm: -> ResourceGroupName : 'test'
==> azure-arm: -> DeploymentName : 'pkrdpn19ib1zopk'
==> azure-arm: Deploying deployment template ...
==> azure-arm: -> ResourceGroupName : 'test'
==> azure-arm: -> DeploymentName : 'pkrdpn19ib1zopk'
==> azure-arm: Getting the VM's IP address ...
==> azure-arm: -> ResourceGroupName : 'test'
==> azure-arm: -> PublicIPAddressName : 'pkripn19ib1zopk'
==> azure-arm: -> NicName : 'pkrnin19ib1zopk'
==> azure-arm: -> Network Connection : 'PrivateEndpoint'
==> azure-arm: -> IP Address : '10.1.0.5'
==> azure-arm: Waiting for WinRM to become available...
==> azure-arm: Timeout waiting for WinRM.
==> azure-arm:
==> azure-arm: The resource group was not created by Packer, deleting individual resources ...
==> azure-arm: -> Deployment: pkrdpn19ib1zopk
==> azure-arm: -> Microsoft.Compute/virtualMachines : 'pkrvmn19ib1zopk'
==> azure-arm: -> Microsoft.Network/networkInterfaces : 'pkrnin19ib1zopk'
==> azure-arm: -> Microsoft.Compute/disks : '/subscriptions/2b87c584-986d-4a3c-9e56-012df416db72/resourceGroups/test/providers/Microsoft.Compute/disks/pkrosn19ib1zopk'
==> azure-arm:
==> azure-arm: The resource group was not created by Packer, deleting individual resources ...
==> azure-arm: Could not retrieve OS Image details
==> azure-arm: -> Deployment: kvpkrdpn19ib1zopk
==> azure-arm: -> Microsoft.KeyVault/vaults/secrets : 'pkrkvn19ib1zopk/packerKeyVaultSecret'
==> azure-arm: -> Microsoft.KeyVault/vaults : 'pkrkvn19ib1zopk'
==> azure-arm: -> : ''
==> azure-arm: Error deleting resource. Please delete manually.
==> azure-arm:
==> azure-arm: Name:
==> azure-arm: Error: Unable to parse path of image
==> azure-arm:
==> azure-arm: The resource group was not created by Packer, not deleting ...
Build 'azure-arm' errored: Timeout waiting for WinRM.
==> Some builds didn't complete successfully and had errors:
--> azure-arm: Timeout waiting for WinRM.
==> Builds finished but no artifacts were created.
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.
The text was updated successfully, but these errors were encountered: