windows-restart fails to detect complete restart during Windows Update #6799
Comments
|
Hey @relip, I believe this was fixed in #6792. Please try a binary from this comment and tell me if it fixes it for you. |
|
Closing as duplicate; if this turns out not to be fixed in 6792, we can reopen :) |
|
did some tests -- here's a log with the new version of Packer and ran with the config is {
"builders": [
{
"type": "googlecompute",
"source_image_family": "windows-2012-r2",
"disk_size": "60",
"machine_type": "n1-highcpu-8",
"communicator": "winrm",
"winrm_username": "packer_user",
"winrm_insecure": true,
"winrm_use_ssl": true,
"metadata": {
"windows-startup-script-cmd": "winrm quickconfig -quiet & net user /add packer_user & net localgroup administrators packer_user /add & winrm set winrm/config/service/auth @{Basic=\"true\"}"
},
"zone": "us-central1-c"
}
],
"provisioners": [
{
"type": "powershell",
"inline": [
"Install-WindowsFeature -Name 'Desktop-Experience'"
]
},
{
"type": "windows-restart"
},
{
"type": "powershell",
"inline": [
"GCESysprep -Noshutdown",
"exit 1"
]
}
]
}Packer should abort since |
|
ran another test with just but with the following provisioners {
"type": "powershell",
"inline": [
"Install-WindowsFeature -Name 'Desktop-Experience'"
]
},
{
"type": "windows-restart"
},
{
"type": "powershell",
"inline": [
"while ($true) { sleep 1; }"
]
}it exits with exit code 1 |
|
It sounds to me like this is an issue with the powershell provisioner, not the windows-restart provisioner. the powershell provisioner isn't waiting for the script to return. |
SwampDragons
added
provisioner/powershell
bug
and removed
provisioner/windows-restart
stage/duplicate
bug
labels
|
Hey @SwampDragons I've been having this issue for several days now.I've followed this issue closely, in addition to this #6733 and this rgl/packer-plugin-windows-update#22. I have tried version 1.2.3, 1.2.5, 1.3.1 and the build with the fix posted in the 6733 with the supposed fix, but none of them have fixed my issue. I am not 100 % sure I have the exact same issue, but it sounds very similar. After installing the OS, the provisioner installs windows updates, then reboots. Then I get a pending reboot detected message and it exits with code 259. I am very happy to help with any testing or debugging this issue as I am eager to get it working. If you need me to attach packerlog.txt or any of my config let me know. |
|
It is related to the restart provisioner and the reason the powershell provisioner got an exit code is that the Windows focibly made the script to exit because rebooting was ongoing as I wrote above 🙂 |
|
@relip is your Powershell provisioner telling the machine to reboot? |
|
I believe you both are running into a known issue -- essentially, our winrm communicator isn't good at handling windows restarts. This is the entire reason we have the windows-restart provisioner; the powershell provisioner can't recover from the disconnection that happens when you restart windows. Many newer users find this frustrating because it means they can't run windows update from within a powershell provisioner; however, if you check out some of the community-based windows projects, you'll see that this can be worked around by running your windows update script as a floppy file, before the provisioning begins. Take a look at https://github.com/joefitzgerald/packer-windows/blob/master/windows_2012_r2.json#L82 for an example of where you'd handle this in your Packer config. |
|
To clarify: Packer will basically always break if you try to initiate a restart from the powershell provisioner, instead of initiating the restart from the windows-restart provisioner. |
|
I do use the windows-restart provisioner. I have also tried doing windows updates during sysprep which didn't work. Exits with code 259. |
|
@chryzsh Can you share your config? The more information I have the easier this will be to debug. |
|
I think I'm missing something in #6799 (comment) -- what combination of provisioners produces the first log excerpt you shared? |
|
@SwampDragons please see the first thread again #6799 (comment) all of this happened because Windows itself reboots two times after installing Windows Feature then reboot the system, and #6799 (comment) is to back up this comment #6799 (comment). To recap:
The |
Yes of course. I changed the extension to .txt to be able to upload them. I attached the powershell script for enabling WinRM that does appear to work just fine. But as you said earlier, I am fairly sure that when windows reboots automatically after installing the first round of updates the WinRM connection drops and isn't properly restored. It's either that or something wrong with how it handles pending reboots. I don't think my build ever gets to the windows-restart provisioner because windows reboots itself immediately after updates. |
|
@relip I'm sorry that this back and forth has gotten complicated and involved repetition. I suspect you're getting frustrated and I'm sorry for that. Text-based communication is often slow and confusing, especially when people end up talking in multiple threads. I really appreciate your recap. I was initially treating this issue as a new bug rather than as a known issue with known workarounds, and that caused me to interact weirdly on it for a while. Sorry for the incoming wall of text; I just want to give as much context as I can. tl;dr: What you're experiencing now is known, and the long-recommended workaround is to do all of your windows updates in sysprep, before winRM is connected, so that the provisioners don't get confused by all of the extraneous reboots that windows updates cause. BUT: I really love your workaround with the registry key and want to implement it Longer explanation: What I was trying to say here:
is that any reboot that isn't initiated and completed by the windows-restart provisioner will cause issues. This includes reboots initiated by Windows itself. I probably shouldn't have used the phrase "powershell" because that seems to have caused confusion. It doesn't matter if you directly call for a restart from a powershell provisioner or if a restart happens as a side effect of another system call (in your case, the installation of the windows feature). You are performing an action that initiates a reboot, separately from the windows-restart provisioner, and it's a known issue that Packer doesn't handle this well. The windows-restart provisioner just performs a single disconnect and reconnect. It sounds to me like you're expecting it to do more than that. You want it to actually wait for several reboots in a row, some of which it didn't initiate. This isn't how the provisioner was designed, so it's not surprising to me that the provisioner isn't working for you. All of this said, I really like your workaround with the windows-restart provisioners, adding a restart-check that actually looks for the registry key It's the first time I've seen someone use this workaround, and it's definitely more flexible than the route of forcing all updates to happen in Sysprep. I'll see if I can add a flag that will tell the restart provisioner to check this key, so that you don't need to run it twice. Would that be an acceptable resolution for you? Again, I'm sorry for the frustration here. Hopefully now we're on the same page and we can move forward. |
SwampDragons
added
enhancement
provisioner/windows-restart
and removed
provisioner/powershell
labels
|
Regarding the windows-update provisioner, as-of 0.6.0, it now also sets the And it seems to do the trick while the machine is rebooting after an update (which sometimes needs more than one reboot). I'm not checking for the mentioned |
|
@rgl thanks! I'll take a look and see if it makes sense for us to incorporate some of your |
|
@relip I wasn't able to get a check against that registry key to solve your use case. Can you share the specialized check command you're sending" |
|
@SwampDragons Sorry I was on vacation since then, saw this comment just now. Having that kind of option would be really appreciated! For the registry key I was looking at different test script at that time -- the key that actually changed was |
|
@relip I hope you had an awesome vacation :) . I'll take a look at whether I can repro using that key. Thanks. |
|
I've made a PR to try to incorporate a registry check into the windows-restart provisioner. I'm struggling at the moment to reproduce this; if anyone wants to give it a try with their real-life use case while I'm getting a reliable repro case set up I'd appreciate the help: linux build: osx build: windows build: |
|
@SwampDragons just tested the build and it doesn't work (sysprep was running during the reboot phase) -- saw the code and looks like it doesn't check |
|
Ah! good catch. I swear I read it 5 times. I'll push an update. |
|
Can you try this one? linux: osx: windows: |
|
Okay, let's try this one more time post- @relip's fix. windows: osx: linux: |
|
@SwampDragons Thanks for the update! Just tested on our actual deployment config and got into infinite loop on checking To check things I RDPed into the node and the key was still there even though the rebooting process is finished. I'm not sure what it is and couldn't find exact information of the registry key/entry but still I can say it is clear that the key does not represent the current state (which requires a reboot or not). I think it's okay to remove the key?
These keys/entries were still there even I rebooted the node multiple after the reboot initiated by Packer. |
|
Definitely. I'll remove, and also add a flag to the provisioner so users can provide their own. |
|
windows: packer.zip osx: linux: |
|
Seems like it works? |
|
🎉 I'll run a few more sanity tests and then merge. |
|
I can confirm that this works with a Windows 10 guest using Virtual Box 5.2 on a Fedora 29 host. Thank you! |
|
I think we can count this as addressed now that #7056 is merged. |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
ghost
locked and limited conversation to collaborators
Packer Version: Packer v1.3.0
OS: Running on CentOS 7, the target host is Windows Sever 2012 R2
Expected Behavior: windows-restart waits until Windows Update gets finished
Actual Behavior: windows-restart consider the machine restarted during the update and lose the connection after the update is done
Config to Reproduce:
{ "builders": [ { "type": "googlecompute", "project_id": "{{user `project_id`}}", "source_image_family": "windows-2012-r2", "disk_size": "60", "machine_type": "n1-highcpu-8", "communicator": "winrm", "winrm_username": "packer_user", "winrm_insecure": true, "winrm_use_ssl": true, "metadata": { "windows-startup-script-cmd": "winrm quickconfig -quiet & net user /add packer_user & net localgroup administrators packer_user /add & winrm set winrm/config/service/auth @{Basic=\"true\"}" }, "zone": "us-central1-c" } ], "provisioners": [ { "type": "powershell", "inline": [ "Install-WindowsFeature -Name 'Desktop-Experience'" ] }, { "type": "windows-restart" }, { "type": "powershell", "inline": ["GCESysPrep -NoShutdown"] } ] }After some investigation it looks like WinRM connection opens during the Windows Update, and the update itself triggers a reboot twice more after reboot triggered by a user. This leads
windows-restartto successfully connect to the server during the update, then move on to the next step and lose the connection on following reboot.Here's Windows event logs during the build:
the second and the third 1074 messages were sent by the system.
To work around this issue, put two
windows-restartprovisioners in a row, one with additionalrestart_check_commandto wait untilHKLM:SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Component Based Servicing\\PackagePendingno longer exist and another one with no extra command line set (actual reboot is already being initiated by Windows Update however but to wait until the final reboot is done).The text was updated successfully, but these errors were encountered: