Upgrade to Vagrant 2.3.6 breaks working machines and plugins

[mloskot]

I upgraded two machines on Windows 11 to from Vagrant 2.3.4 to 2.3.6 and machines that used to work fine simply stopped working.

First, selection of pugins required by https://github.com/kubernetes-sigs/sig-windows-dev-tools/ i.e. vagrant-reload vagrant-vbguest winrm winrm-elevated stopped working and upgrading due to #13156, so I had to apply the workaround:

C:\> vagrant plugin install --plugin-clean-sources --plugin-source https://rubygems.org vagrant-reload vagrant-vbguest winrm winrm-elevated
Installing the 'vagrant-reload' plugin. This can take a few minutes...
Fetching vagrant-reload-0.0.1.gem
Installed the plugin 'vagrant-reload (0.0.1)'!
Installing the 'vagrant-vbguest' plugin. This can take a few minutes...
Fetching micromachine-3.0.0.gem
Fetching vagrant-vbguest-0.31.0.gem
Installed the plugin 'vagrant-vbguest (0.31.0)'!
Installing the 'winrm' plugin. This can take a few minutes...
Installed the plugin 'winrm (2.3.6)'!
Installing the 'winrm-elevated' plugin. This can take a few minutes...
Installed the plugin 'winrm-elevated (1.2.3)'!

Second, well-known Windows Server boxes that used to work, stopped booting:

D:\_vagrant\gusztavvargadr> vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'gusztavvargadr/windows-server-2022-standard-core'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'gusztavvargadr/windows-server-2022-standard-core' version '2102.0.2305' is up to date...
==> default: Setting the name of the VM: gusztavvargadr_default_1684620698880_37533
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 3389 (guest) => 53389 (host) (adapter 1)
    default: 5985 (guest) => 55985 (host) (adapter 1)
    default: 5986 (guest) => 55986 (host) (adapter 1)
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: WinRM address: 127.0.0.1:55985
    default: WinRM username: vagrant
    default: WinRM execution_time_limit: PT2H
    default: WinRM transport: negotiate
An error occurred executing a remote WinRM command.

Shell: Cmd
Command: hostname
Message: Digest initialization failed: initialization error

After downgrading to 2.3.4, everything is back to order:

D:\_vagrant\gusztavvargadr> vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'gusztavvargadr/windows-server-2022-standard-core'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'gusztavvargadr/windows-server-2022-standard-core' version '2102.0.2305' is up to date...
==> default: Setting the name of the VM: gusztavvargadr_default_1684624279796_4773
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 3389 (guest) => 53389 (host) (adapter 1)
    default: 5985 (guest) => 55985 (host) (adapter 1)
    default: 5986 (guest) => 55986 (host) (adapter 1)
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: WinRM address: 127.0.0.1:55985
    default: WinRM username: vagrant
    default: WinRM execution_time_limit: PT2H
    default: WinRM transport: negotiate
==> default: Machine booted and ready!
[default] GuestAdditions 7.0.8 running --- OK.
==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...
    default: /vagrant => D:/_vagrant/gusztavvargadr

[chrisroberts]

Hi there,

Would you please provide a gist of the debug output from running vagrant plugin install vagrant-reload --debug.

Thanks!

[mloskot]

Hi @chrisroberts, I assume you mean to collect the debug output on machine upgraded to Vagrant 2.3.6, correct?

[chrisroberts]

Yes, that is correct

[StephenPooley]

Possibly related to #13177

[mloskot]

@StephenPooley Your report looks like exactly the problem. I did not have a chance to collect the debug output as it needs upgrading to broken environment, while I need a working environment ATM :)

[StephenPooley]

@mloskot Agree. Upgrading to a broken environment does sound like a nugatory activitiy and this problem is really easy to reproduce. I feel that the information to reproduce and the debug output is included in #13177 In summary install vagrant v2.3.6 on 64bit Windows and try to work with a plugin (e.g. vmware_desktop).
The problem also does appear to be the same as previously discussed in hashicorp/vagrant-installers#270 and #13156.
It does appear that both the problem and the fix are well understood.
Over to you @chrisroberts

[chrisroberts]

@mloskot Inspecting the 64 bit Windows package, the cacert file is unexpectedly missing. I'm looking through the build process to determine why it wasn't included as it was properly installed in dev builds prior to release.

Downloading the cacert.pem file and dropping it in C:\HashiCorp\Vagrant\embedded\cacert.pem or setting the SSL_CERT_FILE environment variable to the download path should provide an immediate fix.

The winrm issues were introduced from openssl deprecating engines and replacing them with providers (along with marking MD4 as legacy and not loading it by default). The Ruby openssl library provides no method for loading in the way it was previously available with engines. #13178 provides a way around this so legacy and default providers get loaded.

[chrisroberts]

This development build includes both the missing cacert file and the update for handling openssl 3 discussed in this issue.

[mloskot]

@chrisroberts Thank you for your updates, much appreciated. I will try to pick it up later this week or next week.