[BUG] SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC on alternating cloud.present state IDs during state.apply on vSphere 7.0

[ggiesen]

Description
When using the cloud.present state with the vSphere provider, alternating states fail with SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC. If you add another state ID (corresponding to another VM), then the states that fail change but always alternate (every other state ID fails):

----------
          ID: cloud_salt-proxy02
    Function: cloud.present
        Name: salt-proxy02.example.com
      Result: True
     Comment: Already present instance salt-proxy02.example.com
     Started: 00:01:44.490419
    Duration: 1723.694 ms
     Changes:   
----------
          ID: cloud_salt01
    Function: cloud.present
        Name: salt01.example.com
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3.6/site-packages/salt/state.py", line 2180, in call
                  *cdata["args"], **cdata["kwargs"]
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/states/cloud.py", line 104, in present
                  if __salt__["cloud.has_instance"](name=name, provider=None):
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/modules/cloud.py", line 148, in has_instance
                  data = get_instance(name, provider)
                File "/usr/lib/python3.6/site-packages/salt/modules/cloud.py", line 174, in get_instance
                  data = action(fun="show_instance", names=[name], provider=provider)
                File "/usr/lib/python3.6/site-packages/salt/modules/cloud.py", line 259, in action
                  info = client.action(fun, cloudmap, names, provider, instance, kwargs)
                File "/usr/lib/python3.6/site-packages/salt/cloud/__init__.py", line 486, in action
                  return mapper.do_action(names, kwargs)
                File "/usr/lib/python3.6/site-packages/salt/cloud/__init__.py", line 1482, in do_action
                  vm_name, call="action"
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/cloud/clouds/vmware.py", line 2201, in show_instance
                  _get_si(), vim.VirtualMachine, vm_properties
                File "/usr/lib/python3.6/site-packages/salt/cloud/clouds/vmware.py", line 272, in _get_si
                  url, username, password, protocol=protocol, port=port, verify_ssl=verify_ssl
                File "/usr/lib/python3.6/site-packages/salt/utils/vmware.py", line 478, in get_service_instance
                  service_instance.CurrentTime()
                File "/usr/local/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 706, in <lambda>
                  self.f(*(self.args + (obj,) + args), **kwargs)
                File "/usr/local/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 512, in _InvokeMethod
                  return self._stub.InvokeMethod(self, info, args)
                File "/usr/local/lib/python3.6/site-packages/pyVmomi/SoapAdapter.py", line 1351, in InvokeMethod
                  resp = conn.getresponse()
                File "/usr/lib64/python3.6/http/client.py", line 1361, in getresponse
                  response.begin()
                File "/usr/lib64/python3.6/http/client.py", line 311, in begin
                  version, status, reason = self._read_status()
                File "/usr/lib64/python3.6/http/client.py", line 272, in _read_status
                  line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
                File "/usr/lib64/python3.6/socket.py", line 586, in readinto
                  return self._sock.recv_into(b)
                File "/usr/lib64/python3.6/ssl.py", line 971, in recv_into
                  return self.read(nbytes, buffer)
                File "/usr/lib64/python3.6/ssl.py", line 833, in read
                  return self._sslobj.read(len, buffer)
                File "/usr/lib64/python3.6/ssl.py", line 590, in read
                  v = self._sslobj.read(len, buffer)
              ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2354)
     Started: 00:01:46.214575
    Duration: 1151.133 ms
     Changes:   
----------
          ID: cloud_salt02
    Function: cloud.present
        Name: salt02.example.com
      Result: True
     Comment: Already present instance salt02.example.com
     Started: 00:01:47.366804
    Duration: 1484.715 ms
     Changes:   
----------
          ID: cloud_salt-quorum01
    Function: cloud.present
        Name: salt-quorum01.example.com
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3.6/site-packages/salt/state.py", line 2180, in call
                  *cdata["args"], **cdata["kwargs"]
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/states/cloud.py", line 104, in present
                  if __salt__["cloud.has_instance"](name=name, provider=None):
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/modules/cloud.py", line 148, in has_instance
                  data = get_instance(name, provider)
                File "/usr/lib/python3.6/site-packages/salt/modules/cloud.py", line 174, in get_instance
                  data = action(fun="show_instance", names=[name], provider=provider)
                File "/usr/lib/python3.6/site-packages/salt/modules/cloud.py", line 259, in action
                  info = client.action(fun, cloudmap, names, provider, instance, kwargs)
                File "/usr/lib/python3.6/site-packages/salt/cloud/__init__.py", line 486, in action
                  return mapper.do_action(names, kwargs)
                File "/usr/lib/python3.6/site-packages/salt/cloud/__init__.py", line 1482, in do_action
                  vm_name, call="action"
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/cloud/clouds/vmware.py", line 2201, in show_instance
                  _get_si(), vim.VirtualMachine, vm_properties
                File "/usr/lib/python3.6/site-packages/salt/cloud/clouds/vmware.py", line 272, in _get_si
                  url, username, password, protocol=protocol, port=port, verify_ssl=verify_ssl
                File "/usr/lib/python3.6/site-packages/salt/utils/vmware.py", line 478, in get_service_instance
                  service_instance.CurrentTime()
                File "/usr/local/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 706, in <lambda>
                  self.f(*(self.args + (obj,) + args), **kwargs)
                File "/usr/local/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 512, in _InvokeMethod
                  return self._stub.InvokeMethod(self, info, args)
                File "/usr/local/lib/python3.6/site-packages/pyVmomi/SoapAdapter.py", line 1351, in InvokeMethod
                  resp = conn.getresponse()
                File "/usr/lib64/python3.6/http/client.py", line 1361, in getresponse
                  response.begin()
                File "/usr/lib64/python3.6/http/client.py", line 311, in begin
                  version, status, reason = self._read_status()
                File "/usr/lib64/python3.6/http/client.py", line 272, in _read_status
                  line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
                File "/usr/lib64/python3.6/socket.py", line 586, in readinto
                  return self._sock.recv_into(b)
                File "/usr/lib64/python3.6/ssl.py", line 971, in recv_into
                  return self.read(nbytes, buffer)
                File "/usr/lib64/python3.6/ssl.py", line 833, in read
                  return self._sslobj.read(len, buffer)
                File "/usr/lib64/python3.6/ssl.py", line 590, in read
                  v = self._sslobj.read(len, buffer)
              ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2354)
     Started: 00:01:48.851880
    Duration: 890.016 ms
     Changes:   
----------
          ID: cloud_fog01
    Function: cloud.present
        Name: fog01.example.com
      Result: True
     Comment: Already present instance fog01.example.com
     Started: 00:01:49.742685
    Duration: 1372.79 ms
     Changes:   
----------
          ID: cloud_proxy01
    Function: cloud.present
        Name: proxy01.example.com
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3.6/site-packages/salt/state.py", line 2180, in call
                  *cdata["args"], **cdata["kwargs"]
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/states/cloud.py", line 104, in present
                  if __salt__["cloud.has_instance"](name=name, provider=None):
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/modules/cloud.py", line 148, in has_instance
                  data = get_instance(name, provider)
                File "/usr/lib/python3.6/site-packages/salt/modules/cloud.py", line 174, in get_instance
                  data = action(fun="show_instance", names=[name], provider=provider)
                File "/usr/lib/python3.6/site-packages/salt/modules/cloud.py", line 259, in action
                  info = client.action(fun, cloudmap, names, provider, instance, kwargs)
                File "/usr/lib/python3.6/site-packages/salt/cloud/__init__.py", line 486, in action
                  return mapper.do_action(names, kwargs)
                File "/usr/lib/python3.6/site-packages/salt/cloud/__init__.py", line 1482, in do_action
                  vm_name, call="action"
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/cloud/clouds/vmware.py", line 2201, in show_instance
                  _get_si(), vim.VirtualMachine, vm_properties
                File "/usr/lib/python3.6/site-packages/salt/cloud/clouds/vmware.py", line 272, in _get_si
                  url, username, password, protocol=protocol, port=port, verify_ssl=verify_ssl
                File "/usr/lib/python3.6/site-packages/salt/utils/vmware.py", line 478, in get_service_instance
                  service_instance.CurrentTime()
                File "/usr/local/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 706, in <lambda>
                  self.f(*(self.args + (obj,) + args), **kwargs)
                File "/usr/local/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 512, in _InvokeMethod
                  return self._stub.InvokeMethod(self, info, args)
                File "/usr/local/lib/python3.6/site-packages/pyVmomi/SoapAdapter.py", line 1351, in InvokeMethod
                  resp = conn.getresponse()
                File "/usr/lib64/python3.6/http/client.py", line 1361, in getresponse
                  response.begin()
                File "/usr/lib64/python3.6/http/client.py", line 311, in begin
                  version, status, reason = self._read_status()
                File "/usr/lib64/python3.6/http/client.py", line 272, in _read_status
                  line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
                File "/usr/lib64/python3.6/socket.py", line 586, in readinto
                  return self._sock.recv_into(b)
                File "/usr/lib64/python3.6/ssl.py", line 971, in recv_into
                  return self.read(nbytes, buffer)
                File "/usr/lib64/python3.6/ssl.py", line 833, in read
                  return self._sslobj.read(len, buffer)
                File "/usr/lib64/python3.6/ssl.py", line 590, in read
                  v = self._sslobj.read(len, buffer)
              ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2354)
     Started: 00:01:51.115883
    Duration: 856.015 ms
     Changes:   
----------
          ID: cloud_vault01
    Function: cloud.present
        Name: vault01.example.com
      Result: True
     Comment: Already present instance vault01.example.com
     Started: 00:01:51.972497
    Duration: 1385.258 ms
     Changes:   
----------
          ID: cloud_nautobot01
    Function: cloud.present
        Name: nautobot01.example.com
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3.6/site-packages/salt/state.py", line 2180, in call
                  *cdata["args"], **cdata["kwargs"]
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/states/cloud.py", line 104, in present
                  if __salt__["cloud.has_instance"](name=name, provider=None):
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/modules/cloud.py", line 148, in has_instance
                  data = get_instance(name, provider)
                File "/usr/lib/python3.6/site-packages/salt/modules/cloud.py", line 174, in get_instance
                  data = action(fun="show_instance", names=[name], provider=provider)
                File "/usr/lib/python3.6/site-packages/salt/modules/cloud.py", line 259, in action
                  info = client.action(fun, cloudmap, names, provider, instance, kwargs)
                File "/usr/lib/python3.6/site-packages/salt/cloud/__init__.py", line 486, in action
                  return mapper.do_action(names, kwargs)
                File "/usr/lib/python3.6/site-packages/salt/cloud/__init__.py", line 1482, in do_action
                  vm_name, call="action"
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/cloud/clouds/vmware.py", line 2201, in show_instance
                  _get_si(), vim.VirtualMachine, vm_properties
                File "/usr/lib/python3.6/site-packages/salt/cloud/clouds/vmware.py", line 272, in _get_si
                  url, username, password, protocol=protocol, port=port, verify_ssl=verify_ssl
                File "/usr/lib/python3.6/site-packages/salt/utils/vmware.py", line 478, in get_service_instance
                  service_instance.CurrentTime()
                File "/usr/local/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 706, in <lambda>
                  self.f(*(self.args + (obj,) + args), **kwargs)
                File "/usr/local/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 512, in _InvokeMethod
                  return self._stub.InvokeMethod(self, info, args)
                File "/usr/local/lib/python3.6/site-packages/pyVmomi/SoapAdapter.py", line 1351, in InvokeMethod
                  resp = conn.getresponse()
                File "/usr/lib64/python3.6/http/client.py", line 1361, in getresponse
                  response.begin()
                File "/usr/lib64/python3.6/http/client.py", line 311, in begin
                  version, status, reason = self._read_status()
                File "/usr/lib64/python3.6/http/client.py", line 272, in _read_status
                  line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
                File "/usr/lib64/python3.6/socket.py", line 586, in readinto
                  return self._sock.recv_into(b)
                File "/usr/lib64/python3.6/ssl.py", line 971, in recv_into
                  return self.read(nbytes, buffer)
                File "/usr/lib64/python3.6/ssl.py", line 833, in read
                  return self._sslobj.read(len, buffer)
                File "/usr/lib64/python3.6/ssl.py", line 590, in read
                  v = self._sslobj.read(len, buffer)
              ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2354)
     Started: 00:01:53.358182
    Duration: 1055.439 ms
     Changes:   

Setup
Here's my state file:

# Create salt02-proxy02 VM
cloud_salt-proxy02:
  cloud.present:
    - name: "salt-proxy02.example.com"
    - cloud_provider: vcenter-dc3
    - clonefrom: AlmaLinux_8.5_x64_Template_cloud-init-1.0.2
    - num_cpus: 2
    - memory: 4GB
    - devices:
        disk:
          Hard disk 1:
            size: 40
            controller: SCSI controller 0
            thin_provision: True
        network:
          Network adapter 1:
            name: AUTOMTN
            adapter_type: vmxnet3
            switch_type: distributed
            ip: 198.51.100.24
            gateway: [198.51.100.17]
            subnet_mask: 255.255.255.240
    - domain: example.com
    - dns_servers:
        - 192.0.2.2
        - 192.0.2.3
    - datastore: DATASTORE
    - folder: Internal
    - datacenter: DC
    - resourcepool: Resources
    - host: 203.0.113.68
    - template: False
    - power_on: True
    - extra_config:
# This is cloud-init customization data
{% set metadata = 
'local-hostname: salt-proxy02.example.com
instance-id: salt-proxy02.example.com
network:
  version: 2
  ethernets:
    ens192:
      addresses:
        - 198.51.100.24/28
        - 2001:db8:1516::15/64
      gateway4: 198.51.100.17
      gateway6: 2001:db8:1516::1
      nameservers:
        search: [example.com]
        addresses: [192.0.2.2, 192.0.2.3]
' %}
        guestinfo.metadata: {{ metadata | base64_encode }}
        guestinfo.metadata.encoding: base64
        mem.hotadd: 'yes'
    - annotation: Created by Salt-Cloud
    - deploy: True
    - tmp_dir: /root/tmp/.saltcloud/
    - minion:
        id: salt-proxy02
        master:
          - salt01.example.com
          - salt02.example.com
    - script_args: -j '{"master":["salt01.example.com", "salt02.example.com"],"id":"salt-proxy02","use_superseded":["module.run"],"startup_states":"highstate"}' stable 3004
    - customization: False
    - ssh_username: {{ pillar['vmware']['linux']['default_username'] }}
    - password: {{ pillar['vmware']['linux']['default_password'] }}
    - delvol_on_destroy: True

# Create salt01 VM
cloud_salt01:
  cloud.present:
    - name: "salt01.example.com"
    - cloud_provider: vcenter-dc3
    - clonefrom: AlmaLinux_8.5_x64_Template_cloud-init-1.0.2
    - num_cpus: 2
    - memory: 4GB
    - devices:
        disk:
          Hard disk 1:
            size: 40
            controller: SCSI controller 0
            thin_provision: True
        network:
          Network adapter 1:
            name: AUTOMTN
            adapter_type: vmxnet3
            switch_type: distributed
            ip: 198.51.100.29
            gateway: [198.51.100.17]
            subnet_mask: 255.255.255.240
          Network adapter 2:
            name: TOR3-SALT-NFS
            adapter_type: vmxnet3
            switch_type: distributed
            ip: 198.51.100.100
            subnet_mask: 255.255.255.240
    - domain: example.com
    - dns_servers:
        - 192.0.2.2
        - 192.0.2.3
    - datastore: DATASTORE
    - folder: Internal
    - datacenter: DC
    - resourcepool: Resources
    - host: 203.0.113.68
    - template: False
    - power_on: True
    - extra_config:
# This is cloud-init customization data
{% set metadata = 
'local-hostname: salt01.example.com
instance-id: salt01.example.com
network:
  version: 2
  ethernets:
    ens192:
      addresses:
        - 198.51.100.20/28
        - 2001:db8:1516::11/64
      gateway4: 198.51.100.17
      gateway6: 2001:db8:1516::1
      nameservers:
        search: [example.com]
        addresses: [192.0.2.2, 192.0.2.3]
    ens224:
      addresses:
        - 198.51.100.100/28
' %}
        guestinfo.metadata: {{ metadata | base64_encode }}
        guestinfo.metadata.encoding: base64
        mem.hotadd: 'yes'
    - annotation: Created by Salt-Cloud
    - deploy: True
    - tmp_dir: /root/tmp/.saltcloud/
    - minion:
        id: salt01
        master:
          - salt01.example.com
          - salt02.example.com
    - script_args: -j '{"master":["salt01.example.com", "salt02.example.com"],"id":"salt01","use_superseded":["module.run"],"startup_states":"highstate"}' stable 3004
    - customization: False
    - ssh_username: {{ pillar['vmware']['linux']['default_username'] }}
    - password: {{ pillar['vmware']['linux']['default_password'] }}
    - delvol_on_destroy: True

# Create salt02 VM
cloud_salt02:
  cloud.present:
    - name: "salt02.example.com"
    - cloud_provider: vcenter-dc3
    - clonefrom: AlmaLinux_8.5_x64_Template_cloud-init-1.0.2
    - num_cpus: 2
    - memory: 4GB
    - devices:
        disk:
          Hard disk 1:
            size: 40
            controller: SCSI controller 0
            thin_provision: True
        network:
          Network adapter 1:
            name: AUTOMTN
            adapter_type: vmxnet3
            switch_type: distributed
            ip: 198.51.100.21
            gateway: [198.51.100.17]
            subnet_mask: 255.255.255.240
          Network adapter 2:
            name: TOR3-SALT-NFS
            adapter_type: vmxnet3
            switch_type: distributed
            ip: 198.51.100.101
            subnet_mask: 255.255.255.240
    - domain: example.com
    - dns_servers:
        - 192.0.2.2
        - 192.0.2.3
    - datastore: DATASTORE
    - folder: Internal
    - datacenter: DC
    - resourcepool: Resources
    - host: 203.0.113.68
    - template: False
    - power_on: True
    - extra_config:
# This is cloud-init customization data
{% set metadata = 
'local-hostname: salt02.example.com
instance-id: salt02.example.com
network:
  version: 2
  ethernets:
    ens192:
      addresses:
        - 198.51.100.21/28
        - 2001:db8:1516::12/64
      gateway4: 198.51.100.17
      gateway6: 2001:db8:1516::1
      nameservers:
        search: [example.com]
        addresses: [192.0.2.2, 192.0.2.3]
    ens224:
      addresses:
        - 198.51.100.101/28
' %}
        guestinfo.metadata: {{ metadata | base64_encode }}
        guestinfo.metadata.encoding: base64
        mem.hotadd: 'yes'
    - annotation: Created by Salt-Cloud
    - deploy: True
    - tmp_dir: /root/tmp/.saltcloud/
    - minion:
        id: salt02
        master:
          - salt01.example.com
          - salt02.example.com
    - script_args: -j '{"master":["salt01.example.com", "salt02.example.com"],"id":"salt02","use_superseded":["module.run"],"startup_states":"highstate"}' stable 3004
    - customization: False
    - ssh_username: {{ pillar['vmware']['linux']['default_username'] }}
    - password: {{ pillar['vmware']['linux']['default_password'] }}
    - delvol_on_destroy: True

# Create salt-quorum01 VM
cloud_salt-quorum01:
  cloud.present:
    - name: "salt-quorum01.example.com"
    - cloud_provider: vcenter-dc3
    - clonefrom: AlmaLinux_8.5_x64_Template_cloud-init-1.0.2
    - num_cpus: 1
    - memory: 2GB
    - devices:
        disk:
          Hard disk 1:
            size: 30
            controller: SCSI controller 0
            thin_provision: True
        network:
          Network adapter 1:
            name: AUTOMTN
            adapter_type: vmxnet3
            switch_type: distributed
            ip: 198.51.100.22
            gateway: [198.51.100.17]
            subnet_mask: 255.255.255.240
    - domain: example.com
    - dns_servers:
        - 192.0.2.2
        - 192.0.2.3
    - datastore: DATASTORE
    - folder: Internal
    - datacenter: DC
    - resourcepool: Resources
    - host: 203.0.113.68
    - template: False
    - power_on: True
    - extra_config:
        # This is cloud-init customization data
{% set metadata = 
'local-hostname: salt-quorum01.example.com
instance-id: salt-quorum01.example.com
network:
  version: 2
  ethernets:
    ens192:
      addresses:
        - 198.51.100.22/28
        - 2001:db8:1516::13/64
      gateway4: 198.51.100.17
      gateway6: 2001:db8:1516::1
      nameservers:
        search: [example.com]
        addresses: [192.0.2.2, 192.0.2.3]
' %}
        guestinfo.metadata: {{ metadata | base64_encode }}
        guestinfo.metadata.encoding: base64
        mem.hotadd: 'yes'
    - annotation: Created by Salt-Cloud
    - deploy: True
    - tmp_dir: /root/tmp/.saltcloud/
    - minion:
        id: salt-quorum01
        master:
          - salt01.example.com
          - salt02.example.com
    - script_args: -j '{"master":["salt01.example.com", "salt02.example.com"],"id":"salt-quorum01","use_superseded":["module.run"],"startup_states":"highstate"}' stable 3004
    - customization: False
    - ssh_username: {{ pillar['vmware']['linux']['default_username'] }}
    - password: {{ pillar['vmware']['linux']['default_password'] }}
    - delvol_on_destroy: True

# Create fog01 VM
cloud_fog01:
  cloud.present:
    - name: "fog01.example.com"
    - cloud_provider: vcenter-dc3
    - clonefrom: AlmaLinux_8.5_x64_Template_cloud-init-1.0.2
    - num_cpus: 2
    - memory: 4GB
    - devices:
        disk:
          Hard disk 1:
            size: 100
            controller: SCSI controller 0
            thin_provision: True
        network:
          Network adapter 1:
            name: DEVIMG
            adapter_type: vmxnet3
            switch_type: distributed
            ip: 198.51.100.84
            gateway: [198.51.100.81]
            subnet_mask: 255.255.255.248
    - domain: example.com
    - dns_servers:
        - 192.0.2.2
        - 192.0.2.3
    - datastore: DATASTORE
    - folder: Internal
    - datacenter: DC
    - resourcepool: Resources
    - host: 203.0.113.68
    - template: False
    - power_on: True
    - extra_config:
        # This is cloud-init customization data
{% set metadata = 
'local-hostname: fog01.example.com
instance-id: fog01.example.com
network:
  version: 2
  ethernets:
    ens192:
      addresses:
        - 198.51.100.84/29
        - 2001:db8:1525::11/64
      gateway4: 198.51.100.81
      gateway6: 2001:db8:1525::1
      nameservers:
        search: [example.com]
        addresses: [192.0.2.2, 192.0.2.3]
' %}
        guestinfo.metadata: {{ metadata | base64_encode }}
        guestinfo.metadata.encoding: base64
        mem.hotadd: 'yes'
    - annotation: Created by Salt-Cloud
    - deploy: True
    - tmp_dir: /root/tmp/.saltcloud/
    - minion:
        id: fog01
        master:
          - salt01.example.com
          - salt02.example.com
    - script_args: -j '{"master":["salt01.example.com", "salt02.example.com"],"id":"fog01","use_superseded":["module.run"],"startup_states":"highstate"}' stable 3004
    - customization: False
    - ssh_username: {{ pillar['vmware']['linux']['default_username'] }}
    - password: {{ pillar['vmware']['linux']['default_password'] }}
    - delvol_on_destroy: True

# Create proxy01 VM
cloud_proxy01:
  cloud.present:
    - name: "proxy01.example.com"
    - cloud_provider: vcenter-dc3
    - clonefrom: AlmaLinux_8.5_x64_Template_cloud-init-1.0.2
    - num_cpus: 1
    - memory: 2GB
    - devices:
        disk:
          Hard disk 1:
            size: 40
            controller: SCSI controller 0
            thin_provision: True
        network:
          Network adapter 1:
            name: PROXY
            adapter_type: vmxnet3
            switch_type: distributed
            ip: 198.51.100.68
            gateway: [198.51.100.65]
            subnet_mask: 255.255.255.248
    - domain: example.com
    - dns_servers:
        - 192.0.2.2
        - 192.0.2.3
    - datastore: DATASTORE
    - folder: Internal
    - datacenter: DC
    - resourcepool: Resources
    - host: 203.0.113.68
    - template: False
    - power_on: True
    - extra_config:
        # This is cloud-init customization data
{% set metadata = 
'local-hostname: proxy01.example.com
instance-id: proxy01.example.com
network:
  version: 2
  ethernets:
    ens192:
      addresses:
        - 198.51.100.68/29
        - 2001:db8:1519::11/64
      gateway4: 198.51.100.65
      gateway6: 2001:db8:1519::1
      nameservers:
        search: [example.com]
        addresses: [192.0.2.2, 192.0.2.3]
' %}
        guestinfo.metadata: {{ metadata | base64_encode }}
        guestinfo.metadata.encoding: base64
        mem.hotadd: 'yes'
    - annotation: Created by Salt-Cloud
    - deploy: True
    - tmp_dir: /root/tmp/.saltcloud/
    - minion:
        id: proxy01
        master:
          - salt01.example.com
          - salt02.example.com
    - script_args: -j '{"master":["salt01.example.com", "salt02.example.com"],"id":"proxy01","use_superseded":["module.run"],"startup_states":"highstate"}' stable 3004
    - customization: False
    - ssh_username: {{ pillar['vmware']['linux']['default_username'] }}
    - password: {{ pillar['vmware']['linux']['default_password'] }}
    - delvol_on_destroy: True

# Create vault01 VM
cloud_vault01:
  cloud.present:
    - name: "vault01.example.com"
    - cloud_provider: vcenter-dc3
    - clonefrom: AlmaLinux_8.5_x64_Template_cloud-init-1.0.2
    - num_cpus: 2
    - memory: 8GB
    - devices:
        disk:
          Hard disk 1:
            size: 50
            controller: SCSI controller 0
            thin_provision: True
        network:
          Network adapter 1:
            name: PAM
            adapter_type: vmxnet3
            switch_type: distributed
            ip: 198.51.100.52
            gateway: [198.51.100.49]
            subnet_mask: 255.255.255.240
    - domain: example.com
    - dns_servers:
        - 192.0.2.2
        - 192.0.2.3
    - datastore: DATASTORE
    - folder: Internal
    - datacenter: DC
    - resourcepool: Resources
    - host: 203.0.113.68
    - template: False
    - power_on: True
    - extra_config:
        # This is cloud-init customization data
{% set metadata = 
'local-hostname: vault01.example.com
instance-id: vault01.example.com
network:
  version: 2
  ethernets:
    ens192:
      addresses:
        - 198.51.100.52/28
        - 2001:db8:1518::11/64
      gateway4: 198.51.100.49
      gateway6: 2001:db8:1518::1
      nameservers:
        search: [example.com]
        addresses: [192.0.2.2, 192.0.2.3]
' %}
        guestinfo.metadata: {{ metadata | base64_encode }}
        guestinfo.metadata.encoding: base64
        mem.hotadd: 'yes'
    - annotation: Created by Salt-Cloud
    - deploy: True
    - tmp_dir: /root/tmp/.saltcloud/
    - minion:
        id: vault01
        master:
          - salt01.example.com
          - salt02.example.com
    - script_args: -j '{"master":["salt01.example.com", "salt02.example.com"],"id":"vault01","use_superseded":["module.run"],"startup_states":"highstate"}' stable 3004
    - customization: False
    - ssh_username: {{ pillar['vmware']['linux']['default_username'] }}
    - password: {{ pillar['vmware']['linux']['default_password'] }}
    - delvol_on_destroy: True

# Create nautobot01 VM
cloud_nautobot01:
  cloud.present:
    - name: "nautobot01.example.com"
    - cloud_provider: vcenter-dc3
    - clonefrom: AlmaLinux_8.5_x64_Template_cloud-init-1.0.2
    - num_cpus: 2
    - memory: 2GB
    - devices:
        disk:
          Hard disk 1:
            size: 40
            controller: SCSI controller 0
            thin_provision: True
        network:
          Network adapter 1:
            name: DCIM
            adapter_type: vmxnet3
            switch_type: distributed
            ip: 198.51.100.36
            gateway: [198.51.100.33]
            subnet_mask: 255.255.255.240
    - domain: example.com
    - dns_servers:
        - 192.0.2.2
        - 192.0.2.3
    - datastore: DATASTORE
    - folder: Internal
    - datacenter: DC
    - resourcepool: Resources
    - host: 203.0.113.68
    - template: False
    - power_on: True
    - extra_config:
        # This is cloud-init customization data
{% set metadata = 
'local-hostname: nautobot01.example.com
instance-id: nautobot01.example.com
network:
  version: 2
  ethernets:
    ens192:
      addresses:
        - 198.51.100.36/28
        - 2001:db8:1517::11/64
      gateway4: 198.51.100.33
      gateway6: 2001:db8:1517::1
      nameservers:
        search: [example.com]
        addresses: [192.0.2.2, 192.0.2.3]
' %}
        guestinfo.metadata: {{ metadata | base64_encode }}
        guestinfo.metadata.encoding: base64
        mem.hotadd: 'yes'
    - annotation: Created by Salt-Cloud
    - deploy: True
    - tmp_dir: /root/tmp/.saltcloud/
    - minion:
        id: nautobot01
        master:
          - salt01.example.com
          - salt02.example.com
    - script_args: -j '{"master":["salt01.example.com", "salt02.example.com"],"id":"nautobot01","use_superseded":["module.run"],"startup_states":"highstate"}' stable 3004
    - customization: False
    - ssh_username: {{ pillar['vmware']['linux']['default_username'] }}
    - password: {{ pillar['vmware']['linux']['default_password'] }}
    - delvol_on_destroy: True

/etc/salt/cloud.providers.d/vmware.conf:

vcenter-dc3:
  driver: vmware
  user: <removed>
  password: <removed>
  url: 'vcenter-dc.example.com'

Steps to Reproduce the behavior
Run state.apply on the master minion with the above configuration files

Expected behavior
All state IDs should either report Already present or provision the VM

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3004.1

Dependency Versions:
          cffi: 1.11.5
      cherrypy: unknown
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: 0.26.8
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: 0.26.4
        Python: 3.6.8 (default, Nov 17 2021, 16:10:06)
  python-gnupg: Not Installed
        PyYAML: 3.12
         PyZMQ: 19.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: almalinux 8.5 Arctic Sphynx
        locale: UTF-8
       machine: x86_64
       release: 4.18.0-348.20.1.el8_5.x86_64
        system: Linux
       version: AlmaLinux 8.5 Arctic Sphynx

Additional context
Master has pyvmomi 7.0.3 installed

[ggiesen]

One other thing of note that I didn't catch right away, is that for the VMs that fail, Salt is pre-generating new keys, knocking existing minions offline. This essentially makes this state function unusable as it actively breaks things if used.

[garethgreenaway]

Sent to the team working on the VMware Salt extension.

[a-wildman]

While attempting to work around this issue by implementing a custom Runner (where I catch ssl.SSLError from a dummy call to cloud.* after every actual call to cloud.* I wish to make), I started getting a BrokenPipeError. That led me to #58869, in which I found the workaround listed here also provided a workaround for this issue as well.