Skip to content

MSVC: Extend the msvc cache key due to potential issues with the external cache file consistency #4727

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

MSVC: Extend the msvc cache key due to potential issues with the external cache file consistency #4727

wants to merge 6 commits into from

Conversation

jcbrill
Copy link
Contributor

@jcbrill jcbrill commented Jun 4, 2025

Prior to this PR, the msvc cache key consists of the msvc batch file and the msvc batch file arguments. However, the cache contents are based on both the shell environment variable list and the keep variable list. Additions to the variable lists between SCons branches/versions could be ignored when using an existing external msvc cache file.

The msvc environment shell environment variable list was moved from inside a function to global file-level in #4717 which could enable a user to extend the shell variable list. This could yield unexpected behavior when the msvc cache is employed.

The keep list tuple is defined at file-level and the list variable is specified as the default value in the function to process the msvc output. The optional argument is not passed from the vc code. As written, it is effectively impossible to extend the keep list tuple and have the extended value used within the function to process the msvc output. The function was modified to accept None rather than the original definition. Within the function, an explicit check is made if the argument is None.

An encoded string that represents the shell list variable names and the keep list variable names is added to the msvc cache code key. The extended key should be compatible with existing cache files as the old entries will exist on file and in memory but never be retrieved.

The encoded string is not guaranteed to to avoid collisions which could result in the erroneous cache data be used. However, the likelihood of a collision should be remote.

For example, the encoded string for the default shell and keep variable names lists is:

"22:324:;KcO~6_`zMX93%fhL{_tz`ap9;C_%Mm4dPQ:7:68:Nmc+c%zY)=aG}>Ka4cxwFmq&6EzRg{IX(+}"

The encoded string components are separated by : characters. The components for the default shell and keep variable names lists are:

  • 22: The size of the shell environment variable names list.
  • 324: The total number of characters of the shell environment variable names.
  • ;KcO~6_`zMX93%fhL{_tz`ap9;C_%Mm4dPQ: A b85encoded string of the sha224 digest of a normalized string representing the shell variable names.
  • 7: The size of the keep variables names list.
  • 68: The total number of characters of the keep variable names.
  • Nmc+c%zY)=aG}>Ka4cxwFmq&6EzRg{IX(+}: A b85encoded string of the sha224 digest of a normalized string representing the keep variable names.

Contributor Checklist:

  • I have created a new test or updated the unit tests to cover the new/changed functionality.
  • I have updated CHANGES.txt and RELEASE.txt (and read the README.rst).
  • I have updated the appropriate documentation

jcbrill added 3 commits June 4, 2025 11:19
@jcbrill
MSVC: Add additional msvc environment cache key based on the shell an…
bffc47f 
…d keep variable lists.

Changes:
* Rename the VS_VC_VARS and KEEPLIST variables to MSVC_ENVIRON_SHELLVARS and MSVC_ENVIRON_KEEPVARS, respectively.
* Move the MSVC_ENVIRON_KEEPVARS definition to the top of the file.
* Change MSVC_ENVIRON_SHELLVARS from a list to a tuple.
* Make the default value of the keep argument in get_output None instead of MSVC_ENVIRON_KEEPVARS.  Otherwise, the original tuple is the default and a user modification , while discouraged, would effectively be ignored from the vc code path as the argument is not explicitly specified.
* Add an additional component to the msvc cache key based on the list contents of MSVC_ENVIRON_SHELLVARS and MSVC_ENVIRON_KEEPVARS.  Otherwise, any additions to the shell or keep variable lists would not be reflected in existing cache files.  The cached contents using the default variable lists and User extended variable lists may be different given the same batch file name and arguments.
@jcbrill
MSVC: fix msvc cache test by changing change script argument second e…
38d8690 
…lement ([2]) rather than assuming the script argument is last ([-1]).
@jcbrill
MSVC: rename environment vars namedtuple
d9417e0
@jcbrill
Copy link
Contributor Author

jcbrill commented Jun 5, 2025 •
edited
Loading

The ninja iterative_speedup test seems to be failing sporadically in the windows-2022/windows-latest environment.

One change that seems to get around the issue is to increase the number of source files hard-coded in the test script. The current value is 200. Values of 250 and 400 seemed to "work".

Some of these timing comparisons seem to be at the mercy of the performance of the underlying GH runner in addition to what is being tested.

@mwichmann Any thoughts?

@mwichmann
Copy link
Collaborator

The ninja iterative_speedup test seems to be failing sporadically in the windows-2022/windows-latest environment.

@mwichmann Any thoughts?

Yes - I think it's pretty useless as a mandatory test. It's interesting to have as a demo. You're right, it's been sporadically "failing" like, forever. There's another ninja test that fails all the time on Mac and some Linux (not, fortunately, the Ubuntu that's used by the runners). Irritating. @bdbaddog ?

@jcbrill
MSVC: rework the cache file format and data record format.
1874a6f 
Changes:
* Change the cache file format from a list to records to a dictionary containing a header dictionary and a list of records.
* The header dictionary contains version numbers for the cache file format and the msvc record format.  The contents of the cache file can be discarded if a version change is detected when loading an existing file.
* The record format changed from a data dictionary to a two entry dictionary containing the original data dictionary and a list of paths that existed when the entry was installed in the cache.  The verification list is checked when the file is loaded to make sure all paths still exist.
* The msvc cache key now contains four (4) elements: the msvc batch file, the msvc batch file arguments, an encoded string for the imported variables, and an encoded string for the kept variables.
@jcbrill
Copy link
Contributor Author

jcbrill commented Jun 8, 2025

MSVC cache file changes.

Old format (verified VCToolsInstallDir path):

[
  {
    "key": [
      "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Auxiliary\\Build\\vcvars64.bat",
      null
    ],
    "data": {
      "INCLUDE": [
        ...
      ],
      "LIB": [
        ...
      ],
      "LIBPATH": [
        ...
      ],
      "PATH": [
        ...
      ],
      "VSCMD_ARG_app_plat": [
        ...
      ],
      "VCINSTALLDIR": [
        ...
      ],
      "VCToolsInstallDir": [
        "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\"
      ]
    }
  }
]

New format (verifies verify list: cl_path, VCInstallDir, LIB and LIBPATH):

{
  "header": {
    "file_version": 1,
    "record_version": 1
  },
  "records": [
    {
      "key": [
        "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Auxiliary\\Build\\vcvars64.bat",
        null,
        "22:324:;KcO~6_`zMX93%fhL{_tz`ap9;C_%Mm4dPQ",
        "7:68:Nmc+c%zY)=aG}>Ka4cxwFmq&6EzRg{IX(+}"
      ],
      "val": {
        "data": {
          "INCLUDE": [
            ...
          ],
          "LIB": [
            ...
          ],
          "LIBPATH": [
            ...
          ],
          "PATH": [
            ...
          ],
          "VCINSTALLDIR": [
            ...
          ],
          "VCToolsInstallDir": [
            ...
          ],
          "VSCMD_ARG_app_plat": [
            ...
          ]
        },
        "verify": [
          "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\bin\\hostx64\\x64\\cl.exe",
          "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207",
          "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\atlmfc\\lib\\x64",
          "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\lib\\x64",
          "c:\\program files (x86)\\windows kits\\10\\lib\\10.0.26100.0\\ucrt\\x64",
          "c:\\program files (x86)\\windows kits\\10\\lib\\10.0.26100.0\\um\\x64",
          "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\lib\\x86\\store\\references",
          "c:\\program files (x86)\\windows kits\\10\\unionmetadata\\10.0.26100.0",
          "c:\\program files (x86)\\windows kits\\10\\references\\10.0.26100.0",
          "c:\\windows\\microsoft.net\\framework64\\v4.0.30319"
        ]
      }
    }
  ]
}

New format key examples with default key and modified import/keep lists keys:

{
  "header": {
    "file_version": 1,
    "record_version": 1
  },
  "records": [
    {
      "key": [
        "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Auxiliary\\Build\\vcvars64.bat",
        null,
        "22:324:;KcO~6_`zMX93%fhL{_tz`ap9;C_%Mm4dPQ",
        "7:68:Nmc+c%zY)=aG}>Ka4cxwFmq&6EzRg{IX(+}"
      ],
      "val": {
        ...
      }
    },
    {
      "key": [
        "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Auxiliary\\Build\\vcvars64.bat",
        null,
        "33:519:+5sBJmxZ5@8){%O7s_nLc=lUB2&AhQc??Q2",
        "11:126:oL)e(kQ`ESGW+)Xugr=OB9DY)wA%A{6a!~b"
      ],
      "val": {
        ...
      }
    }
  ]
}

@bdbaddog
Copy link
Contributor

bdbaddog commented Jun 9, 2025

Why not use a tuple of the keep variables and the shell environment variable, instead of this encoded bit?

@jcbrill
Copy link
Contributor Author

jcbrill commented Jun 9, 2025 •
edited
Loading

Why not use a tuple of the keep variables and the shell environment variable, instead of this encoded bit?

Purely space savings. The serialized string written to, and read from, the file could be significantly longer.

For example, the encoded value of the default import variable list is 22:324:;KcO~6_`zMX93%fhL{_tz`ap9;C_%Mm4dPQ which represents 22 elements at 324 characters (without string literals and punctuation) would be serialized as a tuple of approximately 380+ characters.

The second example 33:519:+5sBJmxZ5@8){%O7s_nLc=lUB2&AhQc??Q2 is 33 elements at 519 characters which would be serialized to 600+ characters.

If a user never extends the import variable list or the keep variable list, it doesn't really matter.

The cache file is written every time a new key and data are added.

Writing the full serialized tuple can be done. This one doesn't matter that much to me.

@jcbrill
Copy link
Contributor Author

jcbrill commented Jun 9, 2025

The other "issue" with the cache is that reading/writing the cache file which involves serialization/deserialization from json takes place in MSCommon/common.py. The cache key and data structures are maintained in MSCommon/vc.py.

Tuples are serialized/deserialized as a list. There is an explicit conversion of the key to a tuple when reading the cache file. Adding fields to the key that are tuples would require reworking the file read routines to (likely recursively) convert nested lists to tuples in a general fashion. Otherwise, the specific deserialization is based on data structures defined in an external file which is a more difficult maintenance and understanding problem. This general transformation should not be that difficult.

Converting the tuples to a full string would eliminate the nested tuple problem. Which then is a long 100% accurate representation compared to a shorter less than 100% accurate representation.

@bdbaddog
Copy link
Contributor

bdbaddog commented Jun 9, 2025

"records" can be a dict, then everything in key can be made into a tuple and that can be the key to the "records".. I have a strong preference not to do the encoding as if we ever need to debug it it's much simpler to see the full text.
And unless we're adding many KB or MB, I doubt this should meaningfully impact the time to read/write it?

So to double check I'm understanding this, you'd have a different set of cached config values based on :

  1. the vsvars*.bat script run
  2. the keep variables specified (but not their values?)
  3. the env variables imported from the shell where vsvars*.bat has been run (but not their values?)

No more MSVC_SCRIPT_ARGS can be specified? Is that also part of the keys to the cached config?

Also would you see a need to change the record version and not the file version? Seems like a single file version should be sufficient?

@jcbrill
MSVC: remove encoded key strings for shell and keep variable lists an…
27edc77 
…d replace with joined normalized list strings.
@jcbrill
Copy link
Contributor Author

jcbrill commented Jun 9, 2025

@mwichmann Check for misinformation?

Important

The encoded strings were replaced with strings representing the joined (":"), normalized, sorted, and unique list of variable names. The prior encoding was based on the construction of the replaced strings.

Notes:

  • The import variables list retrieves values from os.environ which is case-insensitive.
  • A regex is constructed for the keep variables which is case-insensitive for preserving values from the msvc environment after the batch file is called.

"records" can be a dict, then everything in key can be made into a tuple and that can be the key to the "records"..

The "challenge" is that the implementation of the cache is done in two different source files: SCons.Tool.MSCommon.common and SCons.Tool.MSCommon.vc.

The interface between the two source files is a simple python dictionary. Since it is a dictionary, there is a key and a value for each dictionary item.

The composition of the key and the composition of the value are maintained in SCons.Tool.MSCommon.vc.

The implementation in SCons.Tool.MSCommon.common is responsible for reading an existing cache file, deserializing the json items, and returning the python cache dictionary. The implementation is also responsible for being passed the python cache dictionary, serializing to json, and writing the contents to disk.

At present, the only structural knowledge of the key in SCons.Tool.MSCommon.common is that the serialized json list can be converted to a tuple and the resulting tuple can be used as a dictionary key. As written, this works as long as the key values do not contain members that cannot be used as a python dictionary key (e.g., a list).

If the key deserialized from the file contained tuples, then the conversion routine in common would have to be modfied to recursively convert lists to tuples so the the resulting key is a valid python dictionary key.

If I remember correctly, at some point in time the key was a string and was later changed to a tuple of strings.

The current PR implementation writes the variables list contents as strings. There is no compelling reason to deserialize these strings back into tuples when the cache file is read.

I have a strong preference not to do the encoding as if we ever need to debug it it's much simpler to see the full text. And unless we're adding many KB or MB, I doubt this should meaningfully impact the time to read/write it?

Fair enough.

The current PR was modified to reflect this preference as string of variable name contents.

The string is going to really long as soon as a user does something like this (i.e., locally 2000+ characters):

SCons.Tool.MSCommon.common.MSVC_ENVIRON_SHELLVARS = tuple(os.environ.keys())

So to double check I'm understanding this, you'd have a different set of cached config values based on :

  1. the vsvars*.bat script run
  2. the keep variables specified (but not their values?)
  3. the env variables imported from the shell where vsvars*.bat has been run (but not their values?)

No more MSVC_SCRIPT_ARGS can be specified? Is that also part of the keys to the cached config?

The key is comprised of:

  1. the vcvars*.bat script run [same as before]
  2. the arguments passed to the vcvars*.bat script [same as before]
  3. a string representing the import variable symbols but not their values [new]
  4. a string representing the keep variable symbols but not their values [new]

The script argument string is the second element of the key tuple (i.e., item 2 in the list above). The existing key tuple was extended by two elements (i.e., items 3 and 4 in the list above).

The payload (i.e., cache dictionary value) is a dictionary comprised of:

  1. the original "dkeep" dictionary (e.g. payload['data']: {...})
  2. the new "verify" list (e.g, payload['verify']: [...])

Prior to this PR, the payload was the "dkeep" dictionary which is used internally to extend the user's environment.

This PR changes the cache value payload to be a dictionary containing the original "dkeep" dictionary and a "verify" list of paths that existed when the cache file was written in an attempt to determine if the user's Visual Studio environment changed.

The prior PR used the value of "VCToolsInstallDir" which is effective for determining if the VS default toolset has changed. However, it only really works for VS2017 and later. This PR adds the path to the compiler executable, the "VCToolsInstallDir" path, the "LIB" paths, and the "LIBPATH" paths. Inclusion of one or both of "LIB" and "LIBPATH" will also detect changes in the Windows SDK and other ancillary versioned components.

The cache dictionary value (i.e., "payload") structure changed in the vc code but does not affect the serialization in the common code as it is still a dictionary.

Caution

This PR differentiates between changes in the variable symbols imported and/or preserved.

If import list environment values change value after the cache is built, the next invocation of SCons will use the cached data which was constructed based on the previous values of the import variables.

Ignoring changes in import list environment values has been the behavior of the cache since its inception.

Any future changes to include the imported variable values would be overkill for all variables. Perhaps a list of variable names to include the value makes sense (which would then need an additional key string). Only imported variables that are changeable by a user would need values recorded. For example, a user is not going to change windir, OS, etc. This then introduces a dilemna with other variables like USERPROFILE in environments with multiple user accounts.

For example, the VSCMD_DEBUG environment variable is currently passed to the msvc batch file environment.
IF the cache is built without defining VSCMD_DEBUG, then defining VSCMD_DEBUG will not work as expected without deleting the cache file first.

Also would you see a need to change the record version and not the file version? Seems like a single file version should be sufficient?

There are two independent elements at play:

  • Knowing how to interpret the structure of the json file in SCons.Tool.MSCommon.common. It is possible to change to json structure of the serialized data written to and from the os file system without changing the content of the keys and values. At some point, additional information could be written to the file which has nothing to do with the msvc key/payload values.
  • Knowing how to interpret the content of the cache dictionary keys and values is imlemented in SCons.Tool.MSCommon.vc. It is possible to change the keys (e.g., adding additonal elements) and/or values (e.g., adding additional keys/values in the payload) without affecting the json serialization file format.

In a perfect world, the two mechanisms would be completely disjointed. At present, there is limited knowledge about the key structure in the deserialization routine (i.e., that it is a tuple of elements that can be used as a python dictionary key).

Default Environment Examples - Before and After PR

  • Before this PR:

      [
    {
      "key": [
        "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Auxiliary\\Build\\vcvars64.bat",
        null
      ],
      "data": {
        "INCLUDE": [
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\include",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\ATLMFC\\include",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Auxiliary\\VS\\include",
          "C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.26100.0\\ucrt",
          "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\um",
          "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\shared",
          "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\winrt",
          "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\cppwinrt"
        ],
        "LIB": [
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\ATLMFC\\lib\\x64",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\lib\\x64",
          "C:\\Program Files (x86)\\Windows Kits\\10\\lib\\10.0.26100.0\\ucrt\\x64",
          "C:\\Program Files (x86)\\Windows Kits\\10\\\\lib\\10.0.26100.0\\\\um\\x64"
        ],
        "LIBPATH": [
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\ATLMFC\\lib\\x64",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\lib\\x64",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\lib\\x86\\store\\references",
          "C:\\Program Files (x86)\\Windows Kits\\10\\UnionMetadata\\10.0.26100.0",
          "C:\\Program Files (x86)\\Windows Kits\\10\\References\\10.0.26100.0",
          "C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319"
        ],
        "PATH": [
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\bin\\HostX64\\x64",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\VC\\VCPackages",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\TestWindow",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\TeamFoundation\\Team Explorer",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\MSBuild\\Current\\bin\\Roslyn",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Team Tools\\DiagnosticsHub\\Collector",
          "C:\\Program Files (x86)\\Windows Kits\\10\\bin\\10.0.26100.0\\\\x64",
          "C:\\Program Files (x86)\\Windows Kits\\10\\bin\\\\x64",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\\\MSBuild\\Current\\Bin\\amd64",
          "C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\Tools\\",
          "C:\\WINDOWS\\System32",
          "C:\\WINDOWS\\System32\\Wbem",
          "C:\\Program Files\\PowerShell\\7\\",
          "C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\CMake\\bin",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\Ninja",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\VC\\Linux\\bin\\ConnectionManagerExe",
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\vcpkg"
        ],
        "VSCMD_ARG_app_plat": [
          "Desktop"
        ],
        "VCINSTALLDIR": [
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\"
        ],
        "VCToolsInstallDir": [
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\"
        ]
      }
    }
  ]

  • After this PR (diff format used for highlighting purposes):

    - [
+ {
+   "header": {
+     "file_version": 1,
+     "record_version": 1
+   },
+   "records": [
      {
        "key": [
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Auxiliary\\Build\\vcvars64.bat",
          null,
+         "comspec:msdevdir:os:powershell_telemetry_optout:psdisablemoduleanalysiscachecleanup:psmoduleanalysiscachepath:vcpkg_disable_metrics:vcpkg_root:vs100comntools:vs110comntools:vs120comntools:vs140comntools:vs150comntools:vs160comntools:vs170comntools:vs71comntools:vs80comntools:vs90comntools:vscmd_debug:vscmd_skip_sendtelemetry:vscomntools:windir",
+         "include:lib:libpath:path:vcinstalldir:vctoolsinstalldir:vscmd_arg_app_plat"
        ],
-       "data": {
+       "val": {
+         "data": {
            "INCLUDE": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\include",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\ATLMFC\\include",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Auxiliary\\VS\\include",
              "C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.26100.0\\ucrt",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\um",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\shared",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\winrt",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\cppwinrt"
            ],
            "LIB": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\ATLMFC\\lib\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\lib\\x64",
              "C:\\Program Files (x86)\\Windows Kits\\10\\lib\\10.0.26100.0\\ucrt\\x64",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\lib\\10.0.26100.0\\\\um\\x64"
            ],
            "LIBPATH": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\ATLMFC\\lib\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\lib\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\lib\\x86\\store\\references",
              "C:\\Program Files (x86)\\Windows Kits\\10\\UnionMetadata\\10.0.26100.0",
              "C:\\Program Files (x86)\\Windows Kits\\10\\References\\10.0.26100.0",
              "C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319"
            ],
            "PATH": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\bin\\HostX64\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\VC\\VCPackages",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\TestWindow",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\TeamFoundation\\Team Explorer",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\MSBuild\\Current\\bin\\Roslyn",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Team Tools\\DiagnosticsHub\\Collector",
              "C:\\Program Files (x86)\\Windows Kits\\10\\bin\\10.0.26100.0\\\\x64",
              "C:\\Program Files (x86)\\Windows Kits\\10\\bin\\\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\\\MSBuild\\Current\\Bin\\amd64",
              "C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\Tools\\",
              "C:\\WINDOWS\\System32",
              "C:\\WINDOWS\\System32\\Wbem",
              "C:\\Program Files\\PowerShell\\7\\",
              "C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\CMake\\bin",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\Ninja",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\VC\\Linux\\bin\\ConnectionManagerExe",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\vcpkg"
            ],
            "VCINSTALLDIR": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\"
            ],
            "VCToolsInstallDir": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\"
            ],
            "VSCMD_ARG_app_plat": [
              "Desktop"
            ]
+         },
+         "verify": [
+           "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\bin\\hostx64\\x64\\cl.exe",
+           "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207",
+           "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\atlmfc\\lib\\x64",
+           "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\lib\\x64",
+           "c:\\program files (x86)\\windows kits\\10\\lib\\10.0.26100.0\\ucrt\\x64",
+           "c:\\program files (x86)\\windows kits\\10\\lib\\10.0.26100.0\\um\\x64",
+           "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\lib\\x86\\store\\references",
+           "c:\\program files (x86)\\windows kits\\10\\unionmetadata\\10.0.26100.0",
+           "c:\\program files (x86)\\windows kits\\10\\references\\10.0.26100.0",
+           "c:\\windows\\microsoft.net\\framework64\\v4.0.30319"
+         ]
        }
      }
+   ]
+ }
- ]

Default Environment and Extended Environment - After PR

Diff used for highlighting purposes.

  {
    "header": {
      "file_version": 1,
      "record_version": 1
    },
    "records": [
      {
        "key": [
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Auxiliary\\Build\\vcvars64.bat",
          null,
          "comspec:msdevdir:os:powershell_telemetry_optout:psdisablemoduleanalysiscachecleanup:psmoduleanalysiscachepath:vcpkg_disable_metrics:vcpkg_root:vs100comntools:vs110comntools:vs120comntools:vs140comntools:vs150comntools:vs160comntools:vs170comntools:vs71comntools:vs80comntools:vs90comntools:vscmd_debug:vscmd_skip_sendtelemetry:vscomntools:windir",
          "include:lib:libpath:path:vcinstalldir:vctoolsinstalldir:vscmd_arg_app_plat"
        ],
        "val": {
          "data": {
            "INCLUDE": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\include",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\ATLMFC\\include",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Auxiliary\\VS\\include",
              "C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.26100.0\\ucrt",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\um",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\shared",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\winrt",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\cppwinrt"
            ],
            "LIB": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\ATLMFC\\lib\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\lib\\x64",
              "C:\\Program Files (x86)\\Windows Kits\\10\\lib\\10.0.26100.0\\ucrt\\x64",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\lib\\10.0.26100.0\\\\um\\x64"
            ],
            "LIBPATH": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\ATLMFC\\lib\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\lib\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\lib\\x86\\store\\references",
              "C:\\Program Files (x86)\\Windows Kits\\10\\UnionMetadata\\10.0.26100.0",
              "C:\\Program Files (x86)\\Windows Kits\\10\\References\\10.0.26100.0",
              "C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319"
            ],
            "PATH": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\bin\\HostX64\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\VC\\VCPackages",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\TestWindow",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\TeamFoundation\\Team Explorer",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\MSBuild\\Current\\bin\\Roslyn",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Team Tools\\DiagnosticsHub\\Collector",
              "C:\\Program Files (x86)\\Windows Kits\\10\\bin\\10.0.26100.0\\\\x64",
              "C:\\Program Files (x86)\\Windows Kits\\10\\bin\\\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\\\MSBuild\\Current\\Bin\\amd64",
              "C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\Tools\\",
              "C:\\WINDOWS\\System32",
              "C:\\WINDOWS\\System32\\Wbem",
              "C:\\Program Files\\PowerShell\\7\\",
              "C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\CMake\\bin",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\Ninja",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\VC\\Linux\\bin\\ConnectionManagerExe",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\vcpkg"
            ],
            "VCINSTALLDIR": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\"
            ],
            "VCToolsInstallDir": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\"
            ],
            "VSCMD_ARG_app_plat": [
              "Desktop"
            ]
          },
          "verify": [
            "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\bin\\hostx64\\x64\\cl.exe",
            "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207",
            "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\atlmfc\\lib\\x64",
            "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\lib\\x64",
            "c:\\program files (x86)\\windows kits\\10\\lib\\10.0.26100.0\\ucrt\\x64",
            "c:\\program files (x86)\\windows kits\\10\\lib\\10.0.26100.0\\um\\x64",
            "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\lib\\x86\\store\\references",
            "c:\\program files (x86)\\windows kits\\10\\unionmetadata\\10.0.26100.0",
            "c:\\program files (x86)\\windows kits\\10\\references\\10.0.26100.0",
            "c:\\windows\\microsoft.net\\framework64\\v4.0.30319"
          ]
        }
      },
      {
        "key": [
          "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Auxiliary\\Build\\vcvars64.bat",
          null,
+         "commonprogramfiles:commonprogramfiles(arm):commonprogramfiles(x86):commonprogramw6432:comspec:msdevdir:os:powershell_telemetry_optout:processor_architecture:processor_architew6432:programdata:programfiles:programfiles(arm):programfiles(x86):programw6432:psdisablemoduleanalysiscachecleanup:psmoduleanalysiscachepath:vcpkg_disable_metrics:vcpkg_root:vs100comntools:vs110comntools:vs120comntools:vs140comntools:vs150comntools:vs160comntools:vs170comntools:vs71comntools:vs80comntools:vs90comntools:vscmd_debug:vscmd_skip_sendtelemetry:vscomntools:windir",
+         "include:lib:libpath:path:vcinstalldir:vctoolsinstalldir:vscmd_arg_app_plat:vscmd_arg_host_arch:vscmd_arg_tgt_arch:vscmd_ver:vsinstalldir"
        ],
        "val": {
          "data": {
            "INCLUDE": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\include",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\ATLMFC\\include",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Auxiliary\\VS\\include",
              "C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.26100.0\\ucrt",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\um",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\shared",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\winrt",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\include\\10.0.26100.0\\\\cppwinrt"
            ],
            "LIB": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\ATLMFC\\lib\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\lib\\x64",
              "C:\\Program Files (x86)\\Windows Kits\\10\\lib\\10.0.26100.0\\ucrt\\x64",
              "C:\\Program Files (x86)\\Windows Kits\\10\\\\lib\\10.0.26100.0\\\\um\\x64"
            ],
            "LIBPATH": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\ATLMFC\\lib\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\lib\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\lib\\x86\\store\\references",
              "C:\\Program Files (x86)\\Windows Kits\\10\\UnionMetadata\\10.0.26100.0",
              "C:\\Program Files (x86)\\Windows Kits\\10\\References\\10.0.26100.0",
              "C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319"
            ],
            "PATH": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\bin\\HostX64\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\VC\\VCPackages",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\TestWindow",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\TeamFoundation\\Team Explorer",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\MSBuild\\Current\\bin\\Roslyn",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Team Tools\\DiagnosticsHub\\Collector",
              "C:\\Program Files (x86)\\Windows Kits\\10\\bin\\10.0.26100.0\\\\x64",
              "C:\\Program Files (x86)\\Windows Kits\\10\\bin\\\\x64",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\\\MSBuild\\Current\\Bin\\amd64",
              "C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\Tools\\",
              "C:\\WINDOWS\\System32",
              "C:\\WINDOWS\\System32\\Wbem",
              "C:\\Program Files\\PowerShell\\7\\",
              "C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\CMake\\bin",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\Ninja",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\IDE\\VC\\Linux\\bin\\ConnectionManagerExe",
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\vcpkg"
            ],
            "VCINSTALLDIR": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\"
            ],
            "VCToolsInstallDir": [
              "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.44.35207\\"
            ],
            "VSCMD_ARG_app_plat": [
              "Desktop"
            ],
+           "VSCMD_ARG_HOST_ARCH": [
+             "x64"
+           ],
+           "VSCMD_ARG_TGT_ARCH": [
+             "x64"
+           ],
+           "VSCMD_VER": [
+             "17.14.3"
+           ],
+           "VSINSTALLDIR": [
+             "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\"
+           ]
          },
          "verify": [
            "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\bin\\hostx64\\x64\\cl.exe",
            "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207",
            "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\atlmfc\\lib\\x64",
            "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\lib\\x64",
            "c:\\program files (x86)\\windows kits\\10\\lib\\10.0.26100.0\\ucrt\\x64",
            "c:\\program files (x86)\\windows kits\\10\\lib\\10.0.26100.0\\um\\x64",
            "c:\\program files\\microsoft visual studio\\2022\\community\\vc\\tools\\msvc\\14.44.35207\\lib\\x86\\store\\references",
            "c:\\program files (x86)\\windows kits\\10\\unionmetadata\\10.0.26100.0",
            "c:\\program files (x86)\\windows kits\\10\\references\\10.0.26100.0",
            "c:\\windows\\microsoft.net\\framework64\\v4.0.30319"
          ]
        }
      }
    ]
  }

@jcbrill
Copy link
Contributor Author

jcbrill commented Jun 9, 2025

@mwichmann New failure on Windows.

test\Parallel\failed-build\failed-build.py failed in the "Full Test Suite on Windows" and the appveyor "Visual Studio 2019; Environment: WINPYTHON=Python39".

Based on the error reported, I would not think it is anything in this PR. Any thoughts?

884/1309 ( 67.53%) C:\Python39\python.exe test\Parallel\failed-build\failed-build.py
STDOUT =========================================================================
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
C:\Python39\python.exe myfail.py f3 f3.in 49863
C:\Python39\python.exe mycopy.py f4 f4.in 49863
scons: building terminated because of errors.
STDERR =========================================================================
0a1,55
> Exception in thread Thread-1:
> Traceback (most recent call last):
>   File "C:\Python39\lib\threading.py", line 980, in _bootstrap_inner
>     self.run()
>   File "C:\Python39\lib\threading.py", line 917, in run
>     self._target(*self._args, **self._kwargs)
>   File "C:\Users\appveyor\AppData\Local\Temp\1\scons\testcmd.3952.01ai1vt5\teststate.py", line 93, in server_thread
>     httpd = socketserver.TCPServer(("127.0.0.1", PORT), S)
>   File "C:\Python39\lib\socketserver.py", line 452, in __init__
>     self.server_bind()
>   File "C:\Python39\lib\socketserver.py", line 466, in server_bind
>     self.socket.bind(self.server_address)
> PermissionError: [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions
> Traceback (most recent call last):
>   File "C:\Users\appveyor\AppData\Local\Temp\1\scons\testcmd.3952.01ai1vt5\myfail.py", line 25, in <module>
>     while not check_test_state() and count < WAIT:
>   File "C:\Users\appveyor\AppData\Local\Temp\1\scons\testcmd.3952.01ai1vt5\myfail.py", line 19, in check_test_state
>     conn.request("GET", "/?get_mycopy_started=1")
>   File "C:\Python39\lib\http\client.py", line 1285, in request
>     self._send_request(method, url, body, headers, encode_chunked)
>   File "C:\Python39\lib\http\client.py", line 1331, in _send_request
>     self.endheaders(body, encode_chunked=encode_chunked)
>   File "C:\Python39\lib\http\client.py", line 1280, in endheaders
>     self._send_output(message_body, encode_chunked=encode_chunked)
>   File "C:\Python39\lib\http\client.py", line 1040, in _send_output
>     self.send(msg)
>   File "C:\Python39\lib\http\client.py", line 980, in send
>     self.connect()
>   File "C:\Python39\lib\http\client.py", line 946, in connect
>     self.sock = self._create_connection(
>   File "C:\Python39\lib\socket.py", line 844, in create_connection
>     raise err
>   File "C:\Python39\lib\socket.py", line 832, in create_connection
>     sock.connect(sa)
> ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
> Traceback (most recent call last):
>   File "C:\Users\appveyor\AppData\Local\Temp\1\scons\testcmd.3952.01ai1vt5\mycopy.py", line 14, in <module>
>     conn.request("GET", "/?set_mycopy_started=1")
>   File "C:\Python39\lib\http\client.py", line 1285, in request
>     self._send_request(method, url, body, headers, encode_chunked)
>   File "C:\Python39\lib\http\client.py", line 1331, in _send_request
>     self.endheaders(body, encode_chunked=encode_chunked)
>   File "C:\Python39\lib\http\client.py", line 1280, in endheaders
>     self._send_output(message_body, encode_chunked=encode_chunked)
>   File "C:\Python39\lib\http\client.py", line 1040, in _send_output
>     self.send(msg)
>   File "C:\Python39\lib\http\client.py", line 980, in send
>     self.connect()
>   File "C:\Python39\lib\http\client.py", line 946, in connect
>     self.sock = self._create_connection(
>   File "C:\Python39\lib\socket.py", line 844, in create_connection
>     raise err
>   File "C:\Python39\lib\socket.py", line 832, in create_connection
>     sock.connect(sa)
> ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
1a57
> scons: *** [f4] Error 1
FAILED test of C:\projects\scons\scripts\scons.py
	at line 764 of C:\projects\scons\testing\framework\TestCommon.py (_complete)
	from line 867 of C:\projects\scons\testing\framework\TestCommon.py (run)
	from line 485 of C:\projects\scons\testing\framework\TestSCons.py (run)
	from line 64 of C:\projects\scons\test\Parallel\failed-build\failed-build.py
Test execution time: 2.7 seconds
886/1309 ( 67.69%) C:\Python39\python.exe test\Parallel\ref_count.py
PASSED

@bdbaddog
Copy link
Contributor

bdbaddog commented Jun 9, 2025

@mwichmann New failure on Windows.

test\Parallel\failed-build\failed-build.py failed in the "Full Test Suite on Windows" and the appveyor "Visual Studio 2019; Environment: WINPYTHON=Python39".

Based on the error reported, I would not think it is anything in this PR. Any thoughts?

884/1309 ( 67.53%) C:\Python39\python.exe test\Parallel\failed-build\failed-build.py
STDOUT =========================================================================
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
C:\Python39\python.exe myfail.py f3 f3.in 49863
C:\Python39\python.exe mycopy.py f4 f4.in 49863
scons: building terminated because of errors.
STDERR =========================================================================
0a1,55
> Exception in thread Thread-1:
> Traceback (most recent call last):
>   File "C:\Python39\lib\threading.py", line 980, in _bootstrap_inner
>     self.run()
>   File "C:\Python39\lib\threading.py", line 917, in run
>     self._target(*self._args, **self._kwargs)
>   File "C:\Users\appveyor\AppData\Local\Temp\1\scons\testcmd.3952.01ai1vt5\teststate.py", line 93, in server_thread
>     httpd = socketserver.TCPServer(("127.0.0.1", PORT), S)
>   File "C:\Python39\lib\socketserver.py", line 452, in __init__
>     self.server_bind()
>   File "C:\Python39\lib\socketserver.py", line 466, in server_bind
>     self.socket.bind(self.server_address)
> PermissionError: [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions
> Traceback (most recent call last):
>   File "C:\Users\appveyor\AppData\Local\Temp\1\scons\testcmd.3952.01ai1vt5\myfail.py", line 25, in <module>
>     while not check_test_state() and count < WAIT:
>   File "C:\Users\appveyor\AppData\Local\Temp\1\scons\testcmd.3952.01ai1vt5\myfail.py", line 19, in check_test_state
>     conn.request("GET", "/?get_mycopy_started=1")
>   File "C:\Python39\lib\http\client.py", line 1285, in request
>     self._send_request(method, url, body, headers, encode_chunked)
>   File "C:\Python39\lib\http\client.py", line 1331, in _send_request
>     self.endheaders(body, encode_chunked=encode_chunked)
>   File "C:\Python39\lib\http\client.py", line 1280, in endheaders
>     self._send_output(message_body, encode_chunked=encode_chunked)
>   File "C:\Python39\lib\http\client.py", line 1040, in _send_output
>     self.send(msg)
>   File "C:\Python39\lib\http\client.py", line 980, in send
>     self.connect()
>   File "C:\Python39\lib\http\client.py", line 946, in connect
>     self.sock = self._create_connection(
>   File "C:\Python39\lib\socket.py", line 844, in create_connection
>     raise err
>   File "C:\Python39\lib\socket.py", line 832, in create_connection
>     sock.connect(sa)
> ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
> Traceback (most recent call last):
>   File "C:\Users\appveyor\AppData\Local\Temp\1\scons\testcmd.3952.01ai1vt5\mycopy.py", line 14, in <module>
>     conn.request("GET", "/?set_mycopy_started=1")
>   File "C:\Python39\lib\http\client.py", line 1285, in request
>     self._send_request(method, url, body, headers, encode_chunked)
>   File "C:\Python39\lib\http\client.py", line 1331, in _send_request
>     self.endheaders(body, encode_chunked=encode_chunked)
>   File "C:\Python39\lib\http\client.py", line 1280, in endheaders
>     self._send_output(message_body, encode_chunked=encode_chunked)
>   File "C:\Python39\lib\http\client.py", line 1040, in _send_output
>     self.send(msg)
>   File "C:\Python39\lib\http\client.py", line 980, in send
>     self.connect()
>   File "C:\Python39\lib\http\client.py", line 946, in connect
>     self.sock = self._create_connection(
>   File "C:\Python39\lib\socket.py", line 844, in create_connection
>     raise err
>   File "C:\Python39\lib\socket.py", line 832, in create_connection
>     sock.connect(sa)
> ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
1a57
> scons: *** [f4] Error 1
FAILED test of C:\projects\scons\scripts\scons.py
	at line 764 of C:\projects\scons\testing\framework\TestCommon.py (_complete)
	from line 867 of C:\projects\scons\testing\framework\TestCommon.py (run)
	from line 485 of C:\projects\scons\testing\framework\TestSCons.py (run)
	from line 64 of C:\projects\scons\test\Parallel\failed-build\failed-build.py
Test execution time: 2.7 seconds
886/1309 ( 67.69%) C:\Python39\python.exe test\Parallel\ref_count.py
PASSED

That's presently failing for other PRs, so you can ignore it for this PR. I think it's something with that particular VM.

@mwichmann
Copy link
Collaborator

that's funky, but why would if fail for both GitHub-WIndows and for Appveyor (one image only)....???

@jcbrill
Copy link
Contributor Author

jcbrill commented Jun 9, 2025

Port already in use?

@jcbrill
Copy link
Contributor Author

jcbrill commented Jun 9, 2025

Warning

Misinformation possible.

The SConstruct in the fixture contains:

# SPDX-License-Identifier: MIT
#
# Copyright The SCons Foundation

import os
import random
import sys
import threading

from teststate import server_thread

PORT = random.randint(10000, 60000)

sys.path.append(os.getcwd())

DefaultEnvironment(tools=[])

# this thread will setup a sever for the different tasks to talk to
# and act as a manager of IPC and the different tasks progressing
# the test through different states
x = threading.Thread(target=server_thread, args=(PORT,))
x.daemon = True
x.start()

MyCopy = Builder(action=[[sys.executable, 'mycopy.py', '$TARGET', '$SOURCE', str(PORT)]])
Fail = Builder(action=[[sys.executable, 'myfail.py', '$TARGETS', '$SOURCE', str(PORT)]])
env = Environment(BUILDERS={'MyCopy' : MyCopy, 'Fail' : Fail})
env.Fail(target='f3', source='f3.in')
env.MyCopy(target='f4', source='f4.in')
env.MyCopy(target='f5', source='f5.in')
env.MyCopy(target='f6', source='f6.in')

Maybe a port in the range is already in use or not allowed for some reason.

I can't be sure if this is the actual file being used as it was looked up quickly.

@jcbrill
Copy link
Contributor Author

jcbrill commented Jun 9, 2025

Seems like it would be more robust to have a loop (perhaps with a limit count) that gets a new random port and starts the server thread. Exit loop on no exception raised or limit count exceeded. Continue in loop if ConnectionRefusedError. Re-raise any other exception type.

Again, could be really wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kont395 kont395 kont395 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jcbrill @mwichmann @bdbaddog @kont395