Tweak the MSVC environment vars cache #4111

mwichmann · 2022-03-08T19:38:05Z

Now performs a sanity check: if the retrieved tools path does not exist, consider the entry invalid so it will be recomputed.
The dictionary key, which is the name of a batch file, is computed a bit differently: the dashes are left off if there are no arguments.
The cachefile is changed to have a .json suffix, for better recognition on Windows systems.

Since this feature is still marked experimental, and these aren't exactly earthshaking changes (basically, will recompute the cache data once), they should be okay without any kind of a transitional cycle.

Questions for reviewers:

This looks at cache_data["VCToolsInstallDir"], which according to the comments isn't used by much, but it seemed to be the only place that had a distinct path value one could check to see if it still exists. Open to alternative suggestions.
should the cache file be named without a leading dot?
should the cache file by default be created in the user's home directory, or would somewhere project-relative be a better choice?

Signed-off-by: Mats Wichmann mats@linux.com

Contributor Checklist:

I have created a new test or updated the unit tests to cover the new/changed functionality.
I have updated CHANGES.txt (and read the README.rst)
I have updated the appropriate documentation

- Now performs a sanity check: if the retrieved tools path does not exist, consider the entry invalid so it will be recomputed. - The dictionary key, which is the name of a batch file, is computed a bit differently: the dashes are left off if there are no arguments. - The cachefile is changed to have a .json suffix, for better recognition on Windows systems. Signed-off-by: Mats Wichmann <mats@linux.com>

bdbaddog · 2022-03-08T22:12:33Z

Might be worth adding a file version to the json file?
So if you make notably changes to the contents it will know to rebuild?

SCons/Tool/MSCommon/vc.py

bdbaddog · 2022-03-10T00:05:55Z

Probably worth noting in CHANGES and RELEASE that the first time you use the updated scons it will rebuild the cache file?

Sharpened up the descriptions of the change in PR 4111. Changed the opening of the cache file to use pathlib functions. Signed-off-by: Mats Wichmann <mats@linux.com>

jcbrill · 2022-03-11T17:57:45Z

This looks at cache_data["VCToolsInstallDir"], which according to the comments isn't used by much, but it seemed to be the only place that had a distinct path value one could check to see if it still exists. Open to alternative suggestions.

I believe that VCToolsInstallDir is only populated for VS2017 and later (i.e., VS2017, VS2019, VS2022).

For VS2015 and earlier, the cache would always be cleared and then populated via a call to run the script. Unless I'm way off base here (which is entirely plausible), effectively there would be no environment caching for 2015 and earlier as there would be an IndexError due to the empty list for VCToolsInstallDir.

Building with the v140 tools via VS2022 (e.g., with script args "--vcvars_ver=14.0") results in a VCToolsInstallDir for VS2015. However, entries for both VS2015 and VS2022 are added to the PATH variable.

Checking the PATH variable is not guaranteed to work all of the time as some versions of MSVC add path entries to the list that do not exist.

mwichmann · 2022-03-11T18:02:03Z

Checking the PATH variable is not guaranteed to work all of the time as some versions of MSVC add path entries to the list that do not exist.

Thanks for the comments. Indeed, that's why l looked for something else than digging around in PATH. The seeming natural thing would be for the "key" in the dictionary to be version-aware, but since this lookup is done based on the batch script name, it isn't. Would it make sense to leave the retrieved value alone if the VCToolsInstallDir came back as an empty list?

jcbrill · 2022-03-11T18:10:15Z

Would it make sense to leave the retrieved value alone if the VCToolsInstallDir came back as an empty list?

I'm not sure there is a bulletproof (or even reliable) solution.

Just thinking out loud: a synthetic key could be added to the cache entry prior to writing that contains a (possibly combined) list of path elements (e.g., PATH, INCLUDE, etc) that are known to exist when the cache entry is written. Upon retrieval the paths in the synthetic entry could be verified. If one or more paths don't exist anymore, recalculate the cache entry. If all verify successfully, remove the synthetic key before returning to the cached entry.

jcbrill · 2022-03-11T18:11:57Z

... or the synthetic key and values could just be part of the environment.

jcbrill · 2022-03-11T18:43:34Z

There is another "fly in the ointment" so to speak: current SCons uses the default/latest toolset which become part of the path. For example:

Cache key: "C:\\Software\\MSVS-2022-143-Com\\VC\\Auxiliary\\Build\\vcvars64.bat--"
Partial path: "C:\\Software\\MSVS-2022-143-Com\\VC\\Tools\\MSVC\\14.31.31103\\bin\\HostX64\\x64"

If a MSVS update were performed that installed a newer toolset while keeping the old toolset, the stale cache entry would still be valid but would not be what a user expects (i.e., the latest installed toolset). If the toolset version were always passed to the msvc batch files, this particular problem effectively disappears.

mwichmann · 2022-03-11T18:47:38Z

Yes, I was worrying about that too. It's perhaps a little artificial, but I actually have a test setup that keeps the last three toolsets so yes, triggering on "directory missing" is not strictly accurate. This issue wasn't introduced by this patch, since that would have been true before - we would have retrieved the entry matching the key, it would be found, would silently work - and would not be the latest version if the previous one were kept.

jcbrill · 2022-03-11T19:01:13Z

SCons could always pass the toolset version for the default version:

"c:\\software\\msvs-2022-143-com\\VC\\Auxiliary\\Build\\vcvarsall.bat" "amd64 --vcvars_ver=14.31.31103"

In this way it becomes part of the cache key and is consistent. Although now there is another problem...

mwichmann · 2022-03-12T17:22:18Z

Ouch, something in this change caused an overly long windows path to be constructed in the environment the CI system sees (worked for me). Don't bother reviewing change 65bdbc9 - I'll fix and update when I get a chance.

Adapt previous change so if cache entry tools dir is empty, don't invalidate the entry, it's an old compiler that doesn't add this info (we always add it, if it didn't come from the batch file it will be an empty list). We don't want to force that case to recalculate, or the value of the cache is lost for those compilers. Signed-off-by: Mats Wichmann <mats@linux.com>

mwichmann · 2022-03-12T17:43:58Z

Okay, I accidentally committed an experimental change in a different file that hadn't intended to. Repushed, let's see if that's better.

jcbrill · 2022-03-16T20:57:10Z

What follows is a suggestion, some observations, and a detailed example of cache file manipulation and when it could be useful.

The current separator between the script name and the arguments is a sequence of two hyphen characters (--):

extra = f"--{args}" if args else ""

The hyphen is legal in both filenames and as batch file arguments. Currently, a user-defined batch file is checked for existence which automatically rules out any characters that are not allowed in windows file names. Modern msvc batch files accept most parameters with a prefix of -, --, or /.

Perhaps using a character that is not legal in a filename would make it easier outside of SCons to process the json cache file and split the cache keys into their component batch file name and arguments.

For example, using the pipe character (|) instead:

extra = f"|{args}" if args else ""

This should guarantee that the cache key could be partitioned (e.g., cache_key.partition('|)) on the pipe character.

Some notes to file away for future reference:

It is likely impossible to guarantee the cache file is consistent with the installed software. There are installation options that do not change the toolset version and do not require additional command-line options (e.g., ATL/MFC installation).
The most common form of invalidating the cache file would be installing a new toolset. As mentioned above, passing the default toolset version as an argument alleviates this problem as it becomes part of the cache key. If a new toolset is installed, the "new default" toolset would not exist in the cache and would be added accordingly with the appropriate toolset version in the cache key.
The windows batch file name is the primary component of the cache key. This may come from the user or from the msvc detection logic. As windows path and file names are not case sensitive, it could be useful to normalize the absolute path of the batch file:
```
script = os.path.normcase(os.path.abspath(os.path.normpath(script)))
```
For example, the following all refer to the same batch file:
```
MSVC_USE_SCRIPT = "..\\SCons\TEST-ARG\\_SCons-SDK60-SetEnv.cmd",
MSVC_USE_SCRIPT = "..\\scons\\test-arg\\_scons-sdk60-setenv.cmd",
MSVC_USE_SCRIPT = "S:\\SCONS\\TEST-ARG\\_SCONS-SDK60-SETENV.cmd",
```
While they are all the same file, these definitions would result in the execution of the batch file three times instead of once.
The same thing can be done with the script arguments as most modern msvc arguments are not case sensitive. However there is at least one exception that is only lower case. Really old SDK (2003 R2 and earlier) arguments are case sensitive and upper case. In general, SDK arguments could be upper case and msvc arguments could be lower case. However, it is probably unwise to apply upper/lower to the argument string. A better approach would target known arguments. It is possible although a bit more involved.
If the cache file grows, it may be better to write the cache once at exit when "dirty" rather than every time a new entry is added.

A useful technique for a build server would be to "prime the msvc cache" via a batch file and an SConstruct file that calls SCons to regenerate the cache file with all msvc batch file invocations that will be used for production builds. This could be run immediately after msvc updates.

The following example consists of deleting the existing cache, generating a new cache, re-writing the cache due to an msvc bug for MSVC 14.1 arm64 targets, and then testing the cache by re-running the same sconstruct that generated the cache.

The MSVC 14.1 arm64 targets fail with the most recent update to SDK 10.0.22000.0 due to an instrinsics issue. The workaround is to use SDK version 10.0.20348.0 or earlier. The example demonstrates how to pre-populate and then process the scons cache file to re-write the 14.1 arm64 targets to use SDK 10.0.20348.0 for default builds. I am not saying this is a good idea, just an outside-the-box example of possible utility.

SConstruct:

import os
import sys

rebuild_cache = True if os.environ.get('$REBUILDING_SCONS_CACHE_MSVC_CONFIG') in ('1', 'true', 'True') else False

build_n = 0
env_list = []

def make_environment(**kwargs):
    global build_n
    global env_list
    build_n += 1
    build = '_build{:03d}'.format(build_n)
    print('Build:', build, repr(kwargs), file=sys.stderr)
    VariantDir(build, '.', duplicate=0)
    env=Environment(**kwargs)
    env.Program(build + '/hello', [build + '/hello.c'])
    env_list.append(env)
    return env

make_environment(MSVC_VERSION='14.3', TARGET_ARCH='x86'  )
make_environment(MSVC_VERSION='14.3', TARGET_ARCH='amd64')
make_environment(MSVC_VERSION='14.3', TARGET_ARCH='arm'  )
make_environment(MSVC_VERSION='14.3', TARGET_ARCH='arm64')

make_environment(MSVC_VERSION='14.2', TARGET_ARCH='x86'  )
make_environment(MSVC_VERSION='14.2', TARGET_ARCH='amd64')
make_environment(MSVC_VERSION='14.2', TARGET_ARCH='arm'  )
make_environment(MSVC_VERSION='14.2', TARGET_ARCH='arm64')

# 2017 14.1 ARM64 https://developercommunity.visualstudio.com/t/libucrtlibstreamobj-:-error-LNK2001:/1544787
#                 intrinsics issue with SDK 10.0.22000.0: use SDK 10.0.20348.0 or earlier

make_environment(MSVC_VERSION='14.1', TARGET_ARCH='x86'  )
make_environment(MSVC_VERSION='14.1', TARGET_ARCH='amd64')
make_environment(MSVC_VERSION='14.1', TARGET_ARCH='arm'  )
if not rebuild_cache:
    make_environment(MSVC_VERSION='14.1', TARGET_ARCH='arm64') # problem with ucrt/sdk (see above)
else:
    make_environment(MSVC_VERSION='14.1', TARGET_ARCH='arm64', MSVC_USE_SCRIPT=r'C:\Software\MSVS-2017-141-Com\VC\Auxiliary\Build\vcvarsamd64_arm64.bat', MSVC_USE_SCRIPT_ARGS='10.0.20348.0')

make_environment(MSVC_VERSION='14.1Exp', TARGET_ARCH='x86'  )
make_environment(MSVC_VERSION='14.1Exp', TARGET_ARCH='amd64')
make_environment(MSVC_VERSION='14.1Exp', TARGET_ARCH='arm'  )
if not rebuild_cache:
    make_environment(MSVC_VERSION='14.1Exp', TARGET_ARCH='arm64') # problem with ucrt/sdk (see above)
else:
    make_environment(MSVC_VERSION='14.1Exp', TARGET_ARCH='arm64', MSVC_USE_SCRIPT=r'C:\Software\MSVS-2017-141-Exp\VC\Auxiliary\Build\vcvarsx86_arm64.bat', MSVC_USE_SCRIPT_ARGS='10.0.20348.0')

make_environment(MSVC_VERSION='14.0', TARGET_ARCH='x86'  )
make_environment(MSVC_VERSION='14.0', TARGET_ARCH='amd64')
make_environment(MSVC_VERSION='14.0', TARGET_ARCH='arm'  )

make_environment(MSVC_VERSION='12.0', TARGET_ARCH='x86'  )
make_environment(MSVC_VERSION='12.0', TARGET_ARCH='amd64')
#make_environment(MSVC_VERSION='12.0', TARGET_ARCH='arm'  ) # compiling desktop apps for arm not supported

make_environment(MSVC_VERSION='11.0Exp', TARGET_ARCH='x86'  )
make_environment(MSVC_VERSION='11.0Exp', TARGET_ARCH='amd64')

make_environment(MSVC_VERSION='10.0Exp', TARGET_ARCH='x86'  )
make_environment(MSVC_VERSION='10.0Exp', TARGET_ARCH='amd64')

make_environment(MSVC_VERSION='9.0', TARGET_ARCH='x86'  )
make_environment(MSVC_VERSION='9.0', TARGET_ARCH='amd64')

make_environment(MSVC_VERSION='8.0', TARGET_ARCH='x86', MSVC_USE_SCRIPT=r'C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin\SetEnv.Cmd', MSVC_USE_SCRIPT_ARGS='/Release /x86 /2003')
make_environment(MSVC_VERSION='8.0', TARGET_ARCH='x86'  )
make_environment(MSVC_VERSION='8.0', TARGET_ARCH='amd64')

make_environment(MSVC_VERSION='7.1', TARGET_ARCH='x86')
make_environment(MSVC_VERSION='7.0', TARGET_ARCH='x86')
make_environment(MSVC_VERSION='6.0', TARGET_ARCH='x86')

Re-write cache script:

import os
import sys
import json

def rewrite_cache(cache_filename):
    rebuild_cache = False
    with open(cache_filename, 'r') as f:
        beg_cache = json.load(f)
    end_cache = {}
    for key, val in beg_cache.items():
        parts = key.partition('--')
        if parts[-1] == '10.0.20348.0':
            rebuild_cache = True
            old_key = key
            key = ''.join(parts[0:2])
            print('Rewriting cache entry:\n    {}\n    {}'.format(old_key, key), file=sys.stderr)
        end_cache[key] = val
    if rebuild_cache:
        with open(cache_filename, 'w') as f:
            json.dump(end_cache, f, indent=2)
    return rebuild_cache

if __name__ == "__main__":
    cache_filename = os.path.expandvars(r'%USERPROFILE%\.scons_msvc_cache')
    if os.path.exists(cache_filename):
        rewritten = rewrite_cache(cache_filename)

Batch file (using a work-in-progress copy of Scons):

@if "%PYEXE%"=="" @call E:\Python\python-embed.bat 3.6.8 64
@echo Enabling scons cache ... 1>&2
@set "SCONS_CACHE_MSVC_CONFIG=1"
@echo Removing .scons_msvc_cache ... 1>&2
@if exist %USERPROFILE%\.scons_msvc_cache @del %USERPROFILE%\.scons_msvc_cache 1>nul 2>nul
@echo Removing build folders ... 1>&2
@call :clean-dir
@echo Rebuilding .scons_msvc_cache ... 1>&2
@set "$REBUILDING_SCONS_CACHE_MSVC_CONFIG=1"
@echo Running scons ...
@%PYEXE% ..\scons-msvc-usesettings\scripts\scons.py %*
@echo Removing build folders ... 1>&2
@call :clean-dir
@set "$REBUILDING_SCONS_CACHE_MSVC_CONFIG="
@echo Rewriting cache ... 1>&2
@%PYEXE% .\rewrite_cache.py
@echo Running scons with modified cache ...
@%PYEXE% ..\scons-msvc-usesettings\scripts\scons.py %*
@echo Removing build folders ... 1>&2
@call :clean-dir
@echo Disabling scons cache ... 1>&2
@set "SCONS_CACHE_MSVC_CONFIG="
@echo Finished 1>&2
@goto :eof

:clean-dir
    @for /d %%G in (".\_build*") do @(
        @del /f /s /q "%%~G" 1>nul 2>nul
        @rmdir /q "%%~G" 1>nul 2>nul
    ) 
    @if exist .sconsign.dblite @del .sconsign.dblite 1>nul 2>nul
    @goto :eof

Stderr output log:

Enabling scons cache ... 
Removing .scons_msvc_cache ... 
Removing build folders ... 
Rebuilding .scons_msvc_cache ... 
Build: _build001 {'MSVC_VERSION': '14.3', 'TARGET_ARCH': 'x86'}
Build: _build002 {'MSVC_VERSION': '14.3', 'TARGET_ARCH': 'amd64'}
Build: _build003 {'MSVC_VERSION': '14.3', 'TARGET_ARCH': 'arm'}
Build: _build004 {'MSVC_VERSION': '14.3', 'TARGET_ARCH': 'arm64'}
Build: _build005 {'MSVC_VERSION': '14.2', 'TARGET_ARCH': 'x86'}
Build: _build006 {'MSVC_VERSION': '14.2', 'TARGET_ARCH': 'amd64'}
Build: _build007 {'MSVC_VERSION': '14.2', 'TARGET_ARCH': 'arm'}
Build: _build008 {'MSVC_VERSION': '14.2', 'TARGET_ARCH': 'arm64'}
Build: _build009 {'MSVC_VERSION': '14.1', 'TARGET_ARCH': 'x86'}
Build: _build010 {'MSVC_VERSION': '14.1', 'TARGET_ARCH': 'amd64'}
Build: _build011 {'MSVC_VERSION': '14.1', 'TARGET_ARCH': 'arm'}
Build: _build012 {'MSVC_VERSION': '14.1', 'TARGET_ARCH': 'arm64', 'MSVC_USE_SCRIPT': 'C:\\Software\\MSVS-2017-141-Com\\VC\\Auxiliary\\Build\\vcvarsamd64_arm64.bat', 'MSVC_USE_SCRIPT_ARGS': '10.0.20348.0'}
Build: _build013 {'MSVC_VERSION': '14.1Exp', 'TARGET_ARCH': 'x86'}
Build: _build014 {'MSVC_VERSION': '14.1Exp', 'TARGET_ARCH': 'amd64'}
Build: _build015 {'MSVC_VERSION': '14.1Exp', 'TARGET_ARCH': 'arm'}
Build: _build016 {'MSVC_VERSION': '14.1Exp', 'TARGET_ARCH': 'arm64', 'MSVC_USE_SCRIPT': 'C:\\Software\\MSVS-2017-141-Exp\\VC\\Auxiliary\\Build\\vcvarsx86_arm64.bat', 'MSVC_USE_SCRIPT_ARGS': '10.0.20348.0'}
Build: _build017 {'MSVC_VERSION': '14.0', 'TARGET_ARCH': 'x86'}
Build: _build018 {'MSVC_VERSION': '14.0', 'TARGET_ARCH': 'amd64'}
Build: _build019 {'MSVC_VERSION': '14.0', 'TARGET_ARCH': 'arm'}
Build: _build020 {'MSVC_VERSION': '12.0', 'TARGET_ARCH': 'x86'}
Build: _build021 {'MSVC_VERSION': '12.0', 'TARGET_ARCH': 'amd64'}
Build: _build022 {'MSVC_VERSION': '11.0Exp', 'TARGET_ARCH': 'x86'}
Build: _build023 {'MSVC_VERSION': '11.0Exp', 'TARGET_ARCH': 'amd64'}
Build: _build024 {'MSVC_VERSION': '10.0Exp', 'TARGET_ARCH': 'x86'}
Build: _build025 {'MSVC_VERSION': '10.0Exp', 'TARGET_ARCH': 'amd64'}
Build: _build026 {'MSVC_VERSION': '9.0', 'TARGET_ARCH': 'x86'}
Build: _build027 {'MSVC_VERSION': '9.0', 'TARGET_ARCH': 'amd64'}
Build: _build028 {'MSVC_VERSION': '8.0', 'TARGET_ARCH': 'x86', 'MSVC_USE_SCRIPT': 'C:\\Program Files\\Microsoft SDKs\\Windows\\v6.0\\Bin\\SetEnv.Cmd', 'MSVC_USE_SCRIPT_ARGS': '/Release /x86 /2003'}
Build: _build029 {'MSVC_VERSION': '8.0', 'TARGET_ARCH': 'x86'}
Build: _build030 {'MSVC_VERSION': '8.0', 'TARGET_ARCH': 'amd64'}
Build: _build031 {'MSVC_VERSION': '7.1', 'TARGET_ARCH': 'x86'}
Build: _build032 {'MSVC_VERSION': '7.0', 'TARGET_ARCH': 'x86'}
Build: _build033 {'MSVC_VERSION': '6.0', 'TARGET_ARCH': 'x86'}
Removing build folders ... 
Rewriting cache ... 
Rewriting cache entry:
    C:\Software\MSVS-2017-141-Com\VC\Auxiliary\Build\vcvarsamd64_arm64.bat--10.0.20348.0
    C:\Software\MSVS-2017-141-Com\VC\Auxiliary\Build\vcvarsamd64_arm64.bat--
Rewriting cache entry:
    C:\Software\MSVS-2017-141-Exp\VC\Auxiliary\Build\vcvarsx86_arm64.bat--10.0.20348.0
    C:\Software\MSVS-2017-141-Exp\VC\Auxiliary\Build\vcvarsx86_arm64.bat--
Build: _build001 {'MSVC_VERSION': '14.3', 'TARGET_ARCH': 'x86'}
Build: _build002 {'MSVC_VERSION': '14.3', 'TARGET_ARCH': 'amd64'}
Build: _build003 {'MSVC_VERSION': '14.3', 'TARGET_ARCH': 'arm'}
Build: _build004 {'MSVC_VERSION': '14.3', 'TARGET_ARCH': 'arm64'}
Build: _build005 {'MSVC_VERSION': '14.2', 'TARGET_ARCH': 'x86'}
Build: _build006 {'MSVC_VERSION': '14.2', 'TARGET_ARCH': 'amd64'}
Build: _build007 {'MSVC_VERSION': '14.2', 'TARGET_ARCH': 'arm'}
Build: _build008 {'MSVC_VERSION': '14.2', 'TARGET_ARCH': 'arm64'}
Build: _build009 {'MSVC_VERSION': '14.1', 'TARGET_ARCH': 'x86'}
Build: _build010 {'MSVC_VERSION': '14.1', 'TARGET_ARCH': 'amd64'}
Build: _build011 {'MSVC_VERSION': '14.1', 'TARGET_ARCH': 'arm'}
Build: _build012 {'MSVC_VERSION': '14.1', 'TARGET_ARCH': 'arm64'}
Build: _build013 {'MSVC_VERSION': '14.1Exp', 'TARGET_ARCH': 'x86'}
Build: _build014 {'MSVC_VERSION': '14.1Exp', 'TARGET_ARCH': 'amd64'}
Build: _build015 {'MSVC_VERSION': '14.1Exp', 'TARGET_ARCH': 'arm'}
Build: _build016 {'MSVC_VERSION': '14.1Exp', 'TARGET_ARCH': 'arm64'}
Build: _build017 {'MSVC_VERSION': '14.0', 'TARGET_ARCH': 'x86'}
Build: _build018 {'MSVC_VERSION': '14.0', 'TARGET_ARCH': 'amd64'}
Build: _build019 {'MSVC_VERSION': '14.0', 'TARGET_ARCH': 'arm'}
Build: _build020 {'MSVC_VERSION': '12.0', 'TARGET_ARCH': 'x86'}
Build: _build021 {'MSVC_VERSION': '12.0', 'TARGET_ARCH': 'amd64'}
Build: _build022 {'MSVC_VERSION': '11.0Exp', 'TARGET_ARCH': 'x86'}
Build: _build023 {'MSVC_VERSION': '11.0Exp', 'TARGET_ARCH': 'amd64'}
Build: _build024 {'MSVC_VERSION': '10.0Exp', 'TARGET_ARCH': 'x86'}
Build: _build025 {'MSVC_VERSION': '10.0Exp', 'TARGET_ARCH': 'amd64'}
Build: _build026 {'MSVC_VERSION': '9.0', 'TARGET_ARCH': 'x86'}
Build: _build027 {'MSVC_VERSION': '9.0', 'TARGET_ARCH': 'amd64'}
Build: _build028 {'MSVC_VERSION': '8.0', 'TARGET_ARCH': 'x86', 'MSVC_USE_SCRIPT': 'C:\\Program Files\\Microsoft SDKs\\Windows\\v6.0\\Bin\\SetEnv.Cmd', 'MSVC_USE_SCRIPT_ARGS': '/Release /x86 /2003'}
Build: _build029 {'MSVC_VERSION': '8.0', 'TARGET_ARCH': 'x86'}
Build: _build030 {'MSVC_VERSION': '8.0', 'TARGET_ARCH': 'amd64'}
Build: _build031 {'MSVC_VERSION': '7.1', 'TARGET_ARCH': 'x86'}
Build: _build032 {'MSVC_VERSION': '7.0', 'TARGET_ARCH': 'x86'}
Build: _build033 {'MSVC_VERSION': '6.0', 'TARGET_ARCH': 'x86'}
Removing build folders ... 
Disabling scons cache ... 
Finished

In the second pass, the cache is used for all builds including the "default" 14.1 and 14.1Exp arm64 target builds with SDK 10.0.20348.0 instead of SDK 10.0.22000.0.

Pre-populating the cache could be useful on build servers after msvc updates.

bdbaddog · 2022-03-16T21:18:04Z

One thought as a comment to @jcbrill 's recent comment could we skip constructing a hash key by building a string, and instead just use a tuple? (They're hash'able as they're immutable (correct me if I'm wrong))

jcbrill · 2022-03-16T21:47:59Z

@bdbaddog Yes it can. However, a tuple is serialized to a list with json. The list would have to be converted to a tuple when loading the cache. This could be done in a dictionary comprehension.

mwichmann · 2022-03-16T23:55:22Z

It's not a secret that this whole thing was a hack - I run the SCons testsuite a lot, and running 1000+ distinct tests where many of them have to go through the msvc setup dance was a real pain. Stuffing this thing in on an experimental basis helped me in running those tests on Windows, though I don't spend that much time there. I'm more than happy to consider changes to help more maintstream usage.

bdbaddog · 2022-03-19T23:12:48Z

A couple thoughts:

@jcbrill - Could we convince you to create a GitHub repo with all of your experimental/prototype logic and then when you share in any PR/Issue you can point to such? I can create such repo owned by SCons (just to insure it's available in perpetuity)
Does the code in this PR fulfill these two goals?
a. not be worse than current implementation
b. Catch some filesystem based invalidation of the current caching mechanism, forcing recompute in those cases.

Sounds like from @jcbrill 's comment above that it's fine for 2015 and newer, but for MSVC's older it's going to invalidate the cache unnecessarily.
If that's the case, can we just apply this mechanism for MSVC's 2015 and newer?

jcbrill · 2022-03-20T15:52:04Z

It's not a secret that this whole thing was a hack - I run the SCons testsuite a lot, and running 1000+ distinct tests where many of them have to go through the msvc setup dance was a real pain. Stuffing this thing in on an experimental basis helped me in running those tests on Windows, though I don't spend that much time there. I'm more than happy to consider changes to help more maintstream usage.

I feel your pain. Some of my test sconstruct files run though 240+ msvc batch files. While I'm sure it is not as bad as your case, a single invocation takes about 5-6 minutes.

2. Does the code in this PR fulfill these two goals?
a. not be worse than current implementation
b. Catch some filesystem based invalidation of the current caching mechanism, forcing recompute in those cases.

a) Yes
b) Yes

Sounds like from @jcbrill 's comment above that it's fine for 2015 and newer, but for MSVC's older it's going to invalidate the cache unnecessarily.

If that's the case, can we just apply this mechanism for MSVC's 2015 and newer?

If I'm reading the code correctly, this is already achieved.

At the risk of annoying @mwichmann more than I already have:

Using a tuple for the key and serializing to a list of dictionaries would be a little more convenient if processing the cache file outside of scons.
"Normalizing" the script path string could increase the liklihood of a cache hit in the face of traditional msvc detection and user-defined batch files via MSVC_USE_SCRIPT.
Deserializing the list of dictionaries takes an extra step.

If my count is correct, it would take approximately two new lines of code and changes to three lines of code.

While testing the proposed changes below, it was revealed that the missing_ok argument to unlink in common.py fails in Python 3.6. It appears that the missing_ok argument was added in Python 3.8.

For my own work outside of scons, I am keen to see these changes work their way into scons.

vc.py fragments:

Existing read cache:

    if script_env_cache is None:
        script_env_cache = common.read_script_env_cache()
    cache_key = f"{script}--{args}" if args else f"{script}"
    cache_data = script_env_cache.get(cache_key, None)
  
Proposed read cache:

    if script_env_cache is None:
        cache = common.read_script_env_cache() # <-- temporary value
        script_env_cache = {tuple(d['script']): d['data'] for d in cache} # <-- reconstitute cache dictionary
    cache_key = (os.path.normcase(os.path.abspath(os.path.normpath(script))), args if args else None) # <-- normalize case of absolute path of normalized path
    cache_data = script_env_cache.get(cache_key, None)

...

Existing write cache:

    # once we updated cache, give a chance to write out if user wanted
    common.write_script_env_cache(script_env_cache)
    
Proposed write cache:
    
    # once we updated cache, give a chance to write out if user wanted
    cache = [{'script': list(key), 'data': val} for key, val in script_env_cache.items()]  # <- convert to list of dictionaries
    common.write_script_env_cache(cache)

I have to think about it some more, but it should be possible to write a batch file/python script/sconstruct file that reads the existing cache, extracts the scripts, deletes the cache file, and re-runs all of the existing batch files to regenerate the cache. The idea being that every time MSVC is updated, manually run the process to automatically update the cache.

Common.py fragments:

    ...
    except TypeError:
        # data can't serialize to json, don't leave partial file
        p.unlink(missing_ok=True) # <-- missing_ok python 3.8 and above
    ...

1. @jcbrill - Could we convince you to create a GitHub repo with all of your experimental/prototype logic and then when you share in any PR/Issue you can point to such? I can create such repo owned by SCons (just to insure it's available in perpetuity)

This is reasonable. With the exception of the standalone script I'm working on, most of the code snippets posted are throw-away efforts to demonstrate functionality. If you think there is value, I'm willing to work with you on this. If we can defer this in the short term, I would be grateful.

I have one more PR that is nearly ready that I would like to get issued and hopefully accepted which makes most of the my work outside of scons usable in scons via a standalone script placed in, or symlinked from, site_scons.

The standalone script could be useful to others and will eventually make its way into its own repository. The innards go through bouts of wholesale changes but the API usable from scons is fairly stable with only minor changes planned.

mwichmann · 2022-03-20T16:24:31Z

At the risk of annoying @mwichmann more than I already have:

Not annoying!

* Using a tuple for the key and serializing to a list of dictionaries would be a little more convenient if processing the cache file outside of scons.
* "Normalizing" the script path string could increase the liklihood of a cache hit in the face of traditional msvc detection and user-defined batch files via `MSVC_USE_SCRIPT`.
* Deserializing the list of dictionaries takes an extra step.
If my count is correct, it would take approximately two new lines of code and changes to three lines of code.

For my own work outside of scons, I am keen to see these changes work their way into scons.

vc.py fragments:

These look reasonable to me, although I have a silly preference for using Pathlib over the os.path stuff. I'll probably add these changes in soon.

Anyway:

While testing the proposed changes below, it was revealed that the missing_ok argument to unlink in common.py fails in Python 3.6. It appears that the missing_ok argument was added in Python 3.8.

Common.py fragments:

    ...
    except TypeError:
        # data can't serialize to json, don't leave partial file
        p.unlink(missing_ok=True) # <-- missing_ok python 3.8 and above
    ...

Oops, okay I'll update this bit right away.

Signed-off-by: Mats Wichmann <mats@linux.com>

jcbrill · 2022-03-26T14:03:11Z

Note for future reference: for MSVS 2015 and later, the msvc system environment includes path elements based on the latest SDK version number (e.g., 10.0.22000) at the time of invocation. All other things being equal, simply installing a new Windows SDK invalidates the cache entries for 2015 and later.

bdbaddog · 2022-03-27T03:58:26Z

Note for future reference: for MSVS 2015 and later, the msvc system environment includes path elements based on the latest SDK version number (e.g., 10.0.22000) at the time of invocation. All other things being equal, simply installing a new Windows SDK invalidates the cache entries for 2015 and later.

Are the older version still there and usable but not the latest (and thus invalidating the cache if it's meant to use latest)?
Or removed when a newer version installed?

jcbrill · 2022-03-27T10:59:58Z

Are the older version still there and usable but not the latest (and thus invalidating the cache if it's meant to use latest)?
Or removed when a newer version?

I believe the "correct" answer is that it is dependent on the user's installation. It may not be the same across all users. Both situations are possible depending on the user's installation habits.

There are check boxes for each SDK version in the Visual Studio installer. I'm not sure if de-selecting an SDK version in the installer removes it from the hard drive or not.

The windows SDKs (by version) also appear in both the windows control panel Programs and Features list and the windows settings apps list so it appears to be possible to uninstall individual SDKs from the control panel and/or settings.

The way the 2022 msvc batch file is written, it appears that unless the user specifies a specific SDK version to use as a command-line argument, the latest SDK version is selected provided that it meets the minimal requirements (app vs desktop, etc).

The basic process in a nutshell: the SDK folder is retrieved from the registry, the include folder directory names are sorted and walked via a dir command, folders beginning with 10. check for the existence of a known file , if the check file exists the SDK version is recorded. By the end of the walk the latest version is recorded. When the SDK is checked, a flag is set if the SDK version recorded was the user specified version. At the end of the walk, if the flag is set the user-specified version is used, otherwise the latest version is used.

Are the older version still there and usable but not the latest (and thus invalidating the cache if it's meant to use latest)?
Or removed when a newer version?

At best, the cache would contain "stale" entries but the environments would contain references to an older SDK that is still present.

At worst, the older SDK was uninstalled and the "stale" entries would refer to non-existent paths.

It should be possible to verify the cache entries for the SDKs but it is likely not as straightforward as one might hope.

mwichmann · 2022-03-29T17:21:17Z

Are the older version still there and usable but not the latest (and thus invalidating the cache if it's meant to use latest)?
Or removed when a newer version?

I believe the "correct" answer is that it is dependent on the user's installation. It may not be the same across all users. Both situations are possible depending on the user's installation habits.

There are check boxes for each SDK version in the Visual Studio installer. I'm not sure if de-selecting an SDK version in the installer removes it from the hard drive or not.

it appears to (I've done this before), though there's a cache of msvc "bits" unless you use some special options to avoid the cache. My vm's which don't have what you'd consider a "real" amount of disk, fill up even when I remove stuff. Sigh.

The windows SDKs (by version) also appear in both the windows control panel Programs and Features list and the windows settings apps list so it appears to be possible to uninstall individual SDKs from the control panel and/or settings.

I have done this also - for the same dumb reason, out of space, it's easier to nuke an sdk from Programs quickly than from the vs installer - and it's easy to bring back using the installer.

jcbrill · 2022-04-28T13:51:49Z

For your amusement and/or horror: a standalone script was developed to test the cache file name options and is available in the linked repository below. I'm not necessarily saying it is a good idea/method but may be a little more robust in the face of user input. If nothing else, it may be easier to test some ideas outside of SCons.

Unsolicited cache file name handling features:

Boolean accepts 0/False to disable cache rather than create a cache file named 0 or False. This eliminates asymmetric behavior of accepting boolean true but not boolean false.
Option "C" from above is implemented (more or less).
Expanduser and realpath are used when processing the user request. This might be important if symlinks are present?
The helper routine issues 2 debug calls: one upon entry and one upon exit no matter the path through the method.
A warning is issued when the user requested a cache file but the file name was set to None (invalid, path does not exist, etc.).
A debug call is issued when a cache write fails due to an IOError.

Code fragment (likely containing useless comments):
https://github.com/jcbrill/scons-prototypes/blob/97692bb6022a75492b656ccea9fc60a85bec18e3/fragments/scons_cache_filename.py#L36-L197

mwichmann · 2022-04-28T15:11:02Z

Horror, maybe: I didn't expect this "quick hack to help my own sanity waiting for the testsuite to run on Windows" thing to take on this kind of life - wondering if it's even worth all this effort? At a quick glance, this looks excellent... I do like that a "false-like" value actually means something if "true-like" does. and yes, integrating this with the rest of this subsystem in using logging in debug mode to record useful information is good, too.

jcbrill · 2022-04-28T15:47:12Z

Horror, maybe: I didn't expect this "quick hack to help my own sanity waiting for the testsuite to run on Windows" thing to take on this kind of life - wondering if it's even worth all this effort?

Probably not. "How hard could it be?" is usually where it goes off the rails...

jcbrill · 2022-04-28T16:18:30Z

I run the MSVC+MSVS tests frequently. With so many versions of msvc installed, the cache for just running the msvc/msvs tests has 21 entries. It is a time saver and useful to me. I appreciate that you added the cache.

bdbaddog · 2022-05-15T01:05:48Z

Is the tuple-ification of the key planned?
@jcbrill could you share the changes If my count is correct, it would take approximately two new lines of code and changes to three lines of code. ?

jcbrill · 2022-05-15T03:21:07Z

Is the tuple-ification of the key planned?
could you share the changes...

Sure. Will work on it tomorrow. Need to find and massage source reference implementation as it was not in either SCons or this branch.

jcbrill · 2022-05-15T16:12:33Z

Code available via https://github.com/jcbrill/scons/tree/mwichmann-msvc-cachefix

Issued a PR to https://github.com/mwichmann/scons/tree/msvc/cachefix

Feedback is welcome.

…json

Change cache key to tuple and read/write list of dictionaries to json

mwichmann · 2022-05-15T18:49:51Z

Merged - thanks. Let's see how the build does, then figure out next steps.

It doesn't need to be named .scons_msvc_cache.json, does not feel like it's really a "hidden" file, it'a a cache. Already renaming in this PR (adding the .json suffix) so sneak this change in as well. Signed-off-by: Mats Wichmann <mats@linux.com>

mwichmann · 2022-05-17T17:37:47Z

Post-merge update: confirmed that old and new cache files are incompatible - if you force an old cache file to be picked up by the current code, it will take a TypeError: string indices must be integers. Since the default cachefile name changed, this should be a minimal problem, but if someone was using a cachefile name of their own choosing, it should be removed before proceeding.

bdbaddog · 2022-05-17T17:44:49Z

can you add another except clause and message?

jcbrill · 2022-05-17T17:53:20Z

Thinking out loud... tt should be possible to detect if a list or dict was returned from json. If a dict, old format and convert to list and new format. Maybe? I would be happy to try later today.

mwichmann · 2022-05-17T18:42:04Z

"detect and baul out with msg" would be okay with me, fwiw. not sure we want to carry conversion code around.

jcbrill · 2022-05-17T23:15:54Z

Something like this:
https://github.com/jcbrill/scons/blob/jbrill-msvc-cachefix/SCons/Tool/MSCommon/common.py#L102-L125

def read_script_env_cache():
    """ fetch cached msvc env vars if requested, else return empty dict """
    envcache = {}
    if CONFIG_CACHE:
        try:
            p = Path(CONFIG_CACHE)
            with p.open('r') as f:
                # Convert the list of cache entry dictionaries read from
                # json to the cache dictionary. Reconstruct the cache key
                # tuple from the key list written to json.
                envcache_list = json.load(f)
                if isinstance(envcache_list, list):
                    envcache = {tuple(d['key']): d['data'] for d in envcache_list}
                else:
                    raise TypeError(
                        'SCONS_CACHE_MSVC_CONFIG cache file read error: expected type {}, found type {}.\n' \
                        '  Remove cache file {} and try again'.format(
                            repr('list'), repr(type(envcache_list).__name__), repr(CONFIG_CACHE)
                        )
                    )
        except FileNotFoundError:
            # don't fail if no cache file, just proceed without it
            pass
    return envcache

Output:

scons: Reading SConscript files ...
Build: _build001 {'MSVC_VERSION': '8.0', 'TARGET_ARCH': 'x86'}
TypeError: SCONS_CACHE_MSVC_CONFIG cache file read error: expected type 'list', found type 'dict'.
  Remove cache file 'S:\\SCons\\Test-Master_\\scons_msvc_cache.json' and try again:
  File "S:\SCons\Test-Master_\sconstruct-master", line 34:
    make_environment(**d)
  File "S:\SCons\Test-Master_\sconstruct-master", line 10:
    VariantDir(build, '.', duplicate=0)
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Script\SConscript.py", line 658:
    env = self.factory()
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Script\SConscript.py", line 638:
    default_env = SCons.Defaults.DefaultEnvironment()
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Defaults.py", line 83:
    _default_env = SCons.Environment.Environment(*args, **kw)
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Environment.py", line 1030:
    apply_tools(self, tools, toolpath)
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Environment.py", line 116:
    _ = env.Tool(tool)
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Environment.py", line 1906:
    tool(self)
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Tool\__init__.py", line 273:
    self.generate(env, *args, **kw)
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Tool\default.py", line 41:
    SCons.Tool.Tool(t)(env)
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Tool\__init__.py", line 273:
    self.generate(env, *args, **kw)
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Tool\mslink.py", line 310:
    msvc_setup_env_once(env)
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Tool\MSCommon\vc.py", line 1064:
    msvc_setup_env(env)
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Tool\MSCommon\vc.py", line 1202:
    d = msvc_find_valid_batch_script(env,version)
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Tool\MSCommon\vc.py", line 1117:
    d = script_env(vc_script, args=arg)
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Tool\MSCommon\vc.py", line 987:
    script_env_cache = common.read_script_env_cache()
  File "S:\SCons\jbrill-msvc-cachefix\scripts\..\SCons\Tool\MSCommon\common.py", line 119:
    repr('list'), repr(type(envcache_list).__name__), repr(CONFIG_CACHE)

bdbaddog · 2022-05-18T17:29:49Z

"detect and baul out with msg" would be okay with me, fwiw. not sure we want to carry conversion code around.

Yes. that's what I was thinking.
If we can avoid failing with some incomprehensible stack trace.
The cache can be considered invalid and discarded.

jcbrill · 2022-05-18T17:49:47Z

Yes. that's what I was thinking.
If we can avoid failing with some incomprehensible stack trace.
The cache can be considered invalid and discarded.

Perhaps not optimal, but once detected that the cache is invalid a warning could be issued indicating the cache was invalid and the CACHE_CONFIG variable could be reset to None which would effectively disable the cache and prevent overwriting but the build would continue. Unless the user removed the cache, the same thing would happen the next time.

Another alternative is to issue a warning that the cache is invalid on the read, but continue on and overwrite the cache during the run. In this case it is self-healing which is all a user is going to do anyway. I'm a little wary of overwriting the cache file.

And so it goes..

bdbaddog · 2022-05-18T19:56:09Z

I'd leave the filename and just overwrite it with whatever is detected per usual...

jcbrill · 2022-05-19T20:53:19Z

How about a warning and continuing?

Cache file likely overwritten. At present, this really only handles the old format change from dict to list.

With a warning, at least a user might understand why the run is taking longer. Warning sent to debug output if enabled as well.

import SCons.Warnings

class MSVCCacheInvalidWarning(SCons.Warnings.WarningOnByDefault):
    pass

...

def read_script_env_cache():
    """ fetch cached msvc env vars if requested, else return empty dict """
    envcache = {}
    if CONFIG_CACHE:
        try:
            p = Path(CONFIG_CACHE)
            with p.open('r') as f:
                # Convert the list of cache entry dictionaries read from
                # json to the cache dictionary. Reconstruct the cache key
                # tuple from the key list written to json.
                envcache_list = json.load(f)
                if isinstance(envcache_list, list):
                    envcache = {tuple(d['key']): d['data'] for d in envcache_list}
                else:
                    # don't fail if incompatible format, just proceed without it
                    warn_msg = "Incompatible format for msvc cache file {}: file may be overwritten.".format(
                        repr(CONFIG_CACHE)
                    )
                    SCons.Warnings.warn(MSVCCacheInvalidWarning, warn_msg)
                    debug(warn_msg)
        except FileNotFoundError:
            # don't fail if no cache file, just proceed without it
            pass
    return envcache

Partial output:

scons: Reading SConscript files ...
Build: _build001 {'MSVC_VERSION': '8.0', 'TARGET_ARCH': 'x86'}

scons: warning: Incompatible format for msvc cache file 'S:\\SCons\\Test-CacheFix\\scons_msvc_cache.json': file may be overwritten.
File "..\jbrill-msvc-cachefix\scripts\scons.py", line 98, in <module>
Build: _build002 {'MSVC_VERSION': '7.1', 'TARGET_ARCH': 'x86'}
Build: _build003 {'MSVC_VERSION': '7.0', 'TARGET_ARCH': 'x86'}
Build: _build004 {'MSVC_VERSION': '6.0', 'TARGET_ARCH': 'x86'}
scons: done reading SConscript files.
scons: Building targets ...

bdbaddog · 2022-05-19T20:57:27Z

How about a warning and continuing?

Cache file likely overwritten. At present, this really only handles the old format change from dict to list.

With a warning, at least a user might understand why the run is taking longer. Warning sent to debug output if enabled as well.

import SCons.Warnings

class MSVCCacheInvalidWarning(SCons.Warnings.WarningOnByDefault):
    pass

...

def read_script_env_cache():
    """ fetch cached msvc env vars if requested, else return empty dict """
    envcache = {}
    if CONFIG_CACHE:
        try:
            p = Path(CONFIG_CACHE)
            with p.open('r') as f:
                # Convert the list of cache entry dictionaries read from
                # json to the cache dictionary. Reconstruct the cache key
                # tuple from the key list written to json.
                envcache_list = json.load(f)
                if isinstance(envcache_list, list):
                    envcache = {tuple(d['key']): d['data'] for d in envcache_list}
                else:
                    # don't fail if incompatible format, just proceed without it
                    warn_msg = "Incompatible format for msvc cache file {}: file may be overwritten.".format(
                        repr(CONFIG_CACHE)
                    )
                    SCons.Warnings.warn(MSVCCacheInvalidWarning, warn_msg)
                    debug(warn_msg)
        except FileNotFoundError:
            # don't fail if no cache file, just proceed without it
            pass
    return envcache

Partial output:

scons: Reading SConscript files ...
Build: _build001 {'MSVC_VERSION': '8.0', 'TARGET_ARCH': 'x86'}

scons: warning: Incompatible format for msvc cache file 'S:\\SCons\\Test-CacheFix\\scons_msvc_cache.json': file may be overwritten.
File "..\jbrill-msvc-cachefix\scripts\scons.py", line 98, in <module>
Build: _build002 {'MSVC_VERSION': '7.1', 'TARGET_ARCH': 'x86'}
Build: _build003 {'MSVC_VERSION': '7.0', 'TARGET_ARCH': 'x86'}
Build: _build004 {'MSVC_VERSION': '6.0', 'TARGET_ARCH': 'x86'}
scons: done reading SConscript files.
scons: Building targets ...

Looks good to me, can you PR it?

jcbrill · 2022-05-19T21:05:27Z

#4152 so it's linked here

mwichmann added the MSVC Microsoft Visual C++ Support label Mar 8, 2022

mwichmann requested a review from bdbaddog March 8, 2022 19:54

bdbaddog reviewed Mar 9, 2022

View reviewed changes

SCons/Tool/MSCommon/vc.py Show resolved Hide resolved

MSVS_CACHE: Reword the CHANGES and RELEASE entries

50ddff5

Sharpened up the descriptions of the change in PR 4111. Changed the opening of the cache file to use pathlib functions. Signed-off-by: Mats Wichmann <mats@linux.com>

mwichmann force-pushed the msvc/cachefix branch from 65bdbc9 to 059f662 Compare March 12, 2022 17:43

Fix use of too-new (Py 3.8) pathlib feature

ec58ef7

Signed-off-by: Mats Wichmann <mats@linux.com>

Merge branch 'master' into msvc/cachefix

6f6dcfe

Change cache key to tuple and read/write list of dictionaries to json

7dcae31

jcbrill mentioned this pull request May 15, 2022

Change cache key to tuple and read/write list of dictionaries to json mwichmann/scons#7

Merged

3 tasks

jcbrill and others added 2 commits May 15, 2022 14:24

Add comments describing reason for data structure conversion to/from …

e7e365e

…json

Merge pull request #7 from jcbrill/mwichmann-msvc-cachefix

c816be1

Change cache key to tuple and read/write list of dictionaries to json

Drop leading '.' from cachefile name

529d45c

It doesn't need to be named .scons_msvc_cache.json, does not feel like it's really a "hidden" file, it'a a cache. Already renaming in this PR (adding the .json suffix) so sneak this change in as well. Signed-off-by: Mats Wichmann <mats@linux.com>

bdbaddog merged commit e1499fe into SCons:master May 15, 2022

msvc bug scrub 04-2022 automation moved this from In progress to Done May 15, 2022

mwichmann added this to the NextRelease milestone May 16, 2022

mwichmann deleted the msvc/cachefix branch May 16, 2022 13:01

jcbrill mentioned this pull request May 16, 2022

MSVC Version 14.1 not found with 14.2 installed #3664

Open

Tweak the MSVC environment vars cache #4111

Tweak the MSVC environment vars cache #4111

Conversation

mwichmann commented Mar 8, 2022 • edited

Contributor Checklist:

bdbaddog commented Mar 8, 2022

bdbaddog commented Mar 10, 2022

jcbrill commented Mar 11, 2022

mwichmann commented Mar 11, 2022

jcbrill commented Mar 11, 2022

jcbrill commented Mar 11, 2022

jcbrill commented Mar 11, 2022

mwichmann commented Mar 11, 2022 • edited

jcbrill commented Mar 11, 2022 • edited

mwichmann commented Mar 12, 2022

mwichmann commented Mar 12, 2022

jcbrill commented Mar 16, 2022 • edited

bdbaddog commented Mar 16, 2022

jcbrill commented Mar 16, 2022

mwichmann commented Mar 16, 2022

bdbaddog commented Mar 19, 2022

jcbrill commented Mar 20, 2022

mwichmann commented Mar 20, 2022

jcbrill commented Mar 26, 2022

bdbaddog commented Mar 27, 2022

jcbrill commented Mar 27, 2022

mwichmann commented Mar 29, 2022

jcbrill commented Apr 28, 2022

mwichmann commented Apr 28, 2022

jcbrill commented Apr 28, 2022

jcbrill commented Apr 28, 2022

bdbaddog commented May 15, 2022

jcbrill commented May 15, 2022

jcbrill commented May 15, 2022

mwichmann commented May 15, 2022

mwichmann commented May 17, 2022

bdbaddog commented May 17, 2022

jcbrill commented May 17, 2022

mwichmann commented May 17, 2022

jcbrill commented May 17, 2022

bdbaddog commented May 18, 2022

jcbrill commented May 18, 2022 • edited

bdbaddog commented May 18, 2022

jcbrill commented May 19, 2022

bdbaddog commented May 19, 2022

jcbrill commented May 19, 2022

mwichmann commented Mar 8, 2022 •

edited

mwichmann commented Mar 11, 2022 •

edited

jcbrill commented Mar 11, 2022 •

edited

jcbrill commented Mar 16, 2022 •

edited

jcbrill commented May 18, 2022 •

edited