Remove environment name from build_cache_dir cache keys
#4574
Comments
|
Are you sure that build environments (flags, etc) are the same for both projects? Please try to run both projects with |
|
I’ll try to see if I can get a run output for you. Hopefully today. I can confirm though, that if I remove my build output for a specific device, and run the build again in esphome for the same device, most (all iirc) of the output - even the final “firmware.elf” comes from the cache. So my assumption is that all things are identical - except for the environment name. |
|
I've not been able to figure out how to run I've captured what I think is the relevant commandline from when I trigger a rebuild of two of my identical devices, and they're shown here: If I remove all the inputs ( Other files are compiled with similar (long) commandlines. Example for |
|
I've tried modifying what I think is the code that does the hashing, to see what the contents of the platformio-core/platformio/project/helpers.py Lines 106 to 114 in 065607b I've put them below. I see multiple path-specific items, not just the Something weird is going on though, because [
[
"common",
[
[
"lib_deps",
""
],
[
"build_flags",
""
],
[
"upload_flags",
""
]
]
],
[
"platformio",
[
[
"description",
"ESPHome 2023.3.0"
],
[
"globallib_dir",
"/piolibs"
],
[
"platforms_dir",
"/data/cache/platformio/platforms"
],
[
"packages_dir",
"/data/cache/platformio/packages"
],
[
"cache_dir",
"/data/cache/platformio/cache"
],
[
"build_dir",
"/data/light-41-1/.pioenvs"
],
[
"libdeps_dir",
"/data/light-41-1/.piolibdeps"
]
]
],
[
"env:light-41-1",
[
[
"board",
"esp8285"
],
[
"board_build.flash_mode",
"dout"
],
[
"board_build.ldscript",
"eagle.flash.1m.ld"
],
[
"build_flags",
[
"-DESPHOME_LOG_LEVEL=ESPHOME_LOG_LEVEL_INFO",
"-DNEW_OOM_ABORT",
"-DPIO_FRAMEWORK_ARDUINO_LWIP2_HIGHER_BANDWIDTH_LOW_FLASH",
"-DUSE_ARDUINO",
"-DUSE_ESP8266",
"-DUSE_ESP8266_FRAMEWORK_ARDUINO",
"-DUSE_STORE_LOG_STR_IN_FLASH",
"-Wno-nonnull-compare",
"-Wno-sign-compare",
"-Wno-unused-but-set-variable",
"-Wno-unused-variable",
"-fno-exceptions"
]
],
[
"extra_scripts",
[
"post:post_build.py"
]
],
[
"framework",
[
"arduino"
]
],
[
"lib_deps",
[
"ottowinter/ESPAsyncTCP-esphome@1.2.3",
"esphome/ESPAsyncWebServer-esphome@2.1.0",
"DNSServer",
"ESP8266WiFi",
"ESP8266mDNS",
"esphome/noise-c@0.1.4",
"bblanchon/ArduinoJson@6.18.5"
]
],
[
"lib_ldf_mode",
"off"
],
[
"platform",
"platformio/espressif8266 @ 3.2.0"
],
[
"platform_packages",
[
"platformio/framework-arduinoespressif8266 @ ~3.30002.0"
]
]
]
]
] |
|
So the hashing code above is obviously a red herring, since it doesn't actually run in my case, but if I set I followed the log message above to SCons:
And here I fear that |
|
Sorry, we can't help you so much with the 3rd party software. If you still experience any issues with PlatformIO Core, please create clean, simple, and independent PlatformIO-based project to reproduce this issue. |
|
I believe the gist of the issue is still the same as my previous ones - how do we build projects with multiple environments fast. ESPHome, ESPurna, etc. build process are already as minimal as we can get the example? Libraries are built separately, framework files are built separately; all while build flags stay the exact same, it becomes not very effective way to spend CPU resources to rebuild everything. |
|
@mcspr , could you reproduce this issue with the bare PlatformIO projects? |
|
@ivankravets https://github.com/mcspr/pio4574 So, there are two identical envs - 'a' and 'b'. Doing some clean-up and building 'a' first > rm -rf .build_cache/
> pio run -s -t clean
> pio run -e a | grep 'from cache' | grep libFrameworkArduino.a
> pio run -s -t clean
> pio run -e a | grep 'from cache' | grep libFrameworkArduino.a
Retrieved `.pio/build/a/libFrameworkArduino.a' from cacheCache picked up our framework file, everything good. Building 'b' next > pio run -e b | grep 'from cache' | grep libFrameworkArduino.aCan't reuse already built file from 'a' per the description above, even when they are identical. 2nd run of 'b' works with the cache related to 'b' only > pio run -s -e b -t clean
> pio run -e b | grep 'from cache' | grep libFrameworkArduino.a
Retrieved `.pio/build/b/libFrameworkArduino.a' from cache |
|
Thanks for the report. This is a bug. Please re-test with |
|
Still does not retrieve cached .a from one env to the other Because of env separation,
Which in turn depends on the env name, which we wanted to avoid |
|
Could it be the limitation of SCons? I think SCons does not cache library archives. cc: @bdbaddog, @mwichmann |
|
I found earlier that SCons internally likely uses the full path name to the file it is working on. This means that any compilation will be unique for different environments by default :/ Maybe stuff can be done with relative paths as suggested. |
|
SCons (unlike many other build tools) can cache all targets it can build. Not just ones created by compilers/linkers. SCons uses the full command line to generate any target (along with content hashes of the sources, including implicit sources (potentially compiler binary, and included header files, libraries, etc) to generate the hash used to name the file in the cache. So yes a full path in the target would cause the cache to be less useful. You can exclude parts of the command line from the hash calculation, but I don't think you could exclude part of the path of the target for an absolute path. BTW SCons (unlike many other build tools) can cache all targets it can build. Not just ones created by compilers/linkers. |
Hacking around the signature generator and removing based on the #4574 (comment) tracing of where it comes from. I guess whether it is relative or absolutet depends on how PIO invokes scons, the way I build it always comes up relative to project root. |
|
BUILD_DIR is not a SCons provided envvar. So that patch wouldn't work in SCons. However there is a |
|
where is the offending directory located relative to the SConstruct? In a sub dir, a sibling dir, or up a few and down a few dirs? |
|
@mcspr, do you see any issues in the PlatformIO Core? Maybe, it is our problem? From what I remember, we always use relative paths to the project directory. |
|
get_relpath() gives a very weird path in that context
Project root is just
What I was trying to do is to have |
The gist of the issue (as I understood it) - cache for certain build output depends on path. Since we have multiple environments, What I tried above is to simply remove the BUILD DIR from the signature calc for the node. When building same thing, relative paths inside build directory cause cache to be shared. |
|
If this is the SCons |
|
@bdbaddog, @mwichmann, thanks again for being willing to help us! I've just created a small project (scons-caching.zip) to reproduce this issue. If you try to build the first time, you will see that for both variants "hello.c" is built. What do PlatformIO users expect, that Do you have any ideas on how to solve it? |
|
I forgot about the log for the project which I provided:
What do we expect when the project is fully cleaned? Something like this: I understand that "build2" has not been built yet. But |
|
Haven't had a chance to look at the repro, but just a quick thought can your build use |
|
@bdbaddog, yes, see the source code of CacheDir('build_cache')
VariantDir('build1', 'src', duplicate=False)
VariantDir('build2', 'src', duplicate=False)
env1 = Environment()
env1.Program('build1/hello.c')
env2 = Environment()
env2.Program('build2/hello.c') |
|
The best I could come up with is to just have a custom env var SCons/scons@master...mcspr:pio4574 Pretty annoying to manage combined with VariantDir, but seems to work. |
So this works fine unless the target with path is included in the generated file by the builder. |
Yes, correct. Let me give you a few real examples. C/C++ developers use 3-rd party libraries/frameworks in the projects. These libraries/frameworks are common per project. Following the example above, each project will have its own Now, when you build a few projects on the same host machine (or CI) developers expect that common libraries/frameworks will not be recompiled for each project. This is actually the context of this issue #4574. If we use common |
@mcspr, could I ask you to file a feature request at https://github.com/SCons/scons/issues and point to this issue? You already reviewed the SCons sources and have a better understanding of this problem. |
|
In this example, you can place the built tree (say of boost - I know that's a common one) in what SCons calls a "repository", then various builds in SCons will populate from that as long as they specify that they use a repository. It's kind of like a cache at the opposite end, if that makes sense. |
|
This is unlikely to be a general change, but rather one specified per builder, or enabled per user, because as I've said above some tools embed the target file/location/other related info in the generated file. |
|
@mwichmann, thanks for the hint with The issue with we faced is to use |
|
the actual objects are copied to the cache as they're built, and copied back when needed. |
|
Had a thought. Not sure if it would resolve this, but can one of you give it a try? So if the Does that make sense? |
|
Right. But, looking at the code path cachedir sig takes, children and Nodes own contents sig match up until it appends path of the node that differs. Only using the Nodes 'children + own' signature instead of 'children + own + path' would definitely fix the issue, 'own' signature is enough to distinguish targets |
|
You are correct, ignoring the target from the build signature won't resolve it. The problem is ignoring the target file and it's path is not a good change for all builders. For your usage (any many) this may be acceptable for compile and link builders. Please file an enhancement request issue in SCons' github repo. |
If I understood the idea correctly... def factory func or scanner somewhere on PIO side?(node, *arg, **kwarg):
x = env.File(node)
x.set_cachedir_bsig_path('')
return x
Will do |
|
I was wondering about this behaviour for a few years, and by coincidence I've decided to digg this out myself today. On topic: ESPHome auto-generates the source directories by itself. What would happen if we would make it generate symlinks for all the library code and the components? Wouldnt these sourcefiles then be seen as identical paths on disk, thus hitting the cache? |
|
Sounds like an approach like https://pnpm.io/ is using. NPM has the same issue, just way worse: If APP dependes on A and B, and A also depends on B, npm creates a folder structure that includes B twice. This does not scale at all. Pnpm fixes this with a hierarchy of symlinks. To make that work here, the links path needs to be resolved - I didn't spot any "readlink"-like approach when I was looking through code - but I might've missed it. |
|
It's not an excuse, just an explanation: because of the Windows symlink situation (previously needed elevated privilege, now needing "developer mode" to be set, neither of which SCons itself can control), SCons has stayed away from depending heavily on symlinks. There are cases, like variant dirs, where symlinking is done and there's a policy difference between platforms on whether to attempt, but that's relatively rare. |
|
I think it's generally a good idea to stay away from relying on symlinks, thats for sure. But in this case, if the source-file happens to be a symlink (because that's simply the was the source-folders were organized), maybe SCons could follow the symlink and store the real location of the file as part of the cache-key instead. What other options do we have left? |
What kind of issue is this?
Start by telling us what problem you’re trying to solve. Often a solution
already exists! Don’t send pull requests to implement new features without first getting our
support. Sometimes we leave features out on purpose to keep the project small.
Configuration
Operating system: docker/linux, host is Debian 10, image is
ghcr.io/esphome/esphome-hassio:2023.3.0PlatformIO Version (
platformio --version):PlatformIO Core, version 6.1.6Description of problem
I'm using
esphometo manage tens of mostly identical devices with identical sets of source code sans a few items such as their individual name and some encryption keys. When I useesphometo compile the firmwares for these devices, they create a separate environment in platformio for each device, which is good for isolation. It does take a long time to compile the mostly identical source code though, for all these devices, when updates are available. I'd like to alleviate that.In another issue, I've been digging in to what can be done, and I've experimented with the
build_cache_dirsetting. It's not currently used by esphome. This setting works wonders for a single device: if I recompile after cleaning the output, the build is instantaneous (as can be), as it uses the cache.The cached output from one device is not used in another, and I've found this topic (https://community.platformio.org/t/build-cache-dir-will-not-share-object-files-between-envs/10011/8) on your community forums that provides a crucial hint. It seems the
env:nameis part of the cache key, and as each device in esphome is a separate environment, I have my issue.I hope this issue can lead to the
env:namebeing removed from the cache key, to allow the build cache to be reused across environments. It's my impression that the rest of the build commandline includes all the details that are in environments, such as boards, platforms and libraries.Steps to Reproduce
build_cache_dirto some directoryActual Results
The cache is not reused between identical builds, if they have two environments.
Expected Results
Environment name should not affect the cache key. It's my impression that the rest of the build commandline (which is part of the key) is enough to distinguish one output from another.
If problems with PlatformIO Build System:
The content of
platformio.ini:This file is modified by me, locally, to include the
cache_build_dir. It is identical to other devices I have, except for the environment name.Additional info
The text was updated successfully, but these errors were encountered: