Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression: Broken templating in EasyBuild 4.7.0 #2881

Closed
mboisson opened this issue Feb 8, 2023 · 12 comments · Fixed by #2882
Closed

Regression: Broken templating in EasyBuild 4.7.0 #2881

mboisson opened this issue Feb 8, 2023 · 12 comments · Fixed by #2882
Milestone

Comments

@mboisson
Copy link
Contributor

mboisson commented Feb 8, 2023

I am not sure exactly when it happened, but recent EasyBuild versions broke template replacement in configopts when using multi_deps, i.e. :

multi_deps = {'Python': ['3.10', '3.9', '3.8'] }
configopts = "-DPYQT5_SIP_DIR=$EBROOTQT5/share/python%(pyshortver)s/site-packages/sip/PyQt5 -DPYQT5_MOD_DIR=$EBROOTQT5/lib/python%(pyshortver)s/site-packages/PyQt5 "

for a CMakeMake EasyBlock (maybe others, I have not tested) no longer yield builds with different configopts argument based on %(pyshortver)s.

In a previous build log, %(pyshortver)s gets replaced for every iteration:

grep run.py /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.16.10/easybuild/*.log | grep cmake | grep running
== 2021-08-31 19:14:35,595 run.py:233 INFO running cmd:  cmake -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.16.10 -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -std=gnu++11' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -std=gnu++11' -DCMAKE_Fortran_COMPILER='gfortran' -DCMAKE_Fortran_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC' -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=FALSE  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DCMAKE_PREFIX_PATH=$EBROOTQT -DQJSON_DIR=$EBROOTQJSON/lib/cmake/qjson -DQSCINTILLA_LIBRARY=$EBROOTQSCINTILLA/lib/libqscintilla2_qt5.so -DWITH_QTWEBKIT=OFF -DPYQT5_SIP_DIR=$EBROOTQT5/share/python3.7/sip/PyQt5 -DQSCI_SIP_DIR=$EBROOTQSCINTILLA/share/sip/PyQt5 /tmp/ebuser/avx2/QGIS/3.16.10/GCC-9.3.0/qgis-3.16.10/
== 2021-08-31 22:14:21,025 run.py:233 INFO running cmd:  cmake -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.16.10 -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -std=gnu++11' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -std=gnu++11' -DCMAKE_Fortran_COMPILER='gfortran' -DCMAKE_Fortran_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC' -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=FALSE  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DCMAKE_PREFIX_PATH=$EBROOTQT -DQJSON_DIR=$EBROOTQJSON/lib/cmake/qjson -DQSCINTILLA_LIBRARY=$EBROOTQSCINTILLA/lib/libqscintilla2_qt5.so -DWITH_QTWEBKIT=OFF -DPYQT5_SIP_DIR=$EBROOTQT5/share/python3.8/sip/PyQt5 -DQSCI_SIP_DIR=$EBROOTQSCINTILLA/share/sip/PyQt5 /tmp/ebuser/avx2/QGIS/3.16.10/GCC-9.3.0/qgis-3.16.10/
== 2021-08-31 23:54:14,144 run.py:233 INFO running cmd:  cmake -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.16.10 -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -std=gnu++11' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -std=gnu++11' -DCMAKE_Fortran_COMPILER='gfortran' -DCMAKE_Fortran_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC' -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=FALSE  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DCMAKE_PREFIX_PATH=$EBROOTQT -DQJSON_DIR=$EBROOTQJSON/lib/cmake/qjson -DQSCINTILLA_LIBRARY=$EBROOTQSCINTILLA/lib/libqscintilla2_qt5.so -DWITH_QTWEBKIT=OFF -DPYQT5_SIP_DIR=$EBROOTQT5/share/python3.9/sip/PyQt5 -DQSCI_SIP_DIR=$EBROOTQSCINTILLA/share/sip/PyQt5 /tmp/ebuser/avx2/QGIS/3.16.10/GCC-9.3.0/qgis-3.16.10/

while in a recent build, it keeps the same value as for the first iteration:

$ grep run.py /tmp/eb-3yw8sr68/easybuild-6ry1d4vr.log | grep cmake | grep running
== 2023-02-08 16:18:35,282 run.py:236 INFO running cmd: patch -p1 -i /home/mboisson/git/easybuild-easyconfigs/easybuild/easyconfigs/q/QGIS/QGIS-3.22.14-FindPyQt5.cmake.patch
== 2023-02-08 16:18:56,589 run.py:236 INFO running cmd:  cmake  -DCMAKE_INSTALL_PREFIX=/home/mboisson/.local/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.22.14 -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DWITH_QTWEBKIT=OFF -DPYQT5_SIP_DIR=$EBROOTQT5/share/python3.10/site-packages/sip/PyQt5 -DPYQT5_MOD_DIR=$EBROOTQT5/lib/python3.10/site-packages/PyQt5 -DQSCI_SIP_DIR=$EBROOTQSCINTILLA/share/sip/PyQt5 -DPYQT5_VERSION_STR=5.15.5  /tmp/mboisson/avx2/QGIS/3.22.14/GCC-9.3.0/qgis-3.22.14/
== 2023-02-08 16:31:47,766 run.py:236 INFO running cmd: patch -p1 -i /home/mboisson/git/easybuild-easyconfigs/easybuild/easyconfigs/q/QGIS/QGIS-3.22.14-FindPyQt5.cmake.patch
== 2023-02-08 16:32:07,346 run.py:236 INFO running cmd:  cmake   -DCMAKE_INSTALL_PREFIX=/home/mboisson/.local/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.22.14 -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DWITH_QTWEBKIT=OFF -DPYQT5_SIP_DIR=$EBROOTQT5/share/python3.10/site-packages/sip/PyQt5 -DPYQT5_MOD_DIR=$EBROOTQT5/lib/python3.10/site-packages/PyQt5 -DQSCI_SIP_DIR=$EBROOTQSCINTILLA/share/sip/PyQt5 -DPYQT5_VERSION_STR=5.15.5  /tmp/mboisson/avx2/QGIS/3.22.14/GCC-9.3.0/qgis-3.22.14/
@mboisson
Copy link
Contributor Author

mboisson commented Feb 8, 2023

According to this, it worked at least up to version 4.6.0:

for f in $(grep -rl "configopts.*pyshortver" /cvmfs/soft.computecanada.ca/easybuild/ebfiles_repo/2020/); do grep easybuild-framework_version $f; done
    "easybuild-framework_version": "4.4.1-r73a32642a48430484a7c9a4654e7afda27268be4",
    "easybuild-framework_version": "4.3.3-r15719a0ece2011cea9ce176d1763cdc620b188f4",
    "easybuild-framework_version": "4.6.0-ra6dbb4b22832c431590f725219d89d82c9946dba",
    "easybuild-framework_version": "4.6.0-ra6dbb4b22832c431590f725219d89d82c9946dba",
    "easybuild-framework_version": "4.6.0-ra6dbb4b22832c431590f725219d89d82c9946dba",
    "easybuild-framework_version": "4.6.0-ra6dbb4b22832c431590f725219d89d82c9946dba",
    "easybuild-framework_version": "4.3.3-r3ff498d1f735c51b313c04c5e4abeeaecc23589b",
    "easybuild-framework_version": "4.3.3-r3ff498d1f735c51b313c04c5e4abeeaecc23589b",
    "easybuild-framework_version": "4.2.2-r64415f53d9bdff498788ce494f68a98641a3883d",
    "easybuild-framework_version": "4.2.2-r64415f53d9bdff498788ce494f68a98641a3883d",
    "easybuild-framework_version": "4.4.0-r5824ede87a67bf66b912bcb3f7d73c27835716af",
    "easybuild-framework_version": "4.4.0-r5824ede87a67bf66b912bcb3f7d73c27835716af",
    "easybuild-framework_version": "4.4.0-r5824ede87a67bf66b912bcb3f7d73c27835716af",
    "easybuild-framework_version": "4.3.3-r15719a0ece2011cea9ce176d1763cdc620b188f4",

Looking at one of these recipes, built with EasyBuild 4.6.0:

grep "run.py" /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hoomd-blue-mpi/3.6.0/easybuild/easybuild-HOOMD-blue-3.6.0-20221117.183917.log | grep cmake | grep running
== 2022-11-17 18:03:15,250 run.py:233 INFO running cmd:  cmake -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hoomd-blue-mpi/3.6.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='mpicc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_CXX_COMPILER='mpicxx' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_Fortran_COMPILER='mpifort' -DCMAKE_Fortran_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=FALSE  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DENABLE_MPI=ON -DENABLE_CUDA=OFF -DENABLE_TBB=ON -DENABLE_LLVM=OFF -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hoomd-blue-mpi/3.6.0/lib/python3.8/site-packages  /tmp/ebuser/avx2/HOOMDblue/3.6.0/gompi-2020a-mpi/hoomd-v3.6.0/
== 2022-11-17 18:15:51,643 run.py:233 INFO running cmd:  cmake -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hoomd-blue-mpi/3.6.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='mpicc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_CXX_COMPILER='mpicxx' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_Fortran_COMPILER='mpifort' -DCMAKE_Fortran_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=FALSE  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DENABLE_MPI=ON -DENABLE_CUDA=OFF -DENABLE_TBB=ON -DENABLE_LLVM=OFF -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hoomd-blue-mpi/3.6.0/lib/python3.9/site-packages  /tmp/ebuser/avx2/HOOMDblue/3.6.0/gompi-2020a-mpi/hoomd-v3.6.0/
== 2022-11-17 18:27:07,729 run.py:233 INFO running cmd:  cmake -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hoomd-blue-mpi/3.6.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='mpicc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_CXX_COMPILER='mpicxx' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_Fortran_COMPILER='mpifort' -DCMAKE_Fortran_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=FALSE  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DENABLE_MPI=ON -DENABLE_CUDA=OFF -DENABLE_TBB=ON -DENABLE_LLVM=OFF -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hoomd-blue-mpi/3.6.0/lib/python3.10/site-packages  /tmp/ebuser/avx2/HOOMDblue/3.6.0/gompi-2020a-mpi/hoomd-v3.6.0/

@mboisson mboisson changed the title Regression: Broken templating in recent EasyBuild Regression: Broken templating in EasyBuild 4.7.0 Feb 8, 2023
@boegel boegel added this to the next release (4.7.1?) milestone Feb 8, 2023
@mboisson
Copy link
Contributor Author

mboisson commented Feb 8, 2023

This commit from @Flamefire is my first suspect. It messes with configopts and has no explicit code to handle templates

0b78b02

@branfosj
Copy link
Member

branfosj commented Feb 8, 2023

If I've got my testing correct:

  • This broke between 4.6.0 and 4.6.1
  • This impacts CMake only - not ConfigureMake

@mboisson
Copy link
Contributor Author

mboisson commented Feb 8, 2023

If I've got my testing correct:

  • This broke between 4.6.0 and 4.6.1
  • This impacts CMake only - not ConfigureMake

That would match the above commit, I think

@boegel
Copy link
Member

boegel commented Feb 8, 2023

I confirmed that this can not be reproduced using MEGAHIT-1.2.9-GCCcore-9.3.0.eb (which uses multi_deps + CMakeMake) when fiddling with preconfigopts, so looks specific to CMakeMake and configopts.

This mean is indeed a likely culprit, so I'll move this issue into the easyblocks repo...

@boegel boegel transferred this issue from easybuilders/easybuild-framework Feb 8, 2023
@mboisson
Copy link
Contributor Author

mboisson commented Feb 8, 2023

There's another recently opened issue (easybuilders/easybuild-easyconfigs#17259) caused by this PR #2514

@mboisson
Copy link
Contributor Author

mboisson commented Feb 8, 2023

To avoid running into this in the future, it may be worth displaying a warning when __setitem__ is called with enable_templating = True in easyconfig.py ?

Flamefire added a commit to Flamefire/easybuild-easyblocks that referenced this issue Feb 9, 2023
We need to get the original value not resolving templates to keep them
e.g. for further iterations.

Fixes easybuilders#2881
@Flamefire
Copy link
Contributor

Flamefire commented Feb 9, 2023

To avoid running into this in the future, it may be worth displaying a warning when __setitem__ is called with enable_templating = True in easyconfig.py ?

Actually the issue is __getitem__ as __setitem__ doesn't do any templating. Furthermore we do a lot of modifications with enabled templating without running into this issue.

I haven't found any of our ECs to be affected by this. If anyone can share one to test this that would help.
Cmd used: grep -F configopts $(grep -F --files-with-matches multi_deps $(grep -rF --files-with-matches CMakeMake easybuild-easyconfigs/easybuild/easyconfigs)) | grep -F '%(py'

Flamefire added a commit to Flamefire/easybuild-easyblocks that referenced this issue Feb 9, 2023
We need to get the original value not resolving templates to keep them
e.g. for further iterations.

Fixes easybuilders#2881
@Flamefire
Copy link
Contributor

That PR may have another regression related to iterating options: First time the configopts are set/changed they are as specified by the easyconfig. During the next iteration they are already changed.

Example (after fix) of a single run with 2 python versions:

configopts=-DPYVER=%(pyshortver)s
  running command "cmake  -DCMAKE_INSTALL_PREFIX=/beegfs/ws/1/s3248973-EasyBuild/easybuild-haswell/software/MetaBAT/2.14-gompi-2019a -DBoost_NO_SYSTEM_PATHS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DBOOST_ROOT=/sw/installed/Boost/1.70.0-gompi-2019a -DPYVER=3.7 /tmp/s3248973-EasyBuild/MetaBAT/2.14/gompi-2019a/MetaBAT-2.14/"

configopts=-DCMAKE_INSTALL_PREFIX=/tmp/easybuild-tmp/eb-D18oYP/__ROOT__/beegfs/ws/1/s3248973-EasyBuild/easybuild-haswell/software/MetaBAT/2.14-gompi-2019a -DBoost_NO_SYSTEM_PATHS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DBOOST_ROOT=/sw/installed/Boost/1.70.0-gompi-2019a -DPYVER=%(pyshortver)s
  running command "cmake   -DCMAKE_INSTALL_PREFIX=/beegfs/ws/1/s3248973-EasyBuild/easybuild-haswell/software/MetaBAT/2.14-gompi-2019a -DBoost_NO_SYSTEM_PATHS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DBOOST_ROOT=/sw/installed/Boost/1.70.0-gompi-2019a -DPYVER=2.7 /tmp/s3248973-EasyBuild/MetaBAT/2.14/gompi-2019a/MetaBAT-2.14/"

@mboisson
Copy link
Contributor Author

mboisson commented Feb 9, 2023

To avoid running into this in the future, it may be worth displaying a warning when __setitem__ is called with enable_templating = True in easyconfig.py ?

Actually the issue is __getitem__ as __setitem__ doesn't do any templating. Furthermore we do a lot of modifications with enabled templating without running into this issue.

I disagree the issue is __getitem__. While __getitem__ is what resolves the templates, __setitem__ is what wrongly replaces values with templates unresolved by values with templates resolved. My point is that one should probably never use __setitem__ with template values resolved.

It is reasonable to call __getitem__ without disabling templating (that's what it's for, obviously), but it is unreasonable to call __setitem__ with the value obtained from __getitem__ with templates resolved. The dangerous action here is __setitem__

I haven't found any of our ECs to be affected by this. If anyone can share one to test this that would help. Cmd used: grep -F configopts $(grep -F --files-with-matches multi_deps $(grep -rF --files-with-matches CMakeMake easybuild-easyconfigs/easybuild/easyconfigs)) | grep -F '%(py'

You will find many in our fork https://github.com/ComputeCanada/easybuild-easyconfigs

@mboisson
Copy link
Contributor Author

mboisson commented Feb 9, 2023

At least, I would add a test like the one proposed in
easybuilders/easybuild-framework#4212

so that the build fails hard and clearly, rather than failing silently and letting the user to ponder

Flamefire added a commit to Flamefire/easybuild-easyblocks that referenced this issue Feb 9, 2023
Partial revert of easybuilders#2541:
Do not modify `cfg['configopts']` but simply read them and pass the
combined result to `run_cmd`.

This fixes an issue with templating while iterating over e.g. multiple Pythons
and also avoids keeping values in configopts over multiple iterations/builds.

Fixes easybuilders#2881
@Flamefire
Copy link
Contributor

There's another recently opened issue (easybuilders/easybuild-easyconfigs#17259) caused by this PR #2514

That is actually a different issue. I fixed that with a patch in easybuilders/easybuild-easyconfigs#17288

At least, I would add a test like the one proposed in easybuilders/easybuild-framework#4212

so that the build fails hard and clearly, rather than failing silently and letting the user to ponder

Not sure how we can reasonably do that as we do self.cfg[...] = ... or even self.cfg[...] += ... almost everywhere. And even self.cfg.update (which we also use a lot) doesn't handle the templating although there are drawbacks doing that (deduplication may fail).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment