Fix lazy loading of template specializations #13139

hahnjo · 2023-06-30T15:02:37Z

No description provided.

github-actions · 2023-06-30T20:55:56Z

Test Results

        9 files         9 suites 1d 20h 55m 10s ⏱️
  2 472 tests   2 472 ✔️ 0 💤 0 ❌
21 147 runs 21 147 ✔️ 0 💤 0 ❌

Results for commit 40cef5e.

♻️ This comment has been updated with latest results.

vgvassilev · 2023-07-01T15:13:49Z

Build failed on mac13/cxx20. Running on macphsft21.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build See console output.

Failing tests:

* [projectroot.roottest.python.distrdf.common.roottest_python_distrdf_common_test_all](https://lcgapp-services.cern.ch/root-jenkins/job/root-pullrequests-build/178636/testReport/projectroot.roottest.python.distrdf/common/roottest_python_distrdf_common_test_all/)

* [projectroot.roottest.python.distrdf.dask.roottest_python_distrdf_dask_test_all](https://lcgapp-services.cern.ch/root-jenkins/job/root-pullrequests-build/178636/testReport/projectroot.roottest.python.distrdf/dask/roottest_python_distrdf_dask_test_all/)

* [projectroot.roottest.python.distrdf.spark.roottest_python_distrdf_spark_test_all](https://lcgapp-services.cern.ch/root-jenkins/job/root-pullrequests-build/178636/testReport/projectroot.roottest.python.distrdf/spark/roottest_python_distrdf_spark_test_all/)

I believe we need to put a cling::Interpreter::PushTransactionRAII in cling::ParseProto::Parse because with the new patch we might trigger a deserialization.

hahnjo · 2023-07-03T08:09:06Z

I believe we need to put a cling::Interpreter::PushTransactionRAII in cling::ParseProto::Parse because with the new patch we might trigger a deserialization.

Ok, not sure I fully understand what you propose... Can you add this change before the final "fixup" commit? You can push to the PR, right, or do I need to enable this somehow?

phsft-bot · 2023-07-03T09:46:40Z

Starting build on ROOT-debian10-i386/soversion, ROOT-performance-centos8-multicore/cxx17, ROOT-ubuntu2204/nortcxxmod, ROOT-ubuntu2004/python3, mac11/noimt, mac12arm/cxx20, windows10/cxx14
How to customize builds

phsft-bot · 2023-07-03T11:41:41Z

Build failed on ROOT-debian10-i386/soversion.
Running on pcepsft11.dyndns.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

projectroot.roottest.root.io.json.roottest_root_io_json_make

hahnjo · 2023-07-03T12:20:58Z

@phsft-bot build just on mac11/cxx17, mac11arm/cxx17, mac12/cxx17, mac12arm/cxx17, mac13/cxx20, mac13arm/cxx20 with flags -DCTEST_TEST_EXCLUDE_NONE=ON

phsft-bot · 2023-07-03T12:21:04Z

Starting build on mac11/cxx17, mac11arm/cxx17, mac12/cxx17, mac12arm/cxx17, mac13/cxx20, mac13arm/cxx20 with flags -DCTEST_TEST_EXCLUDE_NONE=ON
How to customize builds

phsft-bot · 2023-07-03T13:00:36Z

Build failed on mac13arm/cxx20.
Running on macphsft28.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

phsft-bot · 2023-07-03T13:29:31Z

Build failed on mac11arm/cxx17.
Running on macphsft24.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Warnings:

[2023-07-03T12:52:21.831Z] ld: warning: direct access in function 'ROOT::GenerateInitInstanceLocal(TKDTree<int, double> const*)' from file 'math/mathcore/CMakeFiles/G__MathCore.dir/G__MathCore.cxx.o' to global weak symbol 'typeinfo for TKDTree<int, double>' from file 'math/mathcore/CMakeFiles/MathCore.dir/src/TKDTree.cxx.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
[2023-07-03T12:52:21.831Z] ld: warning: direct access in function 'ROOT::GenerateInitInstanceLocal(TKDTree<int, double> const*)' from file 'math/mathcore/CMakeFiles/G__MathCore.dir/G__MathCore.cxx.o' to global weak symbol 'typeinfo for TKDTree<int, double>' from file 'math/mathcore/CMakeFiles/MathCore.dir/src/TKDTree.cxx.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
[2023-07-03T12:52:21.831Z] ld: warning: direct access in function 'ROOT::GenerateInitInstanceLocal(TKDTree<int, float> const*)' from file 'math/mathcore/CMakeFiles/G__MathCore.dir/G__MathCore.cxx.o' to global weak symbol 'typeinfo for TKDTree<int, float>' from file 'math/mathcore/CMakeFiles/MathCore.dir/src/TKDTree.cxx.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
[2023-07-03T12:52:21.831Z] ld: warning: direct access in function 'ROOT::GenerateInitInstanceLocal(TKDTree<int, float> const*)' from file 'math/mathcore/CMakeFiles/G__MathCore.dir/G__MathCore.cxx.o' to global weak symbol 'typeinfo for TKDTree<int, float>' from file 'math/mathcore/CMakeFiles/MathCore.dir/src/TKDTree.cxx.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
[2023-07-03T12:52:31.604Z] ld: warning: direct access in function 'ROOT::GenerateInitInstanceLocal(TMatrixTBase<float> const*)' from file 'math/matrix/CMakeFiles/G__Matrix.dir/G__Matrix.cxx.o' to global weak symbol 'typeinfo for TMatrixTBase<float>' from file 'math/matrix/CMakeFiles/Matrix.dir/src/TMatrixTBase.cxx.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
[2023-07-03T12:52:31.604Z] ld: warning: direct access in function 'ROOT::GenerateInitInstanceLocal(TMatrixTBase<float> const*)' from file 'math/matrix/CMakeFiles/G__Matrix.dir/G__Matrix.cxx.o' to global weak symbol 'typeinfo for TMatrixTBase<float>' from file 'math/matrix/CMakeFiles/Matrix.dir/src/TMatrixTBase.cxx.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
[2023-07-03T12:52:31.604Z] ld: warning: direct access in function 'ROOT::GenerateInitInstanceLocal(TMatrixTBase<double> const*)' from file 'math/matrix/CMakeFiles/G__Matrix.dir/G__Matrix.cxx.o' to global weak symbol 'typeinfo for TMatrixTBase<double>' from file 'math/matrix/CMakeFiles/Matrix.dir/src/TMatrixTBase.cxx.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
[2023-07-03T12:52:31.604Z] ld: warning: direct access in function 'ROOT::GenerateInitInstanceLocal(TMatrixTBase<double> const*)' from file 'math/matrix/CMakeFiles/G__Matrix.dir/G__Matrix.cxx.o' to global weak symbol 'typeinfo for TMatrixTBase<double>' from file 'math/matrix/CMakeFiles/Matrix.dir/src/TMatrixTBase.cxx.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
[2023-07-03T12:52:31.604Z] ld: warning: direct access in function 'ROOT::GenerateInitInstanceLocal(TVectorT<float> const*)' from file 'math/matrix/CMakeFiles/G__Matrix.dir/G__Matrix.cxx.o' to global weak symbol 'typeinfo for TVectorT<float>' from file 'math/matrix/CMakeFiles/Matrix.dir/src/TVectorT.cxx.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
[2023-07-03T12:52:31.604Z] ld: warning: direct access in function 'ROOT::GenerateInitInstanceLocal(TVectorT<float> const*)' from file 'math/matrix/CMakeFiles/G__Matrix.dir/G__Matrix.cxx.o' to global weak symbol 'typeinfo for TVectorT<float>' from file 'math/matrix/CMakeFiles/Matrix.dir/src/TVectorT.cxx.o' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.

And 126 more

Failing tests:

projectroot.roottest.python.numba.roottest_python_numba_numba

hahnjo · 2023-07-03T18:58:59Z

@phsft-bot build with flags -Druntime_cxxmodules=OFF -DCTEST_TEST_EXCLUDE_NONE=ON

phsft-bot · 2023-07-03T18:59:04Z

Starting build on ROOT-debian10-i386/soversion, ROOT-performance-centos8-multicore/cxx17, ROOT-ubuntu2204/nortcxxmod, ROOT-ubuntu2004/python3, mac11/noimt, mac12arm/cxx20, windows10/cxx14 with flags -Druntime_cxxmodules=OFF -DCTEST_TEST_EXCLUDE_NONE=ON
How to customize builds

phsft-bot · 2023-07-03T19:26:05Z

Build failed on ROOT-performance-centos8-multicore/cxx17.
Running on olbdw-01.cern.ch:/data/sftnight/workspace/root-pullrequests-build
See console output.

Failing tests:

projectroot.roottest.root.core.roottest_root_core_execStatusBitsCheck

phsft-bot · 2023-07-03T19:47:39Z

Build failed on mac12arm/cxx20.
Running on macphsft26.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

phsft-bot · 2023-07-03T20:18:20Z

Build failed on mac11/noimt.
Running on macphsft23.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

phsft-bot · 2023-07-04T09:15:23Z

Starting build on ROOT-debian10-i386/soversion, ROOT-performance-centos8-multicore/cxx17, ROOT-ubuntu2204/nortcxxmod, ROOT-ubuntu2004/python3, mac11/noimt, mac12arm/cxx20, windows10/cxx14
How to customize builds

vgvassilev

That's really a great job, @hahnjo!

For hashing, we should not differentiate between templates and qualified templates and only look at the underlying name. See also https://reviews.llvm.org/D153003

This avoids deserialization if necessary, for example in DeclUnloader. See also https://reviews.llvm.org/D154328

Due to hash collisions, it can happen that we load another template specialization with the same hash. This is fine, as long as the next call to findSpecializationImpl does not find a matching Decl for the template arguments.

The LLVM 13 port of the patch missed to adapt findSpecialization of ClassTemplateDecl and VarTemplateDecl. During the upgrade, we only noticed and fixed FunctionTemplateDecl::findSpecialization.

phsft-bot · 2023-07-04T14:15:39Z

Starting build on ROOT-debian10-i386/soversion, ROOT-performance-centos8-multicore/cxx17, ROOT-ubuntu2204/nortcxxmod, ROOT-ubuntu2004/python3, mac11/noimt, mac12arm/cxx20, windows10/cxx14
How to customize builds

phsft-bot · 2023-07-04T16:19:21Z

Build failed on mac11/noimt.
Running on macphsft20.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

projectroot.roottest.root.dataframe.roottest_root_dataframe_test_snapshot

hahnjo added the in:Cling label Jun 30, 2023

hahnjo self-assigned this Jun 30, 2023

hahnjo requested review from vgvassilev and Axel-Naumann as code owners June 30, 2023 15:02

This comment was marked as outdated.

Sign in to view

hahnjo force-pushed the fix-lazy branch from 9ae0c6c to 464199b Compare July 3, 2023 06:13

This comment was marked as outdated.

Sign in to view

vgvassilev mentioned this pull request Jul 3, 2023

ROOT loads unneeded PCMs #13000

Open

1 task

This comment was marked as outdated.

Sign in to view

hahnjo force-pushed the fix-lazy branch from dbc54d5 to 4fca909 Compare July 4, 2023 09:15

vgvassilev approved these changes Jul 4, 2023

View reviewed changes

hahnjo and others added 6 commits July 4, 2023 15:27

[clang] Fix ODR hashing of template template parameters

56dcdbb

For hashing, we should not differentiate between templates and qualified templates and only look at the underlying name. See also https://reviews.llvm.org/D153003

[clang] Add API to iterate already loaded specializations

73a9a58

This avoids deserialization if necessary, for example in DeclUnloader. See also https://reviews.llvm.org/D154328

[clang] Relax assertion on lazily loading template specs

13abc9f

Due to hash collisions, it can happen that we load another template specialization with the same hash. This is fine, as long as the next call to findSpecializationImpl does not find a matching Decl for the template arguments.

ParseTypeName might trigger deserialization.

ecdf035

[clang] Fix up application of D41416

41840bf

The LLVM 13 port of the patch missed to adapt findSpecialization of ClassTemplateDecl and VarTemplateDecl. During the upgrade, we only noticed and fixed FunctionTemplateDecl::findSpecialization.

[llvm-project] Bump to tag ROOT-llvm13-20230704-01

40cef5e

hahnjo force-pushed the fix-lazy branch from 4fca909 to 40cef5e Compare July 4, 2023 14:15

hahnjo merged commit 3231079 into root-project:master Jul 4, 2023
14 of 16 checks passed

hahnjo deleted the fix-lazy branch July 4, 2023 17:13

This was referenced Jul 4, 2023

Revert D41416: "Do not deserialize all lazy template specializations when looking for one." #13112

Closed

Test failures with C++17, but without runtime_cxxmodules #13058

Closed

hahnjo mentioned this pull request Jul 13, 2023

Fix lazy loading of template specializations [v6.28] #13240

Merged

vgvassilev mentioned this pull request Feb 20, 2024

Revert D153003 "[clang] Fix ODR hashing of template template parameters" #14777

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix lazy loading of template specializations #13139

Fix lazy loading of template specializations #13139

hahnjo commented Jun 30, 2023

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Jun 30, 2023 •

edited

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

vgvassilev commented Jul 1, 2023

Failing tests:

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

hahnjo commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

hahnjo commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

hahnjo commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

phsft-bot commented Jul 4, 2023

vgvassilev left a comment

phsft-bot commented Jul 4, 2023

phsft-bot commented Jul 4, 2023

Fix lazy loading of template specializations #13139

Fix lazy loading of template specializations #13139

Conversation

hahnjo commented Jun 30, 2023

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Jun 30, 2023 • edited

Test Results

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

vgvassilev commented Jul 1, 2023

Failing tests:

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

hahnjo commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

Failing tests:

hahnjo commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

Failing tests:

phsft-bot commented Jul 3, 2023

Warnings:

Failing tests:

hahnjo commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

phsft-bot commented Jul 3, 2023

Failing tests:

phsft-bot commented Jul 3, 2023

Failing tests:

phsft-bot commented Jul 3, 2023

Failing tests:

phsft-bot commented Jul 4, 2023

vgvassilev left a comment

Choose a reason for hiding this comment

phsft-bot commented Jul 4, 2023

phsft-bot commented Jul 4, 2023

Failing tests:

github-actions bot commented Jun 30, 2023 •

edited