Systematic build failure of ROOT master in the LCG DEBUG builds since Jan 15th #9594
Comments
|
Update: v6-26-00-patches also affected since last night. |
|
@vgvassilev FYI Relevant part of the backtrace: |
|
@gganis what's the |
|
The second cases you report seem unrelated at a first look: and |
|
The CMake invocation is here. And this is the corresponding log: |
|
# Well, a part the timing correlation, the changes triggering the problem are the ones ported to v6-26-00-patches on Jan 16, 2022, which are related to Vc. |
|
Hi @vgvassilev, yes, reverting this 5 commits fixes the problem. |
|
Any plan when this can be addressed, at least for v6-26-00-patches? The LCG nightlies are not very usable at the moment. |
|
@vgvassilev expects to revert still today. |
|
Is this addressed? Can this be closed? |
|
No, still open: #9641 |
Axel-Naumann
added
the
experiment
|
The real solution of #9736 is probably related to this issue. The ROOT master branch is still failing in our builds. |
|
@andresailer, it looks like we have externally built Vc and we do not pass in the location of its header files. IIRC, we had the same issue for the cuda builds and we solved it by passing the include path in the env variable In order to move forward this issue needs to be fixed on the LCG side. I am cc-ing @peremato as we did such a fix some time ago for the ROOT CUDA builds. I bet he knows where to add it as he already did it once. |
|
I see, we can add that, thank you for the explanation. |
The build failure is a just a symptom of the issue. In fact, ROOT was already built and in its final step it runs a sanity check something like I'd be happy if we could solve this in a nicer way, however I do not think ROOT has enough information about its build process (and I think it should not). For example, even if we detect where Vc (or other relevant information) was at build time, this does not mean a lot. We cannot be sure that it stays at the same place on the deployment node. For instance, CMSSW-like setups then, at install time, reshuffle things around quite a bit. That environment variable is there because of that - essentially provides a way for "devops" (to use a modern word here) to express extra knowledge which is (nearly) impossible to deduce at build time. In addition, I think it'd be pretty bad if ROOT modified users' environment unknowingly. |
|
@vgvassilev would it make sense to export |
|
That will solve the build issue which is just a symptom. However, it will not solve the same problem at deployment time. It seems to be equivalent to removing the Even if we decided to go that route, I am not sure if we have that information in the right places in the build system. |
You assume that build and deployment have a similar settings; not sure that's the case. As another argument for specifying this during ROOT's build, we also specify
We can do it on a case by case basis. We know which "builtins" we depend on wrt runtime includes, and we know where they are during configure / build time. For Vc it's |
|
Here is how we would set ROOT_INCLUDE_PATH for the build https://gitlab.cern.ch/sft/lcgcmake/-/commit/f50af97822f3fc4dba9fd56bee3cb76c39fe5aeb For cuda we had to add a location to ROOT_INCLUDE_PATH for the view after the installation https://gitlab.cern.ch/sft/lcgcmake/-/commit/cc14dbbb56a1cb4b59e68c11b9df3f75c21fc1ba |
While the install might have it at a findable place, the build (and running hsimple.C during build time) might not and ROOT needs help to find the (runtime) headers from Vc. See root-project#9594.
|
If we understand correctly, this should not block 6.26n anymore as the patches that cause this failure have been reverted in 6.26 (they are still present in master). |
|
Correct, 6-26-00-patches is building fine. |
|
A fix should be this: #9765 - I'd certainly appreciate a confirmation! Else I'll just merge and you tell me then. |
While the install might have it at a findable place, the build (and running hsimple.C during build time) might not and ROOT needs help to find the (runtime) headers from Vc. See root-project#9594.
|
While ROOT master is now building successfully for us, there are some downstream packages that suffer from this change. They can probably be fixed by exporting the ROOT_INCLUDE_PATH, but to better understand where things go wrong I made a very simple repeater. I think on any centos7 with the sft cvmfs this should work Just linking an executable to ROOTTPython (https://gitlab.cern.ch/sailer/root_repeater/-/blob/master/CMakeLists.txt#L17) Even though Vc.pcm is sitting right next to MathCore.pcm |
|
The location of the Vc.pcm does not tell ROOT where the include paths of Vc are, in a similar way how the @Axel-Naumann, it sounds like we need to extend your fix in cmake should we want to continue in that direction. |
|
Indeed testing these downstream packages will likely need Vc, and we cannot know where that is, so that's a case for The second part of this issue is about the error message. @vgvassilev do you agree that isn't equivalent to "ROOT cannot find Vc/Vc.h; please provide the include path by adding it to |
|
@vgvassilev anything we can do to improve the error message, see #9594 (comment) ? |
|
@Axel-Naumann, not easily. This crash happens when clang tries to load the dependencies of MathCore, so no easy way to diagnose. Maybe if we iterate over all modules, read their dependencies and respect the loading of the dependencies in that order, we might be able to diagnose this. However, to go through the dependencies of a given PCM without loading it requires implementing some binary reading. I've seen such code, and it is doable but.. |
|
Based on the discussion, it's not clear to me if there's anything left to be done. Can we close this? @vgvassilev |
|
Let me re-iterate @vgvassilev , @andresailer : can we perhaps close? |
|
Let's close it. |
|
Hi @vgvassilev, @Axel-Naumann, It appears this issue is closed, but wasn't yet added to a project. Please add upcoming versions that will include the fix, or 'not applicable' otherwise. Sincerely, |
|
My reproducer does't fail with ROOT HEAD any more, so 🤷 |
|
Hi @vgvassilev, @Axel-Naumann, It appears this issue is closed, but wasn't yet added to a project. Please add upcoming versions that will include the fix, or 'not applicable' otherwise. Sincerely, |
|
@vgvassilev what you need here, as the bot describes it, is a project, not a milestone |
|
I haven't seen more useless thing recently... |
|
It's a little price to pay in order to collect all fixed issues for a certain release - a very useful feature for our users! |
|
So once we mention what's being fixed in the PR and once more elsewhere? In principle that information is rather easy to pull out without having people to specify these things. It is great to have automation to the extent it is useful. The problem with the ROOT repository is that it has a lot of these things with dubious usefulness for the day-to-day work. I am happy to elaborate move if necessary... |
None found
Describe the bug
Since three nights (Jan 15th), the dev3 (ROOT-master) debug builds fail systematically with an error in
See for example this .
Expected behavior
A successful build ;-) .
To Reproduce
LCG nightlies (lcgcmake). Did not try yet in standalone.
Setup
dnf install/ binary download / you built it yourself: built from source (github)Additional context
None for the moment
The text was updated successfully, but these errors were encountered: