Bump CMake to 3.17 #2157

davidrohr · 2020-03-27T15:59:28Z

No description provided.

davidrohr · 2020-03-27T21:01:52Z

@ktf @MichaelLettrich : Could you check what is the problem with alidist_macOS? I have force-pushed once, and it still fails without providing an error log.
The new CMake is required for the TPC Raw Encoder (for some weird dependency reason related to CUDA and ROOT... My bug report was finally fixed in CMake 3.17).

MichaelLettrich · 2020-03-29T09:30:50Z

The only thing I can see is that it is failing to build CMake on macOS. I will switch on debug output for alibuild and see what is going on.

MichaelLettrich · 2020-03-29T09:32:27Z

cmake.sh

@@ -1,14 +1,14 @@
 package: CMake
 version: "%(tag_basename)s"
-tag: "v3.16.3-alice2"
+tag: "v3.17.0"
 source: https://github.com/alisw/CMake
 build_requires:
 - "GCC-Toolchain:(?!osx)"
 - make
 prefer_system: "(?!osx)"


If the bug is fixed in 3.17, then we can use system CMake if it is recent enough, right?

Can you remove the "(?!osx)" part at prefer_system? If 3.17 fixes the bug we should use the brew version instead of always building our own.

davidrohr · 2020-03-29T09:33:37Z

ok, thx, but if the CMake build fails, why is the log empty?
System CMake 3.17 should be OK, but at the moment the prefer_system check is disabled for osx for CMake. Was there a reason for that?

davidrohr · 2020-03-29T09:35:25Z

Actually, I am stupid.. The PR is actually wrong. There is no 3.17 in the alisw fork, we should use the upstream. Let me fix that. But I am wondering how it worked in the linux CI? It should have failed there!

MichaelLettrich · 2020-03-29T09:41:08Z

I have no Idea why the log is empty. I think it is a "design" feature of the system that only the log of the tested package (in this case O2) is published - am I right @ktf ?

davidrohr · 2020-03-29T10:23:26Z

OK, but that doesn't make sense. One should see the log for the package that failed.

MichaelLettrich · 2020-03-29T10:57:32Z

Agree. I will discuss with @ktf as soon as we can. I think we need to make some improvements under the hood as empty logs keep devs in the dark...

MichaelLettrich · 2020-03-29T10:58:28Z

95abf44bd/osx_x86-64/CMake/v3.17.0-1ake:2157: + export GIT_REFERENCE=/build/mirror/cmake
2020-03-29@11:59:02:DEBUG:O2Suite:CMake:2157: + GIT_REFERENCE=/build/mirror/cmake
2020-03-29@11:59:02:DEBUG:O2Suite:CMake:2157: + [[ ! https://gitlab.kitware.com/cmake/cmake == '' ]]
2020-03-29@11:59:02:DEBUG:O2Suite:CMake:2157: + [[ h != \/ ]]
2020-03-29@11:59:02:DEBUG:O2Suite:CMake:2157: + [[ ! -d /System/Volumes/Data/build/ci_checks/alidist_checker_macos_o2_ci_0/sw/SOURCES/CMake/v3.17.0/v3.17.0 ]]2:DEBUG:O2Suite:CMake:2157: + cd /System/Volumes/Data/build/ci_checks/alidist_checker_macos_o2_ci_0/sw/SOURCES/CMake/v3.17.0/v3.17.0@11:59:02:DEBUG:O2Suite:CMake:2157: + git checkout v3.17.0
2020-03-29@11:59:02:DEBUG:O2Suite:CMake:2157: error: pathspec 'v3.17.0' did not match any file(s) known to git

davidrohr · 2020-03-29T11:21:00Z

ok, thx, probably the log was empty because it failed fetching the tag. Just pushed a new version, should be fixed now. Tried locally in alidock and on MacOS.

MichaelLettrich · 2020-03-29T11:34:52Z

I still get the same error in the logs. Seems that alidist is not able to check out that repo for some reason

davidrohr · 2020-03-29T11:44:43Z

Even with the latest commit? It works for me. Could you perhaps wipe the build area?

MichaelLettrich · 2020-03-29T11:59:18Z

First of all I stand corrected: Empty logs are a result of the error happening outside of the build process of a package, for example if git can't check out the repository as in this case.

Secondly, I see that the package is checked out in the sources, i.e.
/build/ci_checks/alidist_checker_macos_o2_ci_0/sw/SOURCES/CMake/v3.17.0/v3.17.0 and is populated. I see that origin still points to https://github.com/alisw/CMake though where there obviously is no v3.17.0 tag ...
Wondering how aliBuild is dealing with the repository switch bc there is obviously only one source in the mirror we use. @ktf can you maybe give me an inside?

davidrohr · 2020-03-29T13:35:19Z

Done, but I think it might still fail to build when the old nonexistent tag is checked out in the src dir.

MichaelLettrich · 2020-03-29T13:56:28Z

Sure the system check was just another thing that was missing. AliBuild and switching between repos is yet another story. I wonder nevertheless why it behaves differently on linux. I hope that now we at least get the same results on both platforms.

MichaelLettrich · 2020-03-30T06:45:19Z

@ktf @sawenzel Travis seems stuck

davidrohr · 2020-03-30T07:22:45Z

Also the MacOS O2 CI is stuck since ~18 hours. Could you have a look?
It is kind of important to merge this rather soon, since it is needed for AliceO2Group/AliceO2#3244

MichaelLettrich · 2020-03-30T07:40:10Z

Yes, the macOS CI was my bad. I quit the builder to do some debuging and forgot to restart it again 😓 its back in business.

davidrohr · 2020-03-30T07:41:41Z

ok, thx, will it automatically pick up the stuck PRs without force-pushing?

MichaelLettrich · 2020-03-30T09:07:19Z

no need to force-push, the build is in process.

MichaelLettrich · 2020-03-30T12:19:55Z

@Barthelemy any ideas why qcg-latest is failing on macos?

davidrohr · 2020-03-30T13:24:11Z

@Barthelemy @MichaelLettrich : I do not see how the MacOS failure here can be related to MacOS. I breaks in npm. I have saved the log if you need it. Force-pushing to try again.

MichaelLettrich · 2020-03-30T13:32:34Z

@davidrohr The error occurs with other prs as well, see #2158 #2156

awegrzyn · 2020-03-30T13:35:16Z

It might be node-gyp confused between python 2 and 3. I'm rebuilding now to investigate.

aphecetche · 2020-03-30T13:38:41Z

AFAIK this is a gyp issue. Last time I struggled with it the conclusion was that it was not python3 ready and so a python2 installation needs to be available, see OGUI-304. Don't know if this has changed since then ? Maybe @awegrzyn knows ?

MichaelLettrich · 2020-03-30T14:02:57Z

@aphecetche the new macOS checkers were setup exclusively with python3. Searching the web told me that gyp is python3 ready since version 5.0.4

awegrzyn · 2020-03-30T14:41:44Z

Unfortunately, I'm not able to reproduce it, I can successfully compile it with Nodejs from aliBuild and brew.

awegrzyn · 2020-03-30T16:49:43Z

This is where node-gyp is failing: https://github.com/nodejs/node-gyp/blob/v5.0.5/gyp/pylib/gyp/mac_tool.py#L247

Even though I use NODE_GYP_FORCE_PYTHON to set /usr/bin/python3 this file runs with python 2.7 (I still have it in my system under /usr/bin/python) as on the top of this file there's:

#!/usr/bin/env python

but build machines set /usr/bin/python -> /usr/bin/python3 I guess this is the problem...

awegrzyn · 2020-03-30T17:01:46Z

We're using node-gyp@v5.0.5 and there's fix for this in node-gyp@v5.0.6.
This #2163 updates to NodeJS that uses node-gyp@v5.1.0 so it should fix the issue.

davidrohr · 2020-03-30T20:57:47Z

@ktf @MichaelLettrich : Could you check the logs of the AliRoot/alidist CI? The log shown here is empty again. This morning, the AliRoot CI was already green, I just rebased after the python fix for the Mac CI and now it fails.

MichaelLettrich · 2020-03-31T08:34:35Z

Looks like some git glitches on build/AliRoot/alidist - don't know what exactly makes this so unreliable...

++ hostname -s
+ DATA='prcheck,checkname=build/O2/alidist/1 host="alientest06",state="looping",cihash="af88634af0a544e7ba704baf0314301a495c1a80",uptime=38281,prid="2157",prok=1 1585604795000000000'ile:zPf5mNrrmt5GaLS@dbod-hltmon.cern.ch:8082/write?db=agile' --data-binary 'prcheck,checkname=build/O2/alidist/1 host="alientest06",state="looping",cihash="af88634af0a544e7ba704baf0314301a495c1a80",uptime=38281,prid="2157",prok=1 1585604795000000000'                     Dload  Upload   Total   Spent    Left  Speed
100   173    0     0  100   173      0   1157 --:--:-- --:--:-- --:--:--  1161
+ '[' -f config/workers-pool-size ']'
+ '[' -f config/worker-index ']'
+ '[' -f config/debug ']'
+ '[' -f config/profile ']'
+ '[' -f config/jobs ']'
+ '[' -f config/timeout ']'
+ '[' -f config/long-timeout ']'
+ '[' -f config/silent ']'
+ '[' '!' -f config/silent ']'
+ unset SILENT
+ '[' '!' -f config/debug ']'
+ unset DEBUG
+ '[' '!' -f config/profile ']'
+ unset PROFILE
+ aliBuild clean
++ find . -maxdepth 2 -name .git -exec dirname '{}' ';'
++ grep -v ali-bot
+ for d in '$(find . -maxdepth 2 -name .git -exec dirname {} \; | grep -v ali-bot)'
+ pushd ./alidist
++ git rev-parse --abbrev-ref HEAD
+ LOCAL_BRANCH=master
+ [[ master != HEAD ]]
+ git update-ref -d 'refs/remotes/origin/pr/*'
+ true
++ date -u +%s
+ [[ 8487 -gt 14400 ]]
+ timeout -s9 600 git fetch origin +master:refs/remotes/origin/master
+ git reset --hard origin/master
+ git clean -fxd
+ popd
+ for d in '$(find . -maxdepth 2 -name .git -exec dirname {} \; | grep -v ali-bot)'
+ pushd ./alibuild
++ git rev-parse --abbrev-ref HEAD
+ LOCAL_BRANCH=HEAD
+ [[ HEAD != HEAD ]]
+ popd
+ for d in '$(find . -maxdepth 2 -name .git -exec dirname {} \; | grep -v ali-bot)'
+ pushd ./O2
++ git rev-parse --abbrev-ref HEAD
+ LOCAL_BRANCH=dev
+ [[ dev != HEAD ]]
+ git update-ref -d 'refs/remotes/origin/pr/*'
error: unable to resolve reference refs/remotes/origin/pr/*: No such file or directory
+ true
++ date -u +%s
+ [[ 8488 -gt 14400 ]]
+ timeout -s9 600 git fetch origin +dev:refs/remotes/origin/dev
+ git reset --hard origin/dev
+ git clean -fxd
+ popd
+ find separate_logs/ -type f -mtime +5 -delete
+ find separate_logs/ -type d -empty -delete
+ [[ alisw/alidist != '' ]]
++ cat force-hashes
++ grep -vE '^#'
++ true
+ HASHES=
+ [[ ! -n '' ]]
++ timeout -s9 600 list-branch-pr --show-main-branch --check-name build/O2/alidist --trust-collaborators --trusted ktf,dberzano alisw/alidist@master --workers-pool-size 2 --worker-index 1 --max-wait 300

MichaelLettrich · 2020-03-31T08:42:14Z

@ktf There seem to be some dns issues with build/O2/alidist-dataflow

Using cached build for CMake
Building CMake@v3.17.0
rsync: getaddrinfo: repo-ci.marathon.mesos 873: Name or service not known
rsync error: error in socket IO (code 10) at clientserver.c(125) [sender=3.1.2]
ERROR: Unable to upload tarball.
+ BUILD_ERROR=1
+ [[ 1 != '' ]]
+ timeout -s9 600 report-pr-errors --default master --no-comments --logs-dest s3://alice-build-logs.s3.cern.ch --log-url https://ali-ci.cern.ch/alice-build-logs/ --pr alisw/alidist#2157@724598dadc38bffaf78f395e2f4e7bbcfd86892f -s build/O2/alidist-dataflow
Searching all logs matching: sw/BUILD/*latest*/log, sw/BUILD/*latest/log
Found:

echo 'Builing on: '`hostname` >> copy-logs/alisw/alidist/2157/724598dadc38bffaf78f395e2f4e7bbcfd86892f/build_O2_alidist-dataflow/fullLog.txt

echo 'The following files are present in the log:' >> copy-logs/alisw/alidist/2157/724598dadc38bffaf78f395e2f4e7bbcfd86892f/build_O2_alidist-dataflow/fullLog.txt

ktf · 2020-03-31T09:17:27Z

Ok. Thank you for checking. I will investigate what is going on.

davidrohr · 2020-03-31T18:46:30Z

@ktf : Do we understand what is the new problem with the CI? Both AliRoot and O2 mac OS CIs have been green in the meantime, but not at the same time. So in principle this PR should be fine. I am sorry I am pushing, but I need it for AliceO2Group/AliceO2#3244

davidrohr · 2020-04-02T07:57:29Z

@ktf : Can we merge this? It is open since 6 days, and it is constantly blocked by randomly failing CI.
All of the CIs have been green in between in the mean time, just not at the same time.
And this is needed for some O2 PRs, which are ready to be merged.
In case it breaks something, I am online and available to fix it.

ktf · 2020-04-02T08:30:33Z

Sorry I missed this. @MichaelLettrich what's up with the mac builds?

This follows recent change in alidist alisw/alidist#2157

davidrohr requested a review from a team as a code owner March 27, 2020 15:59

ghost requested a review from ktf March 27, 2020 15:59

davidrohr force-pushed the master_pull_request2 branch from 10802dc to b2b930f Compare March 27, 2020 16:52

davidrohr force-pushed the master_pull_request2 branch 2 times, most recently from c7e6165 to ddf63e0 Compare March 28, 2020 19:24

davidrohr mentioned this pull request Mar 28, 2020

Work on ZS encoding / decoding + unrelated commits from my dev branch that give conflicts if not merged together AliceO2Group/AliceO2#3244

Merged

MichaelLettrich reviewed Mar 29, 2020

View reviewed changes

davidrohr force-pushed the master_pull_request2 branch from ddf63e0 to df5a24a Compare March 29, 2020 09:36

davidrohr force-pushed the master_pull_request2 branch from df5a24a to a308fb2 Compare March 29, 2020 11:20

davidrohr force-pushed the master_pull_request2 branch from 376870f to 984d45c Compare March 30, 2020 13:24

davidrohr added 3 commits March 30, 2020 22:38

Bump CMake to 3.17

5e8482b

Remove ROOT workaround required for old cmake

4af47f8

Allow system CMake on MacOS

724598d

davidrohr force-pushed the master_pull_request2 branch from 984d45c to 724598d Compare March 30, 2020 20:38

MichaelLettrich approved these changes Mar 30, 2020

View reviewed changes

MichaelLettrich mentioned this pull request Apr 1, 2020

Unconditionally disable GPU tests in CI for incremental builds #2168

Merged

davidrohr mentioned this pull request Apr 1, 2020

calibrations in DPL AliceO2Group/AliceO2#2803

Merged

ktf merged commit 25a243f into alisw:master Apr 2, 2020

preghenella pushed a commit to alicetof/alidist that referenced this pull request Apr 8, 2020

Bump CMake to 3.17 (alisw#2157)

dfeba7c

matthiasrichter mentioned this pull request Apr 21, 2020

Require CMake version 3.17 AliceO2Group/AliceO2#3396

Merged

matthiasrichter added a commit to matthiasrichter/AliceO2 that referenced this pull request Apr 21, 2020

require CMake version 3.17

d8377ca

This follows recent change in alidist alisw/alidist#2157

matthiasrichter added a commit to AliceO2Group/AliceO2 that referenced this pull request Apr 21, 2020

require CMake version 3.17

c6c7f63

This follows recent change in alidist alisw/alidist#2157

Bump CMake to 3.17 #2157

Bump CMake to 3.17 #2157

Conversation

davidrohr commented Mar 27, 2020

davidrohr commented Mar 27, 2020

MichaelLettrich commented Mar 29, 2020

MichaelLettrich Mar 29, 2020

Choose a reason for hiding this comment

MichaelLettrich Mar 29, 2020

Choose a reason for hiding this comment

davidrohr commented Mar 29, 2020

davidrohr commented Mar 29, 2020

MichaelLettrich commented Mar 29, 2020

davidrohr commented Mar 29, 2020

MichaelLettrich commented Mar 29, 2020

MichaelLettrich commented Mar 29, 2020

davidrohr commented Mar 29, 2020

MichaelLettrich commented Mar 29, 2020

davidrohr commented Mar 29, 2020

MichaelLettrich commented Mar 29, 2020

davidrohr commented Mar 29, 2020

MichaelLettrich commented Mar 29, 2020

MichaelLettrich commented Mar 30, 2020

davidrohr commented Mar 30, 2020

MichaelLettrich commented Mar 30, 2020

davidrohr commented Mar 30, 2020

MichaelLettrich commented Mar 30, 2020

MichaelLettrich commented Mar 30, 2020

davidrohr commented Mar 30, 2020

MichaelLettrich commented Mar 30, 2020

awegrzyn commented Mar 30, 2020

aphecetche commented Mar 30, 2020

MichaelLettrich commented Mar 30, 2020

awegrzyn commented Mar 30, 2020

awegrzyn commented Mar 30, 2020

awegrzyn commented Mar 30, 2020

davidrohr commented Mar 30, 2020

MichaelLettrich commented Mar 31, 2020 • edited Loading

MichaelLettrich commented Mar 31, 2020

ktf commented Mar 31, 2020

davidrohr commented Mar 31, 2020

davidrohr commented Apr 2, 2020

ktf commented Apr 2, 2020

MichaelLettrich commented Mar 31, 2020 •

edited

Loading