Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update Intel tools for CMSSW 10.0.x #3610

Conversation

fwyzard
Copy link
Contributor

@fwyzard fwyzard commented Nov 28, 2017

Update the Intel Compiler to version 2018.0.128
Update Intel VTune to version 2018.0.2.525261
Update Intel TBB to 2018 Update 1

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @fwyzard (Andrea Bocci) for branch IB/CMSSW_10_0_X/gcc630.

@cmsbuild, @smuzaffar, @gudrutis, @mrodozov can you please review it and eventually sign? Thanks.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.

@fwyzard
Copy link
Contributor Author

fwyzard commented Nov 28, 2017

@cmsbuild, please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Nov 28, 2017

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/24736/console

@fwyzard fwyzard changed the title Intel tools 2018.0 for CMSSW 10.0.x update Intel tools for CMSSW 10.0.x Nov 28, 2017
@fwyzard
Copy link
Contributor Author

fwyzard commented Nov 28, 2017

@cmsbuild, please test

@cmsbuild
Copy link
Contributor

The tests are being triggered in jenkins.

@cmsbuild
Copy link
Contributor

Pull request #3610 was updated.

@fwyzard fwyzard force-pushed the Intel_Tools_2018.0_for_IB/CMSSW_10_0_X/gcc630 branch from 9a62c4f to 8ebdd0f Compare November 28, 2017 23:53
@fwyzard
Copy link
Contributor Author

fwyzard commented Nov 28, 2017

@cmsbuild, please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Nov 28, 2017

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/24738/console

@cmsbuild
Copy link
Contributor

Pull request #3610 was updated.

@cmsbuild
Copy link
Contributor

@cmsbuild
Copy link
Contributor

Comparison job queued.

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3610/24738/summary.html

There are some workflows for which there are errors in the baseline:
10824.0 step 5
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 27
  • DQMHistoTests: Total histograms compared: 2605338
  • DQMHistoTests: Total failures: 1
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 2605166
  • DQMHistoTests: Total skipped: 171
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 1.1200000001 KiB( 23 files compared)
  • Checked 111 log files, 8 edm output root files, 27 DQM output files

@smuzaffar smuzaffar merged commit 33df0da into cms-sw:IB/CMSSW_10_0_X/gcc630 Nov 29, 2017
@smuzaffar
Copy link
Contributor

@fwyzard , looks like c++17 change (for gcc630) did not go well with boost serialization
https://cmssdt.cern.ch/SDT/cgi-bin/showBuildLogs.py/slc6_amd64_gcc630/www/wed/10.0-wed-23/CMSSW_10_0_X_2017-11-29-2300

/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/tmp/BUILDROOT/4d50a45c6a4a8cce5ffcb3898507a4b8/opt/cmssw/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_0_X_2017-11-29-2300/src/CondFormats/Serialization/python/condformats_serialization_generate.py --output /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/tmp/BUILDROOT/4d50a45c6a4a8cce5ffcb3898507a4b8/opt/cmssw/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_0_X_2017-11-29-2300/tmp/slc6_amd64_gcc630/src/CondFormats/ESObjects/src/CondFormatsESObjects/a/Serialization.cc --package src/CondFormats/ESObjects/ -- -I/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/tmp/BUILDROOT/4d50a45c6a4a8cce5ffcb3898507a4b8/opt/cmssw/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_0_X_2017-11-29-2300/src/CondFormats/ESObjects/src -DGNU_GCC -D_GNU_SOURCE -DCMSSW_GIT_HASH="CMSSW_10_0_X_2017-11-29-2300" -DPROJECT_NAME="CMSSW" -DPROJECT_VERSION="CMSSW_10_0_X_2017-11-29-2300" -DTBB_USE_GLIBCXX_VERSION=60300 -DBOOST_SPIRIT_THREADSAFE -DPHOENIX_THREADSAFE -I/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/tmp/BUILDROOT/4d50a45c6a4a8cce5ffcb3898507a4b8/opt/cmssw/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_0_X_2017-11-29-2300/src -I/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/slc6_amd64_gcc630/lcg/root/6.10.08-mmelna2/include -I/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/slc6_amd64_gcc630/external/pcre/8.37-oenich/include -I/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/slc6_amd64_gcc630/external/boost/1.63.0-fmblme/include -I/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/slc6_amd64_gcc630/external/bz2lib/1.0.6/include -I/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/slc6_amd64_gcc630/external/gsl/2.2.1/include -I/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/slc6_amd64_gcc630/external/libuuid/2.22.2/include -I/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/slc6_amd64_gcc630/external/tbb/2018_U1/include -I/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/slc6_amd64_gcc630/external/xz/5.2.2-oenich/include -I/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/slc6_amd64_gcc630/external/zlib-x86_64/1.2.11/include -I/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/slc6_amd64_gcc630/external/md5/1.0.0/include -I/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/slc6_amd64_gcc630/external/tinyxml/2.5.3-fmblme/include -O2 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++17 -ftree-vectorize -Wstrict-overflow -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -fno-crossjumping -msse3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wunused -Wparentheses -Wno-deprecated -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=maybe-uninitialized -Werror=strict-aliasing -Werror=narrowing -Werror=uninitialized -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -DBOOST_DISABLE_ASSERTS -Wno-error=unused-variable -fdebug-prefix-map=/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w=/opt/cmssw -fdebug-prefix-map=/build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/tmp/BUILDROOT/4d50a45c6a4a8cce5ffcb3898507a4b8/opt/cmssw=/opt/cmssw -g
[2017-11-30 00:36:59,576] ERROR: Diagnostic: [Error] invalid value 'c++17' in '-std=c++17'
[2017-11-30 00:36:59,576] ERROR:    at line 0 in None

I would sugegst to revert it to c++1z for gcc630

@davidlt
Copy link
Contributor

davidlt commented Nov 30, 2017

Yup, we need to keep c++1z because Clang version in GCC 6.3.0 does not accept c++17. IIRC, this is not the case in GCC 7 branch.

@fwyzard
Copy link
Contributor Author

fwyzard commented Nov 30, 2017

Does the serialisation code take the flags from the llvm-cxxcompiler tool ?
If so we can add

    <flags REM_CXXFLAGS="-std=c++17"/>
    <flags CXXFLAGS="-std=c++1z"/>

to llvm-cxxcompiler ?

@davidlt
Copy link
Contributor

davidlt commented Nov 30, 2017

It does a messy thing. It reads original GCC flags and then modifies them enough to make Clang happy (I was against that). Do you (or some tool) explicitly need -std=c++17? If so, we can modify boost serialization to transform -std=c++17 to -std=c++1z internally if Clang versions is bellow 5.0.0 (or something along these lines).

@fwyzard
Copy link
Contributor Author

fwyzard commented Nov 30, 2017

ICC/ICPC (Intel compiler) understands -std=c++17 but not -std=c++1z.
So the alternative would be to revert GCC to -std=c++1z and add to the ICC tool file something along the lines of

    <flags REM_CXXFLAGS="-std=c++1z"/>
    <flags CXXFLAGS="-std=c++17"/>

@smuzaffar
Copy link
Contributor

condformats_serialization_generate.py script calls scram b echo_CondFormatsPACKAGE_CXXFLAGS to get the flags. So adding it to llvm will not help.

@smuzaffar
Copy link
Contributor

@fwyzard, yes we can add it in icc toolfiles. By the way, will .cc files compiled with c++1z and .cu with c++17 work together? Can you test this and add PR?

@fwyzard
Copy link
Contributor Author

fwyzard commented Nov 30, 2017

condformats_serialization_generate.py script calls scram b echo_CondFormatsPACKAGE_CXXFLAGS to get the flags. So adding it to llvm will not help.

Is there no way to get the LLVM flags for a package, even if the compiler is GCC ?
Something like

scram b echo_CondFormatsPACKAGE_CXXFLAGS COMPILER='llvm compile'

?

@smuzaffar
Copy link
Contributor

scram b echo_CondFormatsPACKAGE_CXXFLAGS COMPILER='llvm' should be enough but we need to check with the author of condformats_serialization_generate.py

@davidlt
Copy link
Contributor

davidlt commented Nov 30, 2017

They didn't want to that originally and was already suggested to them (years ago).

@fwyzard
Copy link
Contributor Author

fwyzard commented Nov 30, 2017

@davidlt, who were the original author ? do you remember what was the objection ?

@fwyzard
Copy link
Contributor Author

fwyzard commented Nov 30, 2017

@smuzaffar actually it looks like scram explicitly passes the flags to the tool:

# CondFormat Serialization generation
define generate_CondSerialization
  @$(startlog_$(2))[ -d $(@D) ] ||  $(CMD_mkdir) -p $(@D) &&\
  $(CMD_echo) ">> Generating CondFormat Serialization code from header file $< " &&\
  $(VERB_ECHO) $(COND_SERIALIZATION_SCRIPT) --output $(LOCALTOP)/$@ --package $(dir $3) -- $(call AdjustFlags,$1,,CPPFLAGS CXXFLAGS) &&\
  (            $(COND_SERIALIZATION_SCRIPT) --output $(LOCALTOP)/$@ --package $(dir $3) -- $(call AdjustFlags,$1,,CPPFLAGS CXXFLAGS) || ($(CMD_rm) -f $@ && exit 1)) $(endlog_$(2))
endef

How would one tell scram to use the llvm flags instead ?

@smuzaffar
Copy link
Contributor

smuzaffar commented Nov 30, 2017

@fwyzard, ah right ..... if scram flags are not passed then script runs scram build echo_.....
While building, scram has set gcc as compiler so it is not easy to change the compiler and get flags. We either need to define a flag to add/drop flags for boost serialization or we go with your trick and remove/add flag for icc toolfile.

@fwyzard
Copy link
Contributor Author

fwyzard commented Nov 30, 2017 via email

@smuzaffar
Copy link
Contributor

one can (never tested , might not work due to some internal cached flags) but it would be very expensive call.

@fwyzard fwyzard deleted the Intel_Tools_2018.0_for_IB/CMSSW_10_0_X/gcc630 branch February 16, 2018 17:27
@smuzaffar
Copy link
Contributor

FYI, @fwyzard , new build rules now allow to pass llvm-cxxcompiler flags ( cms-sw/cmssw-config@3417ef6#diff-0a86a53bf24863421e1a627d229895e1R675) to condformats_serialization_generate.py

@fwyzard
Copy link
Contributor Author

fwyzard commented Apr 10, 2018

@smuzaffar nice.
Does it mean from now on we don't need to customise the flags in condformats_serialization_generate.py , and we can just rely on the LLVM flags ?

@smuzaffar
Copy link
Contributor

@fwyzard , yes that is correct. Now condformats_serialization_generate.py should receive only LLVM flags.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants