Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

C++ Modules support (based on Clang) #15248

Closed
davidlt opened this issue Jul 21, 2016 · 35 comments
Closed

C++ Modules support (based on Clang) #15248

davidlt opened this issue Jul 21, 2016 · 35 comments

Comments

@davidlt
Copy link
Contributor

davidlt commented Jul 21, 2016

The current very basic draft exist here: https://github.com/davidlt/cmssw/tree/cpp-modules
It can only be used with DEVEL IBs as it requires Clang pre-3.9. We will be updating Clang as needed in DEVEL IBs.

How to use it?

scram p CMSSW_8_1_DEVEL_X_2016-07-19-1100
cmsenv
git cms-merge-topic davidlt:cpp-modules
rm -rf $CMSSW_BASE/cppmodulescache
mkdir -p $CMSSW_BASE/cppmodulescache

# Modify $CMSSW_BASE/config/BuildFile.xml
vim $CMSSW_BASE/config/BuildFile.xml

# Add the following line:
#
#   <flags INCLUDE="$(LOCALTOP)/src/FWCore/GNUCppStandardLibrary/interface"/>
#
# after:
#
#   <classpath path="+Project/LCG+SubSystem/+Package/tests+binary"/>
#

USER_CXXFLAGS="-fmodules -fmodules-cache-path=$CMSSW_BASE/cppmodulescache" scram b -v -j 1 COMPILER=llvm 2>&1 | tee b.log

# Alternative command:
USER_CXXFLAGS="-fmodules -Xclang -fmodules-local-submodule-visibility -fmodules-cache-path=$CMSSW_BASE/cppmodulescache" scram b -v -j 1 COMPILER=llvm 2>&1 | tee b.log
@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 21, 2016

A new Issue was created by @davidlt .

@davidlange6, @smuzaffar, @davidlt, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@davidlt
Copy link
Contributor Author

davidlt commented Jul 21, 2016

@vgvassilev: iostream.h -> iostream

@davidlt
Copy link
Contributor Author

davidlt commented Jul 21, 2016

@vgvassilev if we use -Xclang -fmodules-local-submodule-visibility we get:

>> Compiling  /mnt/build/davidlt/cppmodules/CMSSW_8_1_DEVEL_X_2016-07-17-2300/src/FWCore/MessageService/bin/Standalone.cpp
/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/llvm/3.8.99/bin/clang++ -c -DGNU_GCC -D_GNU_SOURCE -DCMSSW_GIT_HASH="CMSSW_8_1_DEVEL_X_2016-07-17-2300" -DPROJECT_NAME="CMSSW" -DPROJECT_VERSION="CMSSW_8_1_DEVEL_X_2016-07-17-2300" -DBOOST_SPIRIT_THREADSAFE -DPHOENIX_THREADSAFE -I/mnt/build/davidlt/cppmodules/CMSSW_8_1_DEVEL_X_2016-07-17-2300/src/FWCore/GNUCppStandardLibrary/interface -I/mnt/build/davidlt/cppmodules/CMSSW_8_1_DEVEL_X_2016-07-17-2300/src -I/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-07-17-2300/src -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/lcg/root/6.06.04-agcabg/include -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/boost/1.57.0-giojec/include -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/pcre/8.37/include -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/bz2lib/1.0.6/include -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/libuuid/2.22.2/include -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/python/2.7.11-giojec/include/python2.7 -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/tbb/44_20151115oss/include -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/zlib/1.2.8/include -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/tinyxml/2.5.3/include -O2 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++14 -ftree-vectorize -Wstrict-overflow -Werror=array-bounds -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Wa,--compress-debug-sections -msse3 -felide-constructors -fmessage-length=0 -Wall -Wno-long-long -Wreturn-type -Wunused -Wparentheses -Wno-deprecated -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=uninitialized -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-c99-extensions -Wno-c++11-narrowing -D__STRICT_ANSI__ -Wno-unused-private-field -Wno-unknown-pragmas -Wno-unused-command-line-argument -ftemplate-depth=512 -Wno-error=potentially-evaluated-expression -DBOOST_DISABLE_ASSERTS -Wno-error=unused-variable -fmodules -fmodules-cache-path=/build/davidlt/cppmodules/CMSSW_8_1_DEVEL_X_2016-07-17-2300/cppmodulescache -fPIC -MMD -MF tmp/slc6_amd64_gcc530/src/FWCore/MessageService/bin/Standalone/Standalone.d /mnt/build/davidlt/cppmodules/CMSSW_8_1_DEVEL_X_2016-07-17-2300/src/FWCore/MessageService/bin/Standalone.cpp -o tmp/slc6_amd64_gcc530/src/FWCore/MessageService/bin/Standalone/Standalone.o
In file included from /mnt/build/davidlt/cppmodules/CMSSW_8_1_DEVEL_X_2016-07-17-2300/src/FWCore/MessageService/bin/Standalone.cpp:21:
In file included from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-07-17-2300/src/FWCore/PluginManager/interface/PresenceFactory.h:5:
In file included from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-07-17-2300/src/FWCore/PluginManager/interface/PluginFactory.h:26:
In file included from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-07-17-2300/src/FWCore/PluginManager/interface/PluginFactoryBase.h:32:
In file included from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_DEVEL_X_2016-07-17-2300/src/FWCore/PluginManager/interface/PluginInfo.h:22:
In file included from /cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/boost/1.57.0-giojec/include/boost/filesystem/path.hpp:29:
In file included from /cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/boost/1.57.0-giojec/include/boost/shared_ptr.hpp:17:
In file included from /cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/boost/1.57.0-giojec/include/boost/smart_ptr/shared_ptr.hpp:28:
/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/boost/1.57.0-giojec/include/boost/smart_ptr/detail/shared_count.hpp:417:33: error: declaration of 'unique_ptr' must be imported from module 'GNU_CPP_STL.FWCore/GNUCppStandardLibrary/interface/condition_variable' before it is required
    explicit shared_count( std::unique_ptr<Y, D> & r ): pi_( 0 )
                                ^
/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/gcc/5.3.0/lib/gcc/x86_64-pc-linux-gnu/5.3.0/../../../../include/c++/5.3.0/bits/unique_ptr.h:129:11: note: previous declaration is here
    class unique_ptr
          ^
1 error generated.
config/SCRAM/GMake/Makefile.rules:1917: recipe for target 'tmp/slc6_amd64_gcc530/src/FWCore/MessageService/bin/Standalone/Standalone.o' failed
gmake: *** [tmp/slc6_amd64_gcc530/src/FWCore/MessageService/bin/Standalone/Standalone.o] Error 1

@davidlt
Copy link
Contributor Author

davidlt commented Jul 21, 2016

Correction, same happens even if I don't use -Xclang -fmodules-local-submodule-visibility. I have seen similar happening with std::shared_ptr and others. We also have seen issues with redefinitions of every type in pthread.h which I couldn't reproduce again for now.

@davidlt
Copy link
Contributor Author

davidlt commented Jul 25, 2016

You can find care binary here http://davidlt.web.cern.ch/davidlt/vault/proot/care

I have used it with the failing command:

./care -o partial_rootfs.tar.gz /cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/llvm/3.8.99/bin/clang++ -c -DGNU_GCC -D_GNU_SOURCE -DCMSSW_GIT_HASH="CMSSW_8_1_DEVEL_X_2016-07-19-1100" -DPROJECT_NAME="CMSSW" -DPROJECT_VERSION="CMSSW_8_1_DEVEL_X_2016-07-19-1100" -DBOOST_SPIRIT_THREADSAFE -DPHOENIX_THREADSAFE -I/mnt/build/davidlt/cppmodules/CMSSW_8_1_DEVEL_X_2016-07-19-1100/src/FWCore/GNUCppStandardLibrary/interface -I/mnt/build/davidlt/cppmodules/CMSSW_8_1_DEVEL_X_2016-07-19-1100/src -I/mnt/build/davidlt/cppmodules/CMSSW_8_1_DEVEL_X_2016-07-19-1100/poison -I/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-07-19-1100/src -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/lcg/root/6.06.04-agcabg/include -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/boost/1.57.0-giojec/include -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/pcre/8.37/include -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/bz2lib/1.0.6/include -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/libuuid/2.22.2/include -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/python/2.7.11-giojec/include/python2.7 -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/tbb/44_20151115oss/include -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/zlib/1.2.8/include -I/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/tinyxml/2.5.3/include -O2 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++14 -ftree-vectorize -Wstrict-overflow -Werror=array-bounds -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Wa,--compress-debug-sections -msse3 -felide-constructors -fmessage-length=0 -Wall -Wno-long-long -Wreturn-type -Wunused -Wparentheses -Wno-deprecated -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=uninitialized -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-c99-extensions -Wno-c++11-narrowing -D__STRICT_ANSI__ -Wno-unused-private-field -Wno-unknown-pragmas -Wno-unused-command-line-argument -ftemplate-depth=512 -Wno-error=potentially-evaluated-expression -DBOOST_DISABLE_ASSERTS -Wno-error=unused-variable -fmodules -fmodules-cache-path=/mnt/build/davidlt/cppmodules/CMSSW_8_1_DEVEL_X_2016-07-19-1100/cppmodulescache -fPIC -MMD -MF tmp/slc6_amd64_gcc530/src/FWCore/MessageService/bin/Standalone/Standalone.d /mnt/build/davidlt/cppmodules/CMSSW_8_1_DEVEL_X_2016-07-19-1100/src/FWCore/MessageService/bin/Standalone.cpp -o tmp/slc6_amd64_gcc530/src/FWCore/MessageService/bin/Standalone/Standalone.o

Output

care info: concealed path: $HOME /afs/cern.ch/user/d/davidlt
care info: concealed path: /tmp
care info: revealed path: $PWD /mnt/build/davidlt/cppmodules/CMSSW_8_1_DEVEL_X_2016-07-19-1100
care info: revealed path: /cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/llvm/3.8.99/bin/clang-3.9
care info: ----------------------------------------------------------------------
In file included from /mnt/build/davidlt/cppmodules/CMSSW_8_1_DEVEL_X_2016-07-19-1100/src/FWCore/MessageService/bin/Standalone.cpp:21:
In file included from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-07-19-1100/src/FWCore/PluginManager/interface/PresenceFactory.h:5:
In file included from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-07-19-1100/src/FWCore/PluginManager/interface/PluginFactory.h:26:
In file included from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-07-19-1100/src/FWCore/PluginManager/interface/PluginFactoryBase.h:32:
In file included from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_DEVEL_X_2016-07-19-1100/src/FWCore/PluginManager/interface/PluginInfo.h:22:
In file included from /cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/boost/1.57.0-giojec/include/boost/filesystem/path.hpp:29:
In file included from /cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/boost/1.57.0-giojec/include/boost/shared_ptr.hpp:17:
In file included from /cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/boost/1.57.0-giojec/include/boost/smart_ptr/shared_ptr.hpp:28:
/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/boost/1.57.0-giojec/include/boost/smart_ptr/detail/shared_count.hpp:417:33: error: declaration of 'unique_ptr' must be imported from module 'GNU_CPP_STL.FWCore/GNUCppStandardLibrary/interface/condition_variable' before it is required
    explicit shared_count( std::unique_ptr<Y, D> & r ): pi_( 0 )
                                ^
/cvmfs/cms-ib.cern.ch/2016-30/slc6_amd64_gcc530/external/gcc/5.3.0/lib/gcc/x86_64-pc-linux-gnu/5.3.0/../../../../include/c++/5.3.0/bits/unique_ptr.h:129:11: note: previous declaration is here
    class unique_ptr
          ^
1 error generated.
care info: ----------------------------------------------------------------------
care info: Hints:
care info:   - search for "conceal" in `care -h` if the execution didn't go as expected.
care info:   - run `care -x partial_rootfs.tar.gz` to extract the output archive correctly.

Got 37M partial rootfs tarball with 1278 files touched in this execution. 483 headers are found in this tarball.

You can grab it from: http://davidlt.web.cern.ch/davidlt/vault/cppmodules_partial_rootfs.tar.gz

@vgvassilev
Copy link
Contributor

@davidlt yes I can reproduce the issue outside now. Thanks a lot!

@vgvassilev
Copy link
Contributor

Some debugging aids (suggested by Richard Smith):

  • -fdiagnostics-show-note-include-stack will tell you which module a note comes from
  • #pragma clang __debug dump X allows you to produce an AST dump from within a source file, so you can see which modules declare a given name
  • #pragma clang __debug macro M allows you to dump a macro, which can be useful to see which module(s) export visible include guards for a header

If a name is not visible in a modules build but is visible in a non-modules build, i usually find that's due to one of two things

  • some part of the machinery that provides it depends on macro definitions leaking into a modular header from outside, or
  • there is an include cycle involving a modular header and a non-modular header

@vgvassilev
Copy link
Contributor

A flavor of the unique_ptr issue is in this bug report: https://llvm.org/bugs/show_bug.cgi?id=28794

@vgvassilev
Copy link
Contributor

It may be worth to give another try as the blocking issue was resolved.

@Teemperor
Copy link
Contributor

We now have a list of the cyclic dependencies in the headers of CMS master: https://teemperor.de/pub/cms/cycle_report.cycle.html (Should be updated every few seconds). It also contains some debugging help on how to break those cyclic dependencies.

You can run this program to get the same output for your local changes: https://github.com/Teemperor/circle-break

This was referenced Jun 25, 2020
@smuzaffar
Copy link
Contributor

closing this issue. We have CXXMODULE IBs dedicated for this

drbenmorgan pushed a commit to drbenmorgan/CLHEP that referenced this issue Jan 21, 2022
The C++ modules feature as described in https://clang.llvm.org/docs/Modules.html
allow producing a binary header representation to avoid redundant header
reparsing.

This feature is used in ROOT's dictionary system since ROOT v6.20:
https://github.com/root-project/root/blob/master/README/README.CXXMODULES.md

CMSSW and other experiment migrate their dictionaries to use the provided by
ROOT C++ modules support: cms-sw/cmssw#15248

Dictionaries which transiently include clhep can be further optimized by
building a separate module for CLHEP which this MR aims for.

The current patch introduces a module.modulemap file containing a mapping between
a binary artifact (a module or a pcm file) and a set of header files. The C++
modules are more picky on translation unit encapsulation and thus require all
headers which a translation unit uses to be included. In addition to the
missing include we outline a few virtual destructors to avoid pollution of .o
files with weak virtual tables.

This patch enables builds with modules via rootcling/genreflex/rootcint and
enables compile-time module builds if configured like:
cmake -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_CXX_FLAGS="-fmodules -Xclang -fmodules-local-submodule-visibility" ../clhep/

The module.modulemap file is easy to maintain as it globs for the well-behaved
header files and enumerates already the ones needed special treatment.

Once merged we will backport this to cmssw.

Conflicts:
	Matrix/src/DiagMatrix.cc
	Matrix/src/GenMatrix.cc
	Matrix/src/Matrix.cc
	Matrix/src/SymMatrix.cc
	Matrix/src/Vector.cc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants