[Draft] Improve C++20 modules support #2623

Arthapz · 2022-07-31T13:26:16Z

Headerunits doesn't work on Clang <= 15 (which is currently in RC version)
GCC have some issues with header units, but it's compiler bug related
MSVC work pretty well in all my test cases

this PR introduce add_modulefiles, the objective of this API is to offer in the future, module installation like header installation

a lot of improvement can be made on this code as i'm not fully comfortable with Lua and XMake internals

waruqi · 2022-07-31T13:36:18Z

Thanks, I have some other things I'm working on, maybe I need to spend some time to review and test this patch carefully later, please wait for some time, maybe a few days.

Arthapz · 2022-07-31T13:39:58Z

No problem :3

waruqi · 2022-08-01T02:08:40Z

I try it, but it doesn't seem to be compatible with the lower versions vs

Z:\personal\xmake\tests\projects\c++\modules\hello>xmake -rv
[  0%]: generating.cxx.module.deps src\main.cpp
checking for flags (-std:c++20) ... ok
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.2
9.30133\bin\HostX64\x64\cl.exe /EHsc -nologo -std:c++20 /experimental:module /if
cSearchDir build\.gens\hello\windows\x64\release\rules\modules\cache /ifcSearchD
ir build\.gens\hello\windows\x64\release\stlmodules\cache /stdIfcDir C:\PROGRA~2
\MICROS~1\2019\COMMUN~1\VC\Tools\MSVC\1429~1.301\ifc\x64 /TP /scanDependencies b
uild\.gens\hello\windows\x64\release\rules\modules\cache\src\main.cpp.json src\m
ain.cpp /Fobuild\.objs\hello\windows\x64\release\src\main.cpp.obj
cl : Command line warning D9002 : ignoring unknown option '/scanDependencies'
main.cpp.json
c1xx: fatal error C1083: Cannot open source file: 'build\.gens\hello\windows\x64
\release\rules\modules\cache\src\main.cpp.json': No such file or directory
main.cpp
Generating Code...
error: execv(C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\To
ols\MSVC\14.29.30133\bin\HostX64\x64\cl.exe /EHsc -nologo -std:c++20 /experiment
al:module /ifcSearchDir build\.gens\hello\windows\x64\release\rules\modules\cach
e /ifcSearchDir build\.gens\hello\windows\x64\release\stlmodules\cache /stdIfcDi
r C:\PROGRA~2\MICROS~1\2019\COMMUN~1\VC\Tools\MSVC\1429~1.301\ifc\x64 /TP /scanD
ependencies build\.gens\hello\windows\x64\release\rules\modules\cache\src\main.c
pp.json src\main.cpp /Fobuild\.objs\hello\windows\x64\release\src\main.cpp.obj)
failed(2)

waruqi · 2022-08-01T02:18:30Z

this PR introduce add_modulefiles, the objective of this API is to offer in the future, module installation like header installation

We do not need to introduce new interfaces, which would break the projects of existing users.

Also, we can extract files from add_files("*.mpp") to install them in the after_install of the custom rule as well.

rule("c++.build.modules")
    set_extensions(".mpp", ".mxx", ".cppm", ".ixx")
    after_install(function (target)
        local sourcebatch = target:sourcebatches()["c++.build.modules"]
        -- TODO install *.mpp
        -- sourcebatch.sourcefiles are both
    end

xmake/xmake/rules/c++/modules/xmake.lua

Lines 22 to 23 in e45d940

    
           rule("c++.build.modules") 
        
               set_extensions(".mpp", ".mxx", ".cppm", ".ixx")

Support for cppmodules should be done in the rules as far as possible.

xmake/rules/qt/qmltyperegistrar/xmake.lua

xmake/core/project/target.lua

xmake/rules/c++/modules/build_modules/clang.lua

waruqi · 2022-08-01T02:26:43Z

This patch changes the implementation of a lot of c++ modules, but in many places it is not fully compatible with the current xmake architecture.

I may refer to its implementation to improve c++ modules, but may not be able to merge it directly.

Also, I haven't been able to fully understand some of the improvements in this patches, can you briefly explain them?

waruqi · 2022-08-01T02:32:01Z

Also, I see that you have added support for the std module, can you add some relevant test examples to the tests directory?
and other examples of tests related to new features such as headerunit, ...

Arthapz · 2022-08-01T17:08:41Z

This patch changes the implementation of a lot of c++ modules, but in many places it is not fully compatible with the current xmake architecture.

I may refer to its implementation to improve c++ modules, but may not be able to merge it directly.

Also, I haven't been able to fully understand some of the improvements in this patches, can you briefly explain them?

So, for building modules and header units, we need 3 passes

the first pass we will generate all header units, to have knowledge about them, the C++ standard made a common format to get dependencies knowledge https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p1689r5.html
on MSVC this is supported with /scanDependencies
on clang there is no support a this point of time llvm/llvm-project#51792
on GCC there is a patch (-fdep-format=trtbd) but this is not supported on trunk GCC (didn't found an issue for this)

these flags generate a json file with a list of artefact that are generated by a module, and their dependencies

next we can parse this to generate header units
on MSVC /exportHeader /headerName:{quote/:angle}
on Clang -emit-module -x c++-{user/system}-header (but supported only by LLVM >= 15)
on GCC -x c++-{user/system}-header
stl headers need a special computing as they could be built once for all the project that use the same compiler version

on the second pass we need to sort the modules by dependencies and generate their BMI, but to this right we need to indicate to the compiler how to get module dependencies

on MSVC /reference for modules and /headerUnit:{angle/quote} for header units
on Clang -fmodule-file= for module and header units
on GCC we need to write a mapper file which is a text file with each line as
"<PATH_TO_HEADER_UNIT/MODULE_NAME> <PATH_BMI_FILE>"
and -fmodule-mapper= to use this mapper file

then on the last pass we can build cpp files

waruqi · 2022-08-01T23:23:26Z

Thanks.

This reverts commit 8d8127b.

Arthapz · 2022-08-03T00:39:35Z

Now it use add_files and support older version of VS and Clang, and i added headerunit tests (but they will file on Clang < 15, header units is supported only by clang >= 15)

waruqi · 2022-08-03T01:11:25Z

I have switched this patches to the cxxmodules branch and I will try to merge it to this branch first.

This is because I need to make further improvements to it, such as code style, internal structure improvements, etc.

xmake/rules/c++/modules/xmake.lua

Arthapz · 2022-08-03T05:42:59Z

The /headerName compiler option is available starting in Visual Studio 2019 version 16.10 so what should i do ? assert when trying to use header units before 16.10 ?

If the module project uses headerunit, then it is possible to indicate that it is not supported by assert.

if some module projects do not use headerunit, xmake should be allowed to continue compiling. Examples such as tests/projects/c++/modules/hello

Now I can't compile any of the previous c++ modules examples using vs2019.

Now it use add_files and support older version of VS

I tried it, it always say: does not support c++ module! for vs2019

It should be good now

waruqi · 2022-08-03T06:01:44Z

It works for vs2019 now, but when I trying stl_headerunits test, it will raise

error: @programdir\core\main.lua:280: @programdir\actions\build\main.lua:166: ...mdir\core\sandbox\modules\import\core\base\scheduler.lua:96: ...mdir\core\sandbox\modules\import\core\base\scheduler.lua:56: @programdir\modules\private\async\runjobs.lua:256: @programdir\rules\c++\modules\modules_support\msvc.lua:355: attempt to concatenate a nil value (global 'modulesflag')
stack traceback:
    [@programdir\rules\c++\modules\modules_support\msvc.lua:355]: in function 'get_exportheaderflag'
    [@programdir\rules\c++\modules\modules_support\msvc.lua:129]: in function 'generate_headerunits'
    [@programdir\rules\c++\modules\xmake.lua:117]: in function 'script'
    [@program

Arthapz · 2022-08-03T06:07:16Z

It works for vs2019 now, but when I trying stl_headerunits test, it will raise

error: @programdir\core\main.lua:280: @programdir\actions\build\main.lua:166: ...mdir\core\sandbox\modules\import\core\base\scheduler.lua:96: ...mdir\core\sandbox\modules\import\core\base\scheduler.lua:56: @programdir\modules\private\async\runjobs.lua:256: @programdir\rules\c++\modules\modules_support\msvc.lua:355: attempt to concatenate a nil value (global 'modulesflag')
stack traceback:
    [@programdir\rules\c++\modules\modules_support\msvc.lua:355]: in function 'get_exportheaderflag'
    [@programdir\rules\c++\modules\modules_support\msvc.lua:129]: in function 'generate_headerunits'
    [@programdir\rules\c++\modules\xmake.lua:117]: in function 'script'
    [@program

fixed

xmake/rules/c++/modules/modules_support/common.lua

xmake/rules/c++/modules/modules_support/gcc.lua

waruqi · 2022-08-03T06:23:11Z

It works for vs2019 now, but when I trying stl_headerunits test, it will raise

error: @programdir\core\main.lua:280: @programdir\actions\build\main.lua:166: ...mdir\core\sandbox\modules\import\core\base\scheduler.lua:96: ...mdir\core\sandbox\modules\import\core\base\scheduler.lua:56: @programdir\modules\private\async\runjobs.lua:256: @programdir\rules\c++\modules\modules_support\msvc.lua:355: attempt to concatenate a nil value (global 'modulesflag')
stack traceback:
    [@programdir\rules\c++\modules\modules_support\msvc.lua:355]: in function 'get_exportheaderflag'
    [@programdir\rules\c++\modules\modules_support\msvc.lua:129]: in function 'generate_headerunits'
    [@programdir\rules\c++\modules\xmake.lua:117]: in function 'script'
    [@program

fixed

It works now, thanks.

xmake/rules/c++/modules/modules_support/gcc.lua

waruqi · 2022-08-03T09:20:59Z

I merged this patch to cxxmodules first, then I'll improve it further and test it in detail, and format some code and naming styles.

I still need to take some time to understand the implementation and make sure there are no other issues.

If you have any other improvements, you can continue to commit them to this branch as well.

waruqi · 2022-08-03T09:23:56Z

I have opened a new pr and I will continue some reviews on this one

waruqi · 2022-08-04T13:54:54Z

xmake/rules/c++/modules/xmake.lua

+        local headerunits
+        for _, objectfile in ipairs(batch.objectfiles) do
+            for obj, m in pairs(modules) do
+                if obj == objectfile then


why not use local m = modules[objectfile] to reduce for-loop?

waruqi · 2022-08-04T15:27:55Z

xmake/rules/c++/modules/modules_support/clang.lua

+                batchcmds:show_progress(opt.progress, "${color.build.object}generating.cxx.module.bmi %s", name)
+
+                local bmifile = provide.bmi
+                table.join2(args, { "-c", "-x", "c++-module", "--precompile", provide.sourcefile, "-o", bmifile })


I don't really understand here, if there are multiple values for provides, then there seems to be some problems.

clang -c sourcefile -o bmifile -c sourcefile2 -o bmifile2 ...?

waruqi · 2022-08-04T15:29:18Z

xmake/rules/c++/modules/modules_support/clang.lua

+
+                batchcmds:add_depfiles(provide.sourcefile)
+                batchcmds:set_depmtime(os.mtime(bmifile))
+                batchcmds:set_depcache(target:dependfile(bmifile))


It will be overwritten several times

waruqi · 2022-08-04T15:29:26Z

xmake/rules/c++/modules/modules_support/clang.lua

+            batchcmds:vrunv(compinst:program(), table.join(compinst:compflags({target = target}), common_args, bmifiles, {"-c", "-o", objectfile}))
+
+            batchcmds:set_depmtime(os.mtime(objectfile))
+            batchcmds:set_depcache(target:dependfile(objectfile))


It will be overwritten several times

waruqi · 2022-08-04T15:30:13Z

xmake/rules/c++/modules/modules_support/gcc.lua

+
+                local bmifile = provide.bmi
+                if add_module_to_mapper(mapper_file, name, path.absolute(bmifile, project.directory())) then
+                    table.join2(args, { "-c", provide.sourcefile })


and here.

gcc -o objectfile -c sourcefile1 -c sourcefile2 -c sourcefile3 ...?

waruqi · 2022-08-04T15:31:49Z

xmake/rules/c++/modules/modules_support/msvc.lua

+                batchcmds:set_depmtime(os.mtime(bmifile))
+                batchcmds:set_depcache(target:dependfile(bmifile))
+
+                flag = {referenceflag, name .. "=" .. path.filename(bmifile)}


It will be overwritten several times for each provides

Arthapz added 2 commits July 31, 2022 15:21

Improve C++20 modules support

7788ec5

Update unit tests to new add_modulefiles

8d8127b

Arthapz changed the title ~~Improve C++20 modules support~~ [Draft] Improve C++20 modules support Jul 31, 2022

waruqi reviewed Aug 1, 2022

View reviewed changes

xmake/rules/qt/qmltyperegistrar/xmake.lua Show resolved Hide resolved

waruqi reviewed Aug 1, 2022

View reviewed changes

xmake/core/project/target.lua Outdated Show resolved Hide resolved

waruqi reviewed Aug 1, 2022

View reviewed changes

xmake/rules/c++/modules/build_modules/clang.lua Show resolved Hide resolved

Arthapz added 5 commits August 2, 2022 18:17

Revert "Update unit tests to new add_modulefiles"

dee01ba

This reverts commit 8d8127b.

[C++20 Modules] Remove add/remove_modulefiles

37af4ce

[C++20 Modules] Improve code

ab70577

[C++20 Modules] Improve support on MSVC Clang and GCC

92b8b66

[C++20 Modules] Add tests for headerunits

dcf5a1d

waruqi changed the base branch from dev to cxxmodules August 3, 2022 01:08