Add support of importing modules from packages

[Arthapz]

This is a Proof of concepts, may need a lot of optimisation / refinements

the module distribution files are based on https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2473r1.pdf

[waruqi]

Does it work now? Can you provide an example project with package? and how to test it?

[Arthapz]

this PR works on the small tests i did, but it need furthermore test to ensures it works with more complex project
and for unitary tests i don't know how to test it, maybe add a fake test package to xrepo (or a private repository) that contains modules ?

[waruqi]

this PR works on the small tests i did, but it need furthermore test to ensures it works with more complex project and for unitary tests i don't know how to test it, maybe add a fake test package to xrepo (or a private repository) that contains modules ?

Add a test project and define a local package with modules in this project directly.

then call package.tools.xmake to install it and use add_requires/add_packages to use it.

see https://xmake.io/#/package/local_3rd_source_library

[waruqi]

it works for me

xmake install -y -o /mnt/xmake/tests/projects/c++/modules/packages/build/.packages/b/bar/latest/4e0143c97b65425b855ad5fd03038b6a -P /mnt/xmake/tests/projects/c++/modules/packages/my-repo/packages/b/bar/src --diagnosis --verbose
TEST1.  <--------
TEST2
installing bar ..
installing bar to /mnt/xmake/tests/projects/c++/modules/packages/build/.packages/b/bar/latest/4e0143c97b65425b855ad5fd03038b6a ..
> copy ../../../../../build/.packages/b/bar/latest/cache/build_4e0143c9/linux/x86_64/release/libbar.a to /mnt/xmake/tests/projects/c++/modules/packages/build/.packages/b/bar/latest/4e0143c97b65425b855ad5fd03038b6a/lib
> copy bar.mpp to /mnt/xmake/tests/projects/c++/modules/packages/build/.packages/b/bar/latest/4e0143c97b65425b855ad5fd03038b6a/modules/bar/bar.mpp
install ok!

But it does not build project for gcc.

/usr/bin/gcc -c -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -isystem /mnt/xmake/tests/projects/c++/modules/packages/build/.packages/f/foo/latest/4e0143c97b65425b855ad5fd03038b6a/include -isystem /mnt/xmake/tests/projects/c++/modules/packages/build/.packages/b/bar/latest/4e0143c97b65425b855ad5fd03038b6a/include -fmodules-ts -fmodule-mapper=build/packages/mapper.txt -D_GLIBCXX_USE_CXX11_ABI=0 -DNDEBUG -o build/.objs/packages/linux/x86_64/release/src/main.cpp.o src/main.cpp
checking for flags (-MMD -MF) ... ok
checking for flags (-fdiagnostics-color=always) ... ok
error: src/main.cpp:1:1: error: unknown Compiled Module Interface: no such module
    1 | import foo;
      | ^~~~~~
src/main.cpp:2:1: error: unknown Compiled Module Interface: no such module
    2 | import bar;
      | ^~~~~~
In module imported at src/main.cpp:1:1:
foo: error: failed to read compiled module: Unknown CMI mapping
foo: note: imports must be built before being imported
foo: fatal error: returning to the gate for a mechanical issue
compilation terminated.

[Arthapz]

it works for me

xmake install -y -o /mnt/xmake/tests/projects/c++/modules/packages/build/.packages/b/bar/latest/4e0143c97b65425b855ad5fd03038b6a -P /mnt/xmake/tests/projects/c++/modules/packages/my-repo/packages/b/bar/src --diagnosis --verbose
TEST1.  <--------
TEST2
installing bar ..
installing bar to /mnt/xmake/tests/projects/c++/modules/packages/build/.packages/b/bar/latest/4e0143c97b65425b855ad5fd03038b6a ..
> copy ../../../../../build/.packages/b/bar/latest/cache/build_4e0143c9/linux/x86_64/release/libbar.a to /mnt/xmake/tests/projects/c++/modules/packages/build/.packages/b/bar/latest/4e0143c97b65425b855ad5fd03038b6a/lib
> copy bar.mpp to /mnt/xmake/tests/projects/c++/modules/packages/build/.packages/b/bar/latest/4e0143c97b65425b855ad5fd03038b6a/modules/bar/bar.mpp
install ok!

But it does not build project for gcc.

/usr/bin/gcc -c -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -isystem /mnt/xmake/tests/projects/c++/modules/packages/build/.packages/f/foo/latest/4e0143c97b65425b855ad5fd03038b6a/include -isystem /mnt/xmake/tests/projects/c++/modules/packages/build/.packages/b/bar/latest/4e0143c97b65425b855ad5fd03038b6a/include -fmodules-ts -fmodule-mapper=build/packages/mapper.txt -D_GLIBCXX_USE_CXX11_ABI=0 -DNDEBUG -o build/.objs/packages/linux/x86_64/release/src/main.cpp.o src/main.cpp
checking for flags (-MMD -MF) ... ok
checking for flags (-fdiagnostics-color=always) ... ok
error: src/main.cpp:1:1: error: unknown Compiled Module Interface: no such module
    1 | import foo;
      | ^~~~~~
src/main.cpp:2:1: error: unknown Compiled Module Interface: no such module
    2 | import bar;
      | ^~~~~~
In module imported at src/main.cpp:1:1:
foo: error: failed to read compiled module: Unknown CMI mapping
foo: note: imports must be built before being imported
foo: fatal error: returning to the gate for a mechanical issue
compilation terminated.

yes i spoke too fast didn't updated the local XMake files after editing the repo :D

[Arthapz]

but any idea why the package test build when running xmake -vD --yes but not when running xmake --yes on msvc ?

xmake lua -vD tests/run.lua projects/c++/modules/packages                                                                                                        10:52:54  
running given lua script file tests\run.lua with args:
dump from _print_vlog @@programdir\plugins\lua\main.lua:55
"projects/c++/modules/packages"
> 1 test(s) found
>     C:\Users\arthu\development\repos\xmake\tests\projects\c++\modules\packages\test.lua
running given lua script file tests\runner.lua with args:
dump from _print_vlog @@programdir\plugins\lua\main.lua:55
"C:\Users\arthu\development\repos\xmake\tests\projects\c++\modules\packages\test.lua"
>> testing tests\projects\c++\modules\packages ...
>>     running test_main ...
checking for platform ... windows
checking for architecture ... x64
checking for Microsoft Visual Studio (x64) version ... 2022
checking for Microsoft C/C++ Compiler (x64) version ... 19.35.32124
  => install bar latest .. failed      

[  0%]: generating.module.deps bar.cpp
[  0%]: generating.module.deps bar.mpp
error: compiler(msvc): does not support c++ module!
if you want to get more verbose errors, please see:
  -> C:\Users\arthu\development\repos\xmake\tests\projects\c++\modules\packages\build\.packages\b\bar\latest\cache\installdir.failed\logs\install.txt
error: install failed!
>>     test failed: @programdir\core\sandbox\modules\os.lua:331: exec(xmake f -c --yes) failed(-1)
stack traceback:
    [C]: in function 'error'
    [@programdir\core\base\os.lua:897]:
    [@programdir\core\sandbox\modules\os.lua:331]: in function 'exec'
    [...ts\projects\c++\modules\packages\..\test_headerunits.lua:18]:

>>       function test_main tests\projects\c++\modules\test_headerunits.lua:14
error: @programdir\core\main.lua:280: .\tests\test_utils\print_error.lua:46: aborting because of unhandled error ...
stack traceback:
    [C]: in function 'error'
    [@programdir\core\base\os.lua:897]:
    [.\tests\test_utils\print_error.lua:46]: in function 'print_error'
    [.\tests\runner.lua:55]: in function 'catch'
    [@programdir\core\sandbox\modules\try.lua:123]: in function 'try'
    [.\tests\runner.lua:44]:
    [@programdir\plugins\lua\main.lua:123]:
    [C]: in function 'xpcall'
    [@programdir\core\base\utils.lua:280]:
    [@programdir\core\base\task.lua:501]: in function 'run'
    [@programdir\core\main.lua:278]: in function 'cotask'
    [@programdir\core\base\scheduler.lua:404]:

stack traceback:
        [C]: in function 'error'
        @programdir\core\base\os.lua:897: in function 'base/os.raiselevel'
        (...tail calls...)
        @programdir\core\main.lua:280: in upvalue 'cotask'
        @programdir\core\base\scheduler.lua:404: in function <@programdir\core\base\scheduler.lua:397>
error: @programdir\core\main.lua:280: @programdir\core\sandbox\modules\os.lua:372: execv(xmake lua -v -D tests\runner.lua C:\Users\arthu\development\repos\xmake\tests\projects\c++\modules\packages\test.lua) failed(-1)
stack traceback:
    [C]: in function 'error'
    [@programdir\core\base\os.lua:897]:
    [@programdir\core\sandbox\modules\os.lua:372]: in function 'execv'
    [.\tests\run.lua:14]: in function '_run_test'
    [.\tests\run.lua:52]:
    [@programdir\plugins\lua\main.lua:123]:
    [C]: in function 'xpcall'
    [@programdir\core\base\utils.lua:280]:
    [@programdir\core\base\task.lua:501]: in function 'run'
    [@programdir\core\main.lua:278]: in function 'cotask'
    [@programdir\core\base\scheduler.lua:404]:

stack traceback:
        [C]: in function 'error'
        @programdir\core\base\os.lua:897: in function 'base/os.raiselevel'
        (...tail calls...)
        @programdir\core\main.lua:280: in upvalue 'cotask'
        @programdir\core\base\scheduler.lua:404: in function <@programdir\core\base\scheduler.lua:397>

[Arthapz]

Ok it's this function that fail on msvc without -vD (when XMake is trying to build a package)

function get_ifcoutputflag(target)
    local ifcoutputflag = _g.ifcoutputflag
    if ifcoutputflag == nil then
        local compinst = target:compiler("cxx")
        if compinst:has_flags("-ifcOutput", "cxxflags", {flagskey = "cl_ifc_output"})  then
            ifcoutputflag = "-ifcOutput"
        end
        assert(ifcoutputflag, "compiler(msvc): does not support c++ module!")
        _g.ifcoutputflag = ifcoutputflag or false
    end
    return ifcoutputflag or nil
end

[waruqi]

you can pass -vD in test_headerunits.lua

to see verbose checkflags logs

[Arthapz]

when adding -vD the test pass :/ i don't know how to debug this

[waruqi]

when adding -vD the test pass :/ i don't know how to debug this

I don't know, I need some time to debug it.

maybe need debug
https://github.com/xmake-io/xmake/blob/master/xmake/modules/detect/tools/cl/has_flags.lua
https://github.com/xmake-io/xmake/blob/master/xmake/modules/lib/detect/has_flags.lua

[waruqi]

Ok it's this function that fail on msvc without -vD (when XMake is trying to build a package)

function get_ifcoutputflag(target)
    local ifcoutputflag = _g.ifcoutputflag
    if ifcoutputflag == nil then
        local compinst = target:compiler("cxx")
        if compinst:has_flags("-ifcOutput", "cxxflags", {flagskey = "cl_ifc_output"})  then
            ifcoutputflag = "-ifcOutput"
        end
        assert(ifcoutputflag, "compiler(msvc): does not support c++ module!")
        _g.ifcoutputflag = ifcoutputflag or false
    end
    return ifcoutputflag or nil
end

Because of some incorrect flags detection (missing input arguments), when calling cl /ifcOutput -Foxxx.obj test.cpp to test for flags, -Foxxx.obj is incorrectly used as an input argument to /ifcOutput. cl generates xxx.obj to the default path. When there are multiple parallel tasks, access to this obj file is taken up and the test fails.

Some other msvc flags detections have the same problem.

[waruqi]

It works now, I have fixed it.

[waruqi]

I tested for gcc/clang on my machine, it works now.

[waruqi]

This is a Proof of concepts, may need a lot of optimisation / refinements

the module distribution files are based on https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2473r1.pdf

I sent this pr (experimental support for c++ modules distribution) to sg15@lists.isocpp.org and got some feedback.

This is a great first step. Thanks for pushing this.

I should note, however, that the proposal on that paper didn't reach consensus.

The approach that did reach consensus, later, was the one proposed in https://wg21.link/P2577R2 with the specific convention for the translation in https://wg21.link/P2701R0

Those have already been incorporated into the draft modules experience TR, which you can see the source in https://github.com/cplusplus/modules-ecosystem-tr/blob/master/appendix/discovery-in-prebuilt-library-releases.tex and a recent enough rendering on https://github.com/cplusplus/modules-ecosystem-tr/files/10071184/iso_cpp_modules_ecosystem_technical_report.pdf

Note that we haven't yet reached a consensus on the suffix for the file name containing the metadata, nor on the format of the metadata file itself.

Both still require additional papers and experiments.