Improve and optimize build speed #589

waruqi · 2019-10-08T15:31:54Z

waruqi · 2019-10-09T01:02:55Z

默认的jobs数我改进过了，根据当前pc的cpu core来计算

default jobs = ceil(cpu_core_num * 3 / 2)

通常4个core机子，默认6 jobs，如果在编译服务器上，core较多，也能充分利用，在虚拟机里面如果只有一个core，就只开2jobs，避免cpu被打的太满，导致影响正常操作

不过我仔细看了下，目前的多任务调度策略还不能即时打满max jobs，这块还有待改进，预计还有不少提升空间

waruqi · 2020-03-01T06:48:31Z

跟ninja对比了下，目前看效率差距不大，主要是构建过程中xmake自身的一些损耗，通过执行profile分析，一些主要瓶颈点：

os.exec执行过程中os.argw内部冗余的os.match操作，导致不停去做无谓的文件遍历
编译每个文件，不管是否使用，都会自动解析一遍depfile，然后按照内部格式，重新序列化存储到.d，相当于比ninja多做了一倍的lua序列化、反序列化操作和io存储，而且目前xmake的序列化还是有点慢。

目前的进展：

问题1，我已经优化掉了，提升还是很明显的
问题2，还是要重构下depfile处理上，延迟到实际使用的时候才去解析，这样就不用再额外做序列化和反序列化

另外进程调度这块，2.3.1版本已经重构过了，利用率会高不少，后期再看看是否还有优化的余地。

waruqi · 2020-03-01T09:57:52Z

depfiles的序列化存储我去掉了，延迟到实际加载的时候去解析，对于rebuild效果还是比较明显，构建整个tbox也就比ninja慢了1-2s，之后在看看优化下parse deps，我看ninja对这块还是做了不少优化的。

ninja的deps解析：https://github.com/ninja-build/ninja/blob/master/src/depfile_parser.cc
https://github.com/ninja-build/ninja/blob/master/src/clparser.cc

waruqi · 2020-03-02T02:26:41Z

目前xmake的parse_deps太慢，我优化了下，通过减少split和冗余的trim，path转换也改进了下，提升了30%的效率，并且string.split对于plain text通过native实现优化了下，相比原始版本提升20-30%

parse_deps里面的gsub替换还有改进空间，这块我看目前不支持plain text替换，而xmake里面还是有一大半gsub仅仅只需要单纯的plain text替换，所以也可以通过native话实现个优化版本，比如提供string.replace(str, oldstr, newstr, {plain = true})，后面我实现下

另外，rebuild效率对比下来，mac/win下提升了15-20%，还有不少改进空间

waruqi · 2020-03-02T13:35:45Z

拿tbox库测了下，-j1构建耗时基本跟ninja持平，差的200ms主要是进度信息显示比ninja多导致。

-j12构建，比ninja慢了1s左右

waruqi · 2020-03-04T03:23:31Z

发现多任务进程调度这块，之前的调度算法有问题，没法流畅的细粒度控制流水线，通过改进调度器后，并行构建速度也基本快跟ninja持平了，还剩下的一些优化点：

目前target间还是串行的，link阶段还没发并行处理，可以考虑将所有target的构建jobs统一化，所有编译任务优先并行处理，所有link任务统一最后处理。

waruqi · 2020-03-14T04:17:21Z

build_jobs分支我重构了整个构建系统，完全的批处理job构建，实现跨target间更加细粒度并行化，无依赖的link可以同时并行执行，多个target间无依赖的源文件也可以同时并行构建

目前实测，构建速度已经跟ninja持平，对于target较多的项目，构建速度优化极为明显，有些项目测试还比ninja快一点

不过由于这次改动较大的，虽然对用户层接口理论上做到了兼容，但是并行构建的行为差异，是否还会带来一些其他兼容性问题，还有待详测

@OpportunityLiu @fasiondog @codehz @orzuk-lmj 如果有时间的话，可以一起帮忙测试下你们现有工程的构建是否ok，如果没啥问题，我回头就要merge进dev了。。谢谢

xmake update build_jobs 更新到这个分支版本。。

fasiondog · 2020-03-14T05:47:14Z

PS C:\WINDOWS\system32> xmake update build_jobs
error: unable to select version for range 'build_jobs'

fasiondog · 2020-03-14T05:48:40Z

dev也一直没法update，总是下载就失败了，网络太差了

waruqi · 2020-03-14T05:55:25Z

PS C:\WINDOWS\system32> xmake update build_jobs
error: unable to select version for range 'build_jobs'

忘记push到gitee/gitlab上了，所以git tags list从镜像没拉取到，我刚同步过去了，你再执行下试试

waruqi · 2020-03-14T06:00:35Z

dev也一直没法update，总是下载就失败了，网络太差了

你试试直接到ci上下载 installer.exe试试，https://ci.appveyor.com/project/waruqi/xmake/builds/31462485/job/09tpgsu2h3ao7ges/artifacts

waruqi · 2020-03-16T05:51:37Z

这块暂时优化的差不多了，分布式编译后面会逐步实现，其他一些优化点回头有时间可以考虑下，最近先不动了

waruqi · 2020-03-19T06:11:33Z

目前dev版本的基准测试结果：build https://github.com/xmake-io/xmake/tree/master/core

Termux/Android cpu/8core

-j12

buildsystem	spend time
xmake	24.890s
ninja	25.682s
cmake(gen+make)	5.416s+28.473s
cmake(gen+ninja)	4.458s+24.842s

-j1

buildsystem	spend time
xmake	1m57.707s
ninja	1m52.845s
cmake(gen+make)	5.416s+2m10.539s
cmake(gen+ninja)	4.458s+1m54.868s

MacOS cpu/8core

-j12

buildsystem	spend time
xmake	12.264s
ninja	11.327s
cmake(gen+make)	1.203s+14.030s
cmake(gen+ninja)	0.988s+11.644s

-j1

buildsystem	spend time
xmake	39.937s
ninja	38.995s
cmake(gen+make)	1.203s+41.737s
cmake(gen+ninja)	0.988s+38.022s

codehz · 2020-03-19T06:19:05Z

windows上有异常

configure
{
    arch = x86_64
    ccache = true
    mode = release
    ndk_stdcxx = true
    host = windows
    theme = dark
    kind = static
    buildir = build
    vs = 2019
    plat = msys
    cxx = cl
}
[ 28%]: compiling.release ModLoader\hook_impl.cpp
cl -c -nologo -m64 -Fobuild\\.objs\\ModLoader\\msys\\x86_64\\release\\ModLoader\\hook_impl.cpp.o ModLoader\\hook_impl.cpp
error: @programdir\core\main.lua:261: @programdir\actions\build\main.lua:124: ...mdir\core\sandbox\modules\import\core\base\scheduler.lua:87: ...mdir\core\sandbox\modules\import\core\base\scheduler.lua:47: @programdir\modules\private\async\runjobs.lua:213: @programdir\modules\private\action\build\object.lua:77: @programdir\modules\core\tools\cl.lua:377: @programdir\modules\private\tools\vstool.lua:114: bad argument #4 to 'format' (number expected, got nil)
stack traceback:
    [C]: in function 'format'
    [@programdir\modules\private\tools\vstool.lua:114]:
    [C]: in function 'trycall'
    [@programdir\core\sandbox\modules\try.lua:121]: in function 'try'
    [@programdir\modules\core\tools\cl.lua:341]: in function '_compile1'
    [@programdir\modules\core\tools\cl.lua:431]:
    [C]: in function 'compile'
    [@programdir\modules\private\action\build\object.lua:77]: in function 'script'
    [@programdir\modules\private\action\build\object.lua:90]: in function '_build_object'
    [@programdir\modules\private\action\build\object.lua:115]: in function 'jobfunc'
    [@programdir\modules\private\async\runjobs.lua:190]:
    [C]: in function 'trycall'
    [@programdir\core\sandbox\modules\try.lua:121]: in function 'try'
    [@programdir\modules\private\async\runjobs.lua:187]: in function 'cotask'
    [@programdir\core\base\scheduler.lua:317]:

stack traceback:
        [C]: in function 'error'
        @programdir\core\base\os.lua:781: in function 'raise'
        @programdir\modules\private\async\runjobs.lua:213: in function 'catch'
        @programdir\core\sandbox\modules\try.lua:127: in function 'try'
        @programdir\modules\private\async\runjobs.lua:187: in function 'cotask'
        @programdir\core\base\scheduler.lua:317: in function <@programdir\core\base\scheduler.lua:315>
stack traceback:
        [C]: in function 'error'
        @programdir\core\base\os.lua:781: in function 'raise'
        ...mdir\core\sandbox\modules\import\core\base\scheduler.lua:47: in function 'co_start_named'
        @programdir\modules\private\async\runjobs.lua:185: in function <@programdir\modules\private\async\runjobs.lua:144>        [C]: in function 'trycall'
        @programdir\core\base\scheduler.lua:423: in function 'co_group_begin'
        ...mdir\core\sandbox\modules\import\core\base\scheduler.lua:85: in function 'co_group_begin'
        @programdir\modules\private\async\runjobs.lua:144: in function 'runjobs'
        @programdir\actions\build\build.lua:220: in function 'build'
        @programdir\actions\build\main.lua:116: in function <@programdir\actions\build\main.lua:111>
        [C]: in function 'trycall'
        @programdir\core\sandbox\modules\try.lua:121: in function 'try'
        @programdir\actions\build\main.lua:110: in function <@programdir\actions\build\main.lua:83>
        [C]: in function 'load'
        @programdir\core\base\task.lua:520: in function 'run'
        @programdir\core\main.lua:259: in function 'cotask'
        @programdir\core\base\scheduler.lua:317: in function <@programdir\core\base\scheduler.lua:315>
stack traceback:
        [C]: in function 'error'
        @programdir\core\base\os.lua:781: in function 'raise'
        @programdir\core\main.lua:261: in function 'cotask'
        @programdir\core\base\scheduler.lua:317: in function <@programdir\core\base\scheduler.lua:315>

waruqi · 2020-03-19T06:43:46Z

这个我修了，可以更新到dev再试试，之前重构os.execv没改完整。。

qyangdu · 2020-10-15T08:50:37Z

C++增量编译，depfiles好像没有检查。如果是header文件改变了，无法触发相关cpp文件重新编译

waruqi · 2020-10-15T08:54:30Z

C++增量编译，depfiles好像没有检查。如果是header文件改变了，无法触发相关cpp文件重新编译

有问题，请开issues，并详细描述环境和复现步骤，不要到无关的issues以及closed的issues上反馈。

waruqi · 2022-05-20T15:36:17Z

新增了内置本地 cache 和分布式编译支持 #274

waruqi added the improvement label Oct 8, 2019

waruqi added this to the todo milestone Oct 8, 2019

waruqi mentioned this issue Oct 8, 2019

希望支持ninja #588

Closed

waruqi mentioned this issue Mar 14, 2020

Supports parallel compilation and linking across targets #708

Merged

waruqi modified the milestones: todo, v2.3.2 Mar 14, 2020

waruqi closed this as completed Mar 16, 2020

orzuk-lmj mentioned this issue Mar 16, 2020

關於#589合併後編譯出錯 #713

Closed

RimuruChan mentioned this issue Mar 19, 2023

同一项目XMake编译时长比XMake生成的CMake使用ninja编译慢一分多钟 #3532

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve and optimize build speed #589

Improve and optimize build speed #589

waruqi commented Oct 8, 2019 •

edited

Loading

waruqi commented Oct 9, 2019 •

edited

Loading

waruqi commented Mar 1, 2020 •

edited

Loading

waruqi commented Mar 1, 2020 •

edited

Loading

waruqi commented Mar 2, 2020 •

edited

Loading

waruqi commented Mar 2, 2020

waruqi commented Mar 4, 2020 •

edited

Loading

waruqi commented Mar 14, 2020

fasiondog commented Mar 14, 2020

fasiondog commented Mar 14, 2020

waruqi commented Mar 14, 2020

waruqi commented Mar 14, 2020

waruqi commented Mar 16, 2020

waruqi commented Mar 19, 2020 •

edited

Loading

codehz commented Mar 19, 2020 •

edited

Loading

waruqi commented Mar 19, 2020

qyangdu commented Oct 15, 2020

waruqi commented Oct 15, 2020

waruqi commented May 20, 2022

Improve and optimize build speed #589

Improve and optimize build speed #589

Comments

waruqi commented Oct 8, 2019 • edited Loading

waruqi commented Oct 9, 2019 • edited Loading

waruqi commented Mar 1, 2020 • edited Loading

waruqi commented Mar 1, 2020 • edited Loading

waruqi commented Mar 2, 2020 • edited Loading

waruqi commented Mar 2, 2020

waruqi commented Mar 4, 2020 • edited Loading

waruqi commented Mar 14, 2020

fasiondog commented Mar 14, 2020

fasiondog commented Mar 14, 2020

waruqi commented Mar 14, 2020

waruqi commented Mar 14, 2020

waruqi commented Mar 16, 2020

waruqi commented Mar 19, 2020 • edited Loading

Termux/Android cpu/8core

-j12

-j1

MacOS cpu/8core

-j12

-j1

codehz commented Mar 19, 2020 • edited Loading

waruqi commented Mar 19, 2020

qyangdu commented Oct 15, 2020

waruqi commented Oct 15, 2020

waruqi commented May 20, 2022

waruqi commented Oct 8, 2019 •

edited

Loading

waruqi commented Oct 9, 2019 •

edited

Loading

waruqi commented Mar 1, 2020 •

edited

Loading

waruqi commented Mar 1, 2020 •

edited

Loading

waruqi commented Mar 2, 2020 •

edited

Loading

waruqi commented Mar 4, 2020 •

edited

Loading

waruqi commented Mar 19, 2020 •

edited

Loading

codehz commented Mar 19, 2020 •

edited

Loading