-
-
Notifications
You must be signed in to change notification settings - Fork 784
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
build.across_targets_in_parallel 在依赖链中失效 #5003
Comments
Title: build.across_targets_in_parallel is invalid in the dependency chain |
主要应用场景为 codegen header,依赖 target 在 header 不完善的情况下直接进行编译,导致编译失败。 在这种情况下必须有一个手段让 codegen target block 住后续所有依赖链上 obj 的编译 |
The main application scenario is codegen header, which relies on target to compile directly when the header is incomplete, resulting in compilation failure. In this case, there must be a way for the codegen target block to host the compilation of obj in all subsequent dependency chains. |
得在 final_target 里面设置这个 policy 才行,限制 final_target 里面依赖的所有 target, 禁止跨 target 并行编译 cpp,但是 每个 target 内部的 cpp 还是并行的。 target("final_target")
set_policy("build.across_targets_in_parallel", false) |
This policy must be set in final_target. The deps in final_target are restricted to prohibit parallel compilation of cpp across targets, but the cpp within the target is still parallel. target("final_target")
set_policy("build.across_targets_in_parallel", false) |
参考 https://github.com/xmake-io/xmake/blob/master/tests/projects/other/autogen_codedep/xmake.lua 不要在子 deps 里面设置这个 policy |
假设以下情形 Gen <- A <- B <- C,当 Gen 完成后,A、B、C 之间的 cpp 可以自由的并行编译,并不是整条链上的 target 都必须保证串行 而且实际工程中如果要手动为 final target 添加 policy,需要翻看其依赖的所有 target 有没有启用 codegen,或者在 after_load 里查 deps 自动添加,不知道这会不会有问题 另外,现在 codegen 是作为基础设施存在的,这么做最后可能整个项目都禁用了这一优化,我觉得可能还是在 codegen target 上指定 parallel level 比较合理 |
Assume the following situation: Gen <- A <- B <- C. After Gen is completed, the cpp between A, B, and C can be freely compiled in parallel. Not all targets on the entire chain must guarantee serialization. And if you want to manually add a policy to the final target in an actual project, you need to check whether all the targets it depends on have codegen enabled, or check in after_load to see if deps are automatically added. I don’t know if this will be a problem. In addition, now that codegen exists as infrastructure, this optimization may be disabled for the entire project. I think it may be more reasonable to specify parallel level on the codegen target. |
我知道,但目前仅支持 Gen <- ... X 中 Gen 和 X 之间需要确保顺序的 X 上设置 policy 。。不支持在 Gen 上设置,也不太好实现。。 |
I know, but currently only Gen <- ... X needs to be set on X to ensure order between Gen and X. . Setting up on Gen is not supported and not easy to implement. . |
思考,那能不能折中一下,手动指定 X 需要强制确保顺序的 target_name,然后由 user 给出解算好的 codegen_targets,这样子会不会更好实现一些,主要相比于粗暴的全部 block 这样子的时间提升还是不小的 |
Thinking about it, can we make a compromise? Manually specify the target_name of The time improvement is still not small. |
限制就是子 target 粒度的 policy 不好实现,你指定 target name 就跟直接在 Gen 里面设置 policy 没啥区别。。 或者可以在 Gen 后面加个 fake target Y 去设置 policy 限制住,Gen <- Y (policy) ... <- X 这个 Y 仅仅依赖一个 Gen 。。最小化范围。。这个目前应该是可以的 也就是你上面那个 internal_target 里面设置 target("internal_target")
set_policy("build.across_targets_in_parallel", false) |
The limitation is that sub-target granular policies are difficult to implement. Specifying the target name is no different from setting the policy directly in Gen. . Or you can add a fake target Y after Gen to set policy restrictions, Gen <- Y (policy) ... <- X This Y depends only on one Gen. . Minimize scope. . This should be possible for now That is what you set in the internal_target above. target("internal_target")
set_policy("build.across_targets_in_parallel", false) |
add_rules("mode.debug", "mode.release")
target("foo")
set_kind("object")
add_files("src/test.cpp")
before_build(function (target)
print("before_build 1")
os.exec("sleep 10")
print("before_build 2")
end)
target("bar")
set_kind("object")
add_files("src/test.cpp")
add_deps("foo")
set_policy("build.across_targets_in_parallel", false)
target("test")
set_kind("binary")
add_files("src/*.cpp")
add_deps("bar")
after_build(function (target)
print("after_build")
end) $ xmake
before_build 1
before_build 2
[ 30%]: cache compiling.release src/foo.cpp
[ 50%]: cache compiling.release src/bar.cpp
[ 80%]: cache compiling.release src/main.cpp
[ 90%]: linking.release test
after_build
[100%]: build ok, spent 11.807s |
add_rules("mode.debug", "mode.release")
target("gen")
set_kind("phony")
before_build(function (target)
print("begin codegen")
os.exec("python gen.py")
print("end codegen")
end)
target("proxy")
set_kind("phony")
add_deps("gen")
set_policy("build.across_targets_in_parallel", false)
target("A")
set_kind("shared")
add_includedirs("include")
add_files("src/a.cpp")
add_deps("proxy")
target("B")
set_kind("shared")
add_includedirs("include")
add_files("src/b.cpp")
add_deps("A") begin codegen
[ 33%]: compiling.release src\b.cpp
end codegen
[ 66%]: compiling.release src\a.cpp
error: b.cpp
src\b.cpp(1): fatal error C1083: Cannot open include file: 'test.hpp': No such file or directory gen.py 生成了一个 header 提供给 a.cpp 使用,b.cpp 模拟了依赖 a 导致也依赖 codegen header 的情况 import time
time.sleep(0.1)
with open("include/test.hpp", "w") as f:
f.write("#pragma once\n") gen<-proxy<-A<-B,其中 A 被顺利的阻止,但是间接依赖的 B 依旧会并行编译 begin codegen
[ 41%]: compiling.release src\a.cpp
end codegen
error: a.cpp
src\a.cpp(1): fatal error C1083: Cannot open include file: 'test.hpp': No such file or directory 唯一能成功编译的做法是将所有用到 gen 的 module 都加上这个 policy |
但目前的实现架构,不太好支持,也许后面等有时间 可以考虑改进,但短期内 不太好搞,没这么多时间。或者可以用 native module 去实现 codegen ,或者只能对 parent target 进行 policy |
主要是从任何非内部手段都没法控制 jobbatch 派发啊,即使用 native module 逻辑上就是行不通的 |
不是,使用 native modules,你可以在实际调用 codegen 的任何地方,开始 build codegen 去执行它,总归是最早被执行的。。 |
我在本地实现了一下我的想法,测试是有效的,提交了一个 pr #5011 |
如果目前要支持 gen 里面 policy ,确实只能这么做,不过 review 可能要等两天了,最近有点忙,没有时间细看和测试 |
If you currently want to support the policy in gen, you really can only do this. However, the review may have to wait two days. I have been a little busy recently and have no time to take a closer look and test. |
收到,感谢 |
Received, thanks |
我改了下,再试试 |
I changed it and try again |
我这边没问题 |
No problem on my side |
我 merge 了, |
I merged, |
Xmake 版本
2.9.1
操作系统版本和架构
Any OS
描述问题
间接依赖target时,被依赖target的before_build不能保证在编译前执行。
期待的结果
标记policy后,before_build after_build应作为特殊事件,保序执行。
工程配置
`
-- codegen_proxy -> internal_target -> final_target
target("codegen_proxy")
set_kind("object")
before_build(function(target)
os.runv("python", {"codegen.py"})
end)
set_policy("build.across_targets_in_parallel", false, {public = true, inherit = true})
target_end()
target("internal_target")
set_kind("object")
add_deps("codegen_proxy", {public = true, inherit = true})
-- final_target need result from codegen.py
target("final_target")
set_kind("binary")
add_files("src/main.cpp")
after_build(function(target)
os.rm(path.join(os.scriptdir(), "src/test.generate.h"))
end)
add_deps("internal_target", {public = true, inherit = true})
`
附加信息和错误日志
"test.generate.h" No such file or directory.
The text was updated successfully, but these errors were encountered: