Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build.across_targets_in_parallel 在依赖链中失效 #5003

Closed
MaxwellGengYF opened this issue Apr 23, 2024 · 30 comments
Closed

build.across_targets_in_parallel 在依赖链中失效 #5003

MaxwellGengYF opened this issue Apr 23, 2024 · 30 comments
Labels
Milestone

Comments

@MaxwellGengYF
Copy link
Contributor

Xmake 版本

2.9.1

操作系统版本和架构

Any OS

描述问题

间接依赖target时,被依赖target的before_build不能保证在编译前执行。

期待的结果

标记policy后,before_build after_build应作为特殊事件,保序执行。

工程配置

`
-- codegen_proxy -> internal_target -> final_target
target("codegen_proxy")
set_kind("object")
before_build(function(target)
os.runv("python", {"codegen.py"})
end)
set_policy("build.across_targets_in_parallel", false, {public = true, inherit = true})
target_end()

target("internal_target")
set_kind("object")
add_deps("codegen_proxy", {public = true, inherit = true})
-- final_target need result from codegen.py
target("final_target")
set_kind("binary")
add_files("src/main.cpp")
after_build(function(target)
os.rm(path.join(os.scriptdir(), "src/test.generate.h"))
end)
add_deps("internal_target", {public = true, inherit = true})
`

附加信息和错误日志

"test.generate.h" No such file or directory.

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


Title: build.across_targets_in_parallel is invalid in the dependency chain

@zhurongjun
Copy link
Contributor

主要应用场景为 codegen header,依赖 target 在 header 不完善的情况下直接进行编译,导致编译失败。

在这种情况下必须有一个手段让 codegen target block 住后续所有依赖链上 obj 的编译

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


The main application scenario is codegen header, which relies on target to compile directly when the header is incomplete, resulting in compilation failure.

In this case, there must be a way for the codegen target block to host the compilation of obj in all subsequent dependency chains.

@waruqi
Copy link
Member

waruqi commented Apr 24, 2024

得在 final_target 里面设置这个 policy 才行,限制 final_target 里面依赖的所有 target, 禁止跨 target 并行编译 cpp,但是 每个 target 内部的 cpp 还是并行的。

target("final_target")
    set_policy("build.across_targets_in_parallel", false)

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


This policy must be set in final_target. The deps in final_target are restricted to prohibit parallel compilation of cpp across targets, but the cpp within the target is still parallel.

target("final_target")
    set_policy("build.across_targets_in_parallel", false)

@waruqi
Copy link
Member

waruqi commented Apr 24, 2024

参考 https://github.com/xmake-io/xmake/blob/master/tests/projects/other/autogen_codedep/xmake.lua

不要在子 deps 里面设置这个 policy

@zhurongjun
Copy link
Contributor

得在 final_target 里面设置这个 policy 才行,限制 final_target 里面依赖的所有 target, 禁止跨 target 并行编译 cpp,但是 每个 target 内部的 cpp 还是并行的。

target("final_target")
    set_policy("build.across_targets_in_parallel", false)

假设以下情形 Gen <- A <- B <- C,当 Gen 完成后,A、B、C 之间的 cpp 可以自由的并行编译,并不是整条链上的 target 都必须保证串行

而且实际工程中如果要手动为 final target 添加 policy,需要翻看其依赖的所有 target 有没有启用 codegen,或者在 after_load 里查 deps 自动添加,不知道这会不会有问题

另外,现在 codegen 是作为基础设施存在的,这么做最后可能整个项目都禁用了这一优化,我觉得可能还是在 codegen target 上指定 parallel level 比较合理

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


This policy must be set in final_target to limit all targets that final_target depends on and prohibit parallel compilation of cpp across targets. However, the cpp within each target is still parallel.

``lua
target("final_target")
set_policy("build.across_targets_in_parallel", false)

Assume the following situation: Gen <- A <- B <- C. After Gen is completed, the cpp between A, B, and C can be freely compiled in parallel. Not all targets on the entire chain must guarantee serialization.

And if you want to manually add a policy to the final target in an actual project, you need to check whether all the targets it depends on have codegen enabled, or check in after_load to see if deps are automatically added. I don’t know if this will be a problem.

In addition, now that codegen exists as infrastructure, this optimization may be disabled for the entire project. I think it may be more reasonable to specify parallel level on the codegen target.

@waruqi
Copy link
Member

waruqi commented Apr 24, 2024

得在 final_target 里面设置这个 policy 才行,限制 final_target 里面依赖的所有 target, 禁止跨 target 并行编译 cpp,但是 每个 target 内部的 cpp 还是并行的。

target("final_target")
    set_policy("build.across_targets_in_parallel", false)

假设以下情形 Gen <- A <- B <- C,当 Gen 完成后,A、B、C 之间的 cpp 可以自由的并行编译,并不是整条链上的 target 都必须保证串行

而且实际工程中如果要手动为 final target 添加 policy,需要翻看其依赖的所有 target 有没有启用 codegen,或者在 after_load 里查 deps 自动添加,不知道这会不会有问题

另外,现在 codegen 是作为基础设施存在的,这么做最后可能整个项目都禁用了这一优化,我觉得可能还是在 codegen target 上指定 parallel level 比较合理

我知道,但目前仅支持 Gen <- ... X 中 Gen 和 X 之间需要确保顺序的 X 上设置 policy 。。不支持在 Gen 上设置,也不太好实现。。

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


This policy must be set in final_target to limit all targets that final_target depends on and prohibit parallel compilation of cpp across targets. However, the cpp within each target is still parallel.

target("final_target")
set_policy("build.across_targets_in_parallel", false)

Assume the following situation: Gen <- A <- B <- C. After Gen is completed, the cpp between A, B, and C can be freely compiled in parallel. Not all targets on the entire chain must guarantee serialization.

And if you want to manually add a policy to the final target in an actual project, you need to check whether all the targets it depends on have codegen enabled, or check in after_load to see if deps are automatically added. I don’t know if this will be a problem.

In addition, now that codegen exists as infrastructure, this optimization may be disabled for the entire project. I think it may be more reasonable to specify parallel level on the codegen target.

I know, but currently only Gen <- ... X needs to be set on X to ensure order between Gen and X. . Setting up on Gen is not supported and not easy to implement. .

@zhurongjun
Copy link
Contributor

得在 final_target 里面设置这个 policy 才行,限制 final_target 里面依赖的所有 target, 禁止跨 target 并行编译 cpp,但是 每个 target 内部的 cpp 还是并行的。

target("final_target")
    set_policy("build.across_targets_in_parallel", false)

假设以下情形 Gen <- A <- B <- C,当 Gen 完成后,A、B、C 之间的 cpp 可以自由的并行编译,并不是整条链上的 target 都必须保证串行
而且实际工程中如果要手动为 final target 添加 policy,需要翻看其依赖的所有 target 有没有启用 codegen,或者在 after_load 里查 deps 自动添加,不知道这会不会有问题
另外,现在 codegen 是作为基础设施存在的,这么做最后可能整个项目都禁用了这一优化,我觉得可能还是在 codegen target 上指定 parallel level 比较合理

我知道,但目前仅支持 Gen <- ... X 中 Gen 和 X 之间需要确保顺序的 X 上设置 policy 。。不支持在 Gen 上设置,也不太好实现。。

思考,那能不能折中一下,手动指定 X 需要强制确保顺序的 target_name,然后由 user 给出解算好的 codegen_targets,这样子会不会更好实现一些,主要相比于粗暴的全部 block 这样子的时间提升还是不小的

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


This policy must be set in final_target to limit all targets that final_target depends on and prohibit parallel compilation of cpp across targets. However, the cpp within each target is still parallel.

target("final_target")
set_policy("build.across_targets_in_parallel", false)

Assume the following situation: Gen <- A <- B <- C. When Gen is completed, the cpp between A, B, and C can be freely compiled in parallel. Not all targets on the entire chain must guarantee serialization.
And if you want to manually add a policy to the final target in an actual project, you need to check whether all the targets it depends on have codegen enabled, or check in after_load to see if deps are automatically added. I don’t know if this will be a problem.
In addition, now that codegen exists as infrastructure, this optimization may be disabled for the entire project. I think it may be more reasonable to specify parallel level on the codegen target.

I know, but currently only Gen is supported <-... In X, policy needs to be set on X to ensure order between Gen and X. . Setting up on Gen is not supported and not easy to implement. .

Thinking about it, can we make a compromise? Manually specify the target_name of The time improvement is still not small.

@waruqi
Copy link
Member

waruqi commented Apr 24, 2024

得在 final_target 里面设置这个 policy 才行,限制 final_target 里面依赖的所有 target, 禁止跨 target 并行编译 cpp,但是 每个 target 内部的 cpp 还是并行的。

target("final_target")
    set_policy("build.across_targets_in_parallel", false)

假设以下情形 Gen <- A <- B <- C,当 Gen 完成后,A、B、C 之间的 cpp 可以自由的并行编译,并不是整条链上的 target 都必须保证串行
而且实际工程中如果要手动为 final target 添加 policy,需要翻看其依赖的所有 target 有没有启用 codegen,或者在 after_load 里查 deps 自动添加,不知道这会不会有问题
另外,现在 codegen 是作为基础设施存在的,这么做最后可能整个项目都禁用了这一优化,我觉得可能还是在 codegen target 上指定 parallel level 比较合理

我知道,但目前仅支持 Gen <- ... X 中 Gen 和 X 之间需要确保顺序的 X 上设置 policy 。。不支持在 Gen 上设置,也不太好实现。。

思考,那能不能折中一下,手动指定 X 需要强制确保顺序的 target_name,然后由 user 给出解算好的 codegen_targets,这样子会不会更好实现一些,主要相比于粗暴的全部 block 这样子的时间提升还是不小的

限制就是子 target 粒度的 policy 不好实现,你指定 target name 就跟直接在 Gen 里面设置 policy 没啥区别。。

或者可以在 Gen 后面加个 fake target Y 去设置 policy 限制住,Gen <- Y (policy) ... <- X

这个 Y 仅仅依赖一个 Gen 。。最小化范围。。这个目前应该是可以的

也就是你上面那个 internal_target 里面设置

target("internal_target")
    set_policy("build.across_targets_in_parallel", false)

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


This policy must be set in final_target to limit all targets that final_target depends on and prohibit parallel compilation of cpp across targets. However, the cpp within each target is still parallel.
``lua
target("final_target")
set_policy("build.across_targets_in_parallel", false)

Assume the following situation: Gen <- A <- B <- C. When Gen is completed, the cpp between A, B, and C can be freely compiled in parallel. Not all targets on the entire chain must guarantee serialization.
And if you want to manually add a policy to the final target in the actual project, you need to check whether all the targets it depends on have codegen enabled, or check after_load to see if deps are automatically added. I don’t know if this will be a problem.
In addition, now that codegen exists as infrastructure, this optimization may be disabled for the entire project. I think it may be more reasonable to specify parallel level on the codegen target.

I know, but currently only supports Gen <-... In X, policy needs to be set on X to ensure order between Gen and X. . Setting up on Gen is not supported and not easy to implement. .

Thinking about it, can we make a compromise? Manually specify the target_name of The time improvement is still quite big.

The limitation is that sub-target granular policies are difficult to implement. Specifying the target name is no different from setting the policy directly in Gen. .

Or you can add a fake target Y after Gen to set policy restrictions, Gen <- Y (policy) ... <- X

This Y depends only on one Gen. . Minimize scope. . This should be possible for now

That is what you set in the internal_target above.

target("internal_target")
    set_policy("build.across_targets_in_parallel", false)

@waruqi
Copy link
Member

waruqi commented Apr 24, 2024

add_rules("mode.debug", "mode.release")


target("foo")
    set_kind("object")
    add_files("src/test.cpp")
    before_build(function (target)
        print("before_build 1")
        os.exec("sleep 10")
        print("before_build 2")
    end)

target("bar")
    set_kind("object")
    add_files("src/test.cpp")
    add_deps("foo")
    set_policy("build.across_targets_in_parallel", false)

target("test")
    set_kind("binary")
    add_files("src/*.cpp")
    add_deps("bar")
    after_build(function (target)
        print("after_build")
    end)
$ xmake
before_build 1
before_build 2
[ 30%]: cache compiling.release src/foo.cpp
[ 50%]: cache compiling.release src/bar.cpp
[ 80%]: cache compiling.release src/main.cpp
[ 90%]: linking.release test
after_build
[100%]: build ok, spent 11.807s

@zhurongjun
Copy link
Contributor

add_rules("mode.debug", "mode.release")

target("gen")
    set_kind("phony")
    before_build(function (target) 
        print("begin codegen")
        os.exec("python gen.py")
        print("end codegen")
    end)

target("proxy")
    set_kind("phony")
    add_deps("gen")
    set_policy("build.across_targets_in_parallel", false)

target("A")
    set_kind("shared")
    add_includedirs("include")
    add_files("src/a.cpp")
    add_deps("proxy")

target("B")
    set_kind("shared")
    add_includedirs("include")
    add_files("src/b.cpp")
    add_deps("A")
begin codegen
[ 33%]: compiling.release src\b.cpp
end codegen
[ 66%]: compiling.release src\a.cpp
error: b.cpp
src\b.cpp(1): fatal error C1083: Cannot open include file: 'test.hpp': No such file or directory

gen.py 生成了一个 header 提供给 a.cpp 使用,b.cpp 模拟了依赖 a 导致也依赖 codegen header 的情况

import time
time.sleep(0.1)
with open("include/test.hpp", "w") as f:
    f.write("#pragma once\n")

gen<-proxy<-A<-B,其中 A 被顺利的阻止,但是间接依赖的 B 依旧会并行编译
如果将 set_policy 移动到 B target 中,那么会变成

begin codegen
[ 41%]: compiling.release src\a.cpp
end codegen
error: a.cpp
src\a.cpp(1): fatal error C1083: Cannot open include file: 'test.hpp': No such file or directory

唯一能成功编译的做法是将所有用到 gen 的 module 都加上这个 policy

@waruqi
Copy link
Member

waruqi commented Apr 24, 2024

add_rules("mode.debug", "mode.release")

target("gen")
    set_kind("phony")
    before_build(function (target) 
        print("begin codegen")
        os.exec("python gen.py")
        print("end codegen")
    end)

target("proxy")
    set_kind("phony")
    add_deps("gen")
    set_policy("build.across_targets_in_parallel", false)

target("A")
    set_kind("shared")
    add_includedirs("include")
    add_files("src/a.cpp")
    add_deps("proxy")

target("B")
    set_kind("shared")
    add_includedirs("include")
    add_files("src/b.cpp")
    add_deps("A")
begin codegen
[ 33%]: compiling.release src\b.cpp
end codegen
[ 66%]: compiling.release src\a.cpp
error: b.cpp
src\b.cpp(1): fatal error C1083: Cannot open include file: 'test.hpp': No such file or directory

gen.py 生成了一个 header 提供给 a.cpp 使用,b.cpp 模拟了依赖 a 导致也依赖 codegen header 的情况

import time
time.sleep(0.1)
with open("include/test.hpp", "w") as f:
    f.write("#pragma once\n")

gen<-proxy<-A<-B,其中 A 被顺利的阻止,但是间接依赖的 B 依旧会并行编译 如果将 set_policy 移动到 B target 中,那么会变成

begin codegen
[ 41%]: compiling.release src\a.cpp
end codegen
error: a.cpp
src\a.cpp(1): fatal error C1083: Cannot open include file: 'test.hpp': No such file or directory

唯一能成功编译的做法是将所有用到 gen 的 module 都加上这个 policy

但目前的实现架构,不太好支持,也许后面等有时间 可以考虑改进,但短期内 不太好搞,没这么多时间。或者可以用 native module 去实现 codegen ,或者只能对 parent target 进行 policy

@SaeruHikari
Copy link

add_rules("mode.debug", "mode.release")

target("gen")
    set_kind("phony")
    before_build(function (target) 
        print("begin codegen")
        os.exec("python gen.py")
        print("end codegen")
    end)

target("proxy")
    set_kind("phony")
    add_deps("gen")
    set_policy("build.across_targets_in_parallel", false)

target("A")
    set_kind("shared")
    add_includedirs("include")
    add_files("src/a.cpp")
    add_deps("proxy")

target("B")
    set_kind("shared")
    add_includedirs("include")
    add_files("src/b.cpp")
    add_deps("A")
begin codegen
[ 33%]: compiling.release src\b.cpp
end codegen
[ 66%]: compiling.release src\a.cpp
error: b.cpp
src\b.cpp(1): fatal error C1083: Cannot open include file: 'test.hpp': No such file or directory

gen.py 生成了一个 header 提供给 a.cpp 使用,b.cpp 模拟了依赖 a 导致也依赖 codegen header 的情况

import time
time.sleep(0.1)
with open("include/test.hpp", "w") as f:
    f.write("#pragma once\n")

gen<-proxy<-A<-B,其中 A 被顺利的阻止,但是间接依赖的 B 依旧会并行编译 如果将 set_policy 移动到 B target 中,那么会变成

begin codegen
[ 41%]: compiling.release src\a.cpp
end codegen
error: a.cpp
src\a.cpp(1): fatal error C1083: Cannot open include file: 'test.hpp': No such file or directory

唯一能成功编译的做法是将所有用到 gen 的 module 都加上这个 policy

但目前的实现架构,不太好支持,也许后面等有时间 可以考虑改进,但短期内 不太好搞,没这么多时间。或者可以用 native module 去实现 codegen ,或者只能对 parent target 进行 policy

主要是从任何非内部手段都没法控制 jobbatch 派发啊,即使用 native module 逻辑上就是行不通的

@waruqi
Copy link
Member

waruqi commented Apr 25, 2024

add_rules("mode.debug", "mode.release")

target("gen")
    set_kind("phony")
    before_build(function (target) 
        print("begin codegen")
        os.exec("python gen.py")
        print("end codegen")
    end)

target("proxy")
    set_kind("phony")
    add_deps("gen")
    set_policy("build.across_targets_in_parallel", false)

target("A")
    set_kind("shared")
    add_includedirs("include")
    add_files("src/a.cpp")
    add_deps("proxy")

target("B")
    set_kind("shared")
    add_includedirs("include")
    add_files("src/b.cpp")
    add_deps("A")
begin codegen
[ 33%]: compiling.release src\b.cpp
end codegen
[ 66%]: compiling.release src\a.cpp
error: b.cpp
src\b.cpp(1): fatal error C1083: Cannot open include file: 'test.hpp': No such file or directory

gen.py 生成了一个 header 提供给 a.cpp 使用,b.cpp 模拟了依赖 a 导致也依赖 codegen header 的情况

import time
time.sleep(0.1)
with open("include/test.hpp", "w") as f:
    f.write("#pragma once\n")

gen<-proxy<-A<-B,其中 A 被顺利的阻止,但是间接依赖的 B 依旧会并行编译 如果将 set_policy 移动到 B target 中,那么会变成

begin codegen
[ 41%]: compiling.release src\a.cpp
end codegen
error: a.cpp
src\a.cpp(1): fatal error C1083: Cannot open include file: 'test.hpp': No such file or directory

唯一能成功编译的做法是将所有用到 gen 的 module 都加上这个 policy

但目前的实现架构,不太好支持,也许后面等有时间 可以考虑改进,但短期内 不太好搞,没这么多时间。或者可以用 native module 去实现 codegen ,或者只能对 parent target 进行 policy

主要是从任何非内部手段都没法控制 jobbatch 派发啊,即使用 native module 逻辑上就是行不通的

不是,使用 native modules,你可以在实际调用 codegen 的任何地方,开始 build codegen 去执行它,总归是最早被执行的。。

@zhurongjun
Copy link
Contributor

我在本地实现了一下我的想法,测试是有效的,提交了一个 pr #5011
这个功能和 build.across_targets_in_parallel 有较大的出入,我觉得还是作为新的 policy 比较合理

@waruqi
Copy link
Member

waruqi commented Apr 25, 2024

我在本地实现了一下我的想法,测试是有效的,提交了一个 pr #5011 这个功能和 build.across_targets_in_parallel 有较大的出入,我觉得还是作为新的 policy 比较合理

如果目前要支持 gen 里面 policy ,确实只能这么做,不过 review 可能要等两天了,最近有点忙,没有时间细看和测试

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


I implemented my idea locally and the test was effective. I submitted a PR #5011. This function is quite different from build.across_targets_in_parallel. I think it is more reasonable to use it as a new policy.

If you currently want to support the policy in gen, you really can only do this. However, the review may have to wait two days. I have been a little busy recently and have no time to take a closer look and test.

@zhurongjun
Copy link
Contributor

我在本地实现了一下我的想法,测试是有效的,提交了一个 pr #5011 这个功能和 build.across_targets_in_parallel 有较大的出入,我觉得还是作为新的 policy 比较合理

如果目前要支持 gen 里面 policy ,确实只能这么做,不过 review 可能要等两天了,最近有点忙,没有时间细看和测试

收到,感谢

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


I implemented my idea locally and the test was effective. I submitted a pr #5011. This function is quite different from build.across_targets_in_parallel. I think it is more reasonable to use it as a new policy.

If you currently want to support the policy in gen, you really can only do this. However, the review may have to wait two days. I have been a little busy recently and have no time to take a closer look and test.

Received, thanks

@waruqi
Copy link
Member

waruqi commented Apr 25, 2024

我在本地实现了一下我的想法,测试是有效的,提交了一个 pr #5011 这个功能和 build.across_targets_in_parallel 有较大的出入,我觉得还是作为新的 policy 比较合理

我改了下,再试试

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


I implemented my idea locally and the test was effective. I submitted a PR #5011. This function is quite different from build.across_targets_in_parallel. I think it is more reasonable to use it as a new policy.

I changed it and try again

@zhurongjun
Copy link
Contributor

我在本地实现了一下我的想法,测试是有效的,提交了一个 pr #5011 这个功能和 build.across_targets_in_parallel 有较大的出入,我觉得还是作为新的 policy 比较合理

我改了下,再试试

我这边没问题

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


I implemented my idea locally and the test was effective. I submitted a pr #5011. This function is quite different from build.across_targets_in_parallel. I think it is more reasonable to use it as a new policy.

I changed it and try again

No problem on my side

@waruqi
Copy link
Member

waruqi commented Apr 26, 2024

我 merge 了, xmake update -s dev

@waruqi waruqi closed this as completed Apr 26, 2024
@waruqi waruqi added this to the v2.9.2 milestone Apr 26, 2024
@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically.


I merged, xmake update -s dev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants