Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CPL-7730: Fission support #73

Merged
merged 15 commits into from
Feb 17, 2023
Merged

CPL-7730: Fission support #73

merged 15 commits into from
Feb 17, 2023

Conversation

OliLay
Copy link
Collaborator

@OliLay OliLay commented Jan 31, 2023

Adds fission support for homcc:

  • Detect the -gsplit-dwarf flag
  • If given, send back the dwo file (for each object file) from the server to the client using the CompilationResultMessage
  • There were some changes that were needed on the server side to make fission possible: as the path to the .dwo files inside the object files depends on the output arguments (-o), and previously we were ignoring the output argument on the server side, we needed to include output argument handling on the server. Also, output paths must be relative so that the path inside the object files is also relative and not absolute (which would break on the client then). The server now also directly returns the path for the client to save the object file, the special handling inside the client has been removed.

E2e tests and a unit tests for utilities are included.

Compatibility should be theoretically ensured for both: older clients connecting to newer servers and the other way around as we just added a field. There could be issues with compatibility regarding the changed output path behavior, I have to think about that again. So before we release, we should test compatibility.

Below are some findings on the initial implementation that prevented it from working, I keep it here as it may be useful at a later point in time:

Steps to reproduce:

python ./homcc/client/main.py --host 127.0.0.1:1337 clang++ -Iexample/include example/src/foo.cpp -gsplit-dwarf -g -o result/foo_redirected.o -c
python ./homcc/client/main.py --host 127.0.0.1:1337 clang++ -Iexample/include example/src/main.cpp -gsplit-dwarf -g -o result/main_redirected.o -c
# clang
clang++ -Iexample/include example/src/main.cpp -gsplit-dwarf -g -o result/main_redirected_clang.o -c
clang++ -Iexample/include example/src/foo.cpp -gsplit-dwarf -g -o result/foo_redirected_clang.o -c

cd result
clang++ foo_redirected_clang.o main_redirected_clang.o -o bin/clang
python ./../homcc/client/main.py --host 127.0.0.1:1337 clang++ foo_redirected.o main_redirected.o -o bin/homcc

cd bin
gdb clang # works as expected
gdb homcc # yields error below

GDB error in case of using this with homcc

warning: Could not find DWO CU foo.dwo(0x25d9ef127f452be5) referenced by CU at offset 0x0 [in module /home/o.layer/homcc/result/bin/homcc]
Dwarf Error: unexpected tag 'DW_TAG_skeleton_unit' at offset 0x0 [in module /home/o.layer/homcc/result/bin/homcc]

Inspecting the object files with readelf yields the following:

(venv) o.layer@cellap:~/homcc/result$ readelf -wi foo_redirected_clang.o | grep dwo
readelf: Warning: Unrecognized form: 0x23
readelf: Warning: Unrecognized form: 0x23
    <1e>   DW_AT_dwo_name    : (indexed string: 0x1): result/foo_redirected_clang.dwo
(venv) o.layer@cellap:~/homcc/result$ readelf -wi foo_redirected.o | grep dwo
readelf: Warning: Unrecognized form: 0x23
readelf: Warning: Unrecognized form: 0x23
    <1e>   DW_AT_dwo_name    : (indexed string: 0x1): foo.dwo
(venv) o.layer@cellap:~/homcc/result$ 

Note for the object file generated by homcc, the name stored in DW_AT_dwo_name is not correct. This is because we compile the cpp file on the server side without the output flag, and therefore the .dwo file has a different name it refers to.

The only possible solution that comes to my mind is to move to -o logic to the server, and also use -o on the server. The client then just saves the .o/.dwo file under the path returned from the server and does not need to have any -o file logic any more.

@OliLay OliLay requested a review from spirsch January 31, 2023 08:27
@OliLay OliLay marked this pull request as draft February 1, 2023 07:19
homcc/client/compilation.py Outdated Show resolved Hide resolved
homcc/client/compilation.py Outdated Show resolved Hide resolved
homcc/client/compilation.py Outdated Show resolved Hide resolved
homcc/common/messages.py Outdated Show resolved Hide resolved
homcc/client/compilation.py Outdated Show resolved Hide resolved
homcc/common/arguments.py Outdated Show resolved Hide resolved
tests/e2e/e2e_test.py Outdated Show resolved Hide resolved
@OliLay OliLay requested a review from spirsch February 16, 2023 10:44
@OliLay OliLay marked this pull request as ready for review February 16, 2023 10:44
homcc/common/arguments.py Outdated Show resolved Hide resolved
homcc/common/messages.py Outdated Show resolved Hide resolved
homcc/common/messages.py Outdated Show resolved Hide resolved
homcc/common/messages.py Outdated Show resolved Hide resolved
tests/server/environment_test.py Outdated Show resolved Hide resolved
tests/server/environment_test.py Outdated Show resolved Hide resolved
homcc/server/environment.py Outdated Show resolved Hide resolved
homcc/server/environment.py Outdated Show resolved Hide resolved
@OliLay OliLay requested a review from spirsch February 17, 2023 08:11
tests/server/environment_test.py Outdated Show resolved Hide resolved
@OliLay OliLay merged commit ed239e1 into main Feb 17, 2023
@OliLay OliLay deleted the CPL-7730-fission branch February 17, 2023 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants