TracerX setup on Test-Comp under Ubuntu22

[rasoolmaghareh]

We will discuss the steps and issues for running TracerX on Test-comp here.

[rasoolmaghareh]

Steps from @yxliang01

Note this instruction aims to recreate a very similar environment (to the extent possible as per descriptions the orgranizers have given) to test-comp23, so the steps are bit different from Rasool's previous steps which aimed to setup for the lastest development version.

1. Ubuntu22 server edition
2. git clone --branch=testcomp23 --depth=1 --shallow-submodules --recurse-submodules https://gitlab.com/sosy-lab/test-comp/bench-defs
3. Follow the dockerfile at scripts/test/Dockerfile.user.2023 for the list of packages and perform apt install accordingly
4. cd bench-defs/
5. pip3 install --user --requirement=scripts/requirements.txt
6. pip3 install --user ./benchexec/
7. echo 'PATH=$PATH:$HOME/.local/bin/' >> ~/.bashrc
8. Follow benchexec/doc/INSTALL.md section "Setting up Cgroups" and "Setting up Cgroups on Machines with systemd"
9. sed -i 's/--zipResultFiles//g' scripts/configure.sh
10. sudo usermod -aG benchexec id --user --name
11. reboot and all done

Notes

1. test-comp seems to use the server edition of Ubuntu22. Using desktop edition in theory can work, but it's known that both of them have different default settings. To the best of my knowledge, using desktop edition might be related to some of our earlier failure and they use different default kernel. So, you might need to do more if using the desktop edition.
2. the steps above should be done using normal user (not root)
3. requirement: cpu execution threads >= 8, RAM (not counting swap) >= 15GB, better to have storage >= 50GB
4. To Rasool: step 9 is to resolve a potential bug in their script, however, in your instructions, you seem to not have this issue. May I confirm this (to know if I've made wrong patch to their codebase)?
5. I've ignored the optional setup for benchexec as I work with hardware that doesn't seem to support them and it's of limited power. But they can make benchmarking more reliable, so I would recommend to try them.
6. I recommend not to run on virtual machines since it requires correct timer/cache settings etc to make the results reliable and I'm unsure how yet. But, if you've managed to achive reliable benchmarking in VMs, it would be great if you can share.
7. Using PPA to install benchexec is easier, but the scripts were developed for a version that seems to be unavailable on PPA. So, I recommend following the steps above to install benchexec instead.
   Let me know if you've any questions, XL

[rasoolmaghareh]

Regarding this point:

To Rasool: step 9 is to resolve a potential bug in their script, however, in your instructions, you seem to not have this issue. May I confirm this (to know if I've made wrong patch to their codebase)?

I didn't have this issue.

[ArpitaDutta]

Hi @yxliang01,

As per your instructions, I have installed the Ubuntu22 server version for testcomp23.
While installation I have chosen the default options for the 1. Network connections, 2. Storage configurations and 3. SSH setup.
I have not done any manual settings.

But now while performing the git clone --branch=testcomp23 --depth=1 --shallow-submodules --recurse-submodules https://gitlab.com/sosy-lab/test-comp/bench-defs
The server is throwing the following error:
fatal: unable to access 'https://gitlab.com/sosy-lab/test-comp/bench-defs/': Could not resolve host: gitlab.com

Do you have any suggestions to resolve this?

[rasoolmaghareh]

It looks like a network or access issue. Try to see if this link https://gitlab.com/sosy-lab/test-comp/bench-defs/ can be opened in a browser.

[ArpitaDutta]

Yes, it was due to the network issue.

[ArpitaDutta]

A small update in the instructions given by XL.

For Step 3, we need to follow the dockerfile at scrtips/test/Dockerfile.test to get the list of packages using apt install.

[rasoolmaghareh]

Were you able to fully install it?

[ArpitaDutta]

The installation is almost completed.

However, for most of the tools I am getting the following warning/error messages.

• WARNING - CPU throttled itself during benchmarking due to overheating. Benchmark results are unreliable!
  unknown
• WARNING - System has swapped during benchmarking. Benchmark results are unreliable!
  done
• Out of memory or time out.

[ArpitaDutta]

I am trying to fix these issues with the help of XL.

[ArpitaDutta]

These are the installation steps for TestComp-23 locally. The instructions are taken from XL email. I have made additions to Steps 3 and 4. The Step 3 and Step 4 installations are related to dependencies and they are of utmost importance.

1. Use Ubuntu22 server edition ( I have used Ubuntu Server 22.04.3 LTS at my end )
   

2. git clone --branch=testcomp23 --depth=1 --shallow-submodules --recurse-submodules https://gitlab.com/sosy-lab/test-comp/bench-defs
   

3. Follow the dockerfile at scripts/test/Dockerfile.user.2023 for the list of packages and perform apt install accordingly
   

4. We need to follow the dockerfile at scrtips/test/Dockerfile.test to get the list of packages using apt install         [Step 3 and 4 are most important for getting all the dependencies for the testcomp23 setup]
   

5. cd bench-defs/
   

6. pip3 install --user --requirement=scripts/requirements.txt
   

7. pip3 install --user ./benchexec/
   

8. echo 'PATH=$PATH:$HOME/.local/bin/' >> ~/.bashrc
   

9. Follow benchexec/doc/INSTALL.md section "Setting up Cgroups" and "Setting up Cgroups on Machines with systemd"
   

10. sed -i 's/--zipResultFiles//g' scripts/configure.sh
    

11. sudo usermod -aG benchexec id --user --name
    

12. reboot and all done