Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep prebuilt ELF files up to date #451

Open
vacantron opened this issue Jun 5, 2024 · 5 comments
Open

Keep prebuilt ELF files up to date #451

vacantron opened this issue Jun 5, 2024 · 5 comments
Assignees

Comments

@vacantron
Copy link
Collaborator

Due to #444 , we need to update the binary with the new toolchain.

@vacantron vacantron self-assigned this Jun 5, 2024
@jserv
Copy link
Contributor

jserv commented Jun 5, 2024

I think it is time to handle the prebuilt executable files in a more elegant way. The proposed change is to separate the executable files existing in the build directory from the rv32emu repository. This means we will have two repositories: rv32emu and rv32emu-prebuilt. The former will preserve all the source files and build scripts (or recipes), while the latter will maintain the prebuilt executable files. These prebuilt files will be triggered by GitHub Actions once the CI pipeline detects changes in the corresponding files of the rv32emu repository as rv32emu-bench does. The advantage of the proposed change is that it includes a lean repository footprint and is suitable for future system emulation. In other words, it will not be necessary to fetch bare-metal ELF files for system emulations.

The expected flow:

  1. Create recipes for building ELF files from source files and/or external repositories. By default, the optimization level -O2 should be applied, and the architecture flags such as rv32i, rv32im, rv32imac, etc., would be passed depending on the configurations of the recipes describing how an ELF file should be built from the source.
  2. The CI pipeline detects changes to the above recipes for monitoring test files and triggers the build for these, then uploads the ELF files to the rv32emu-prebuilt repository.
  3. When the rv32emu repository is first checked out, there are no prebuilt ELF files, and the check target of the build system would perform git clone to get the prebuilt ELF files with the --depth=1 option for faster download, and then run the prebuilt executable files.
  4. By default, the generation of ELF files comes with strip to lower the file size, but some files should be excluded from the strip list. For example, if we want to validate remote GDB, the file should have debugging symbols.

@ChinYikMing did some prior work for rv32emu-bench.

@jserv jserv changed the title Update prebuilt testbenches Keep prebuilt ELF files up to date Jun 5, 2024
@ChinYikMing
Copy link
Collaborator

ChinYikMing commented Jun 5, 2024

In this work flow, the deploy-wasm CI will be affected such that it cannot monitor that whether the built elf has been changed. The possible adaption could be monitor the source files of the executable files. If the source files change, the deploy-wasm CI can use the needs keyword in the CI workflow to wait the corresponding executable built CI runs and pushes executable files to rv32emu-prebuilt repository. After that, deploy-wasm CI fetches the latest executable files from rv32emu-prebuilt repository.

@jserv
Copy link
Contributor

jserv commented Jun 11, 2024

The file tests/arch-test-target/sail_cSim/riscv_sim_RV32 should be built via the above automated flow to have the linux-x86_64 binary accordingly.

@henrybear327
Copy link
Collaborator

henrybear327 commented Jun 13, 2024

The file tests/arch-test-target/sail_cSim/riscv_sim_RV32 should be built via the above automated flow to have the linux-x86_64 binary accordingly.

#455 is also impacted by this elf recompilation delay.

I have done the compiler packaging and recompiling sail reference model before actually, and it's done in a container, so I can quickly help on this matter.

However I recall that using a newer toolchain to build rv32emu at the time would cause the tests to fail, also, building a reference model using the latest dependencies would cause some tests to fail, too. So there might be some work to do...

All the dockerfile that I worked on is under the directory docker.

I will update both Dockerfiles and push to DockerHub in the meantime, as #455 requires updating the compiler toolchain and recompiling the benchmark elfs, otherwise, our benchmarking CI is currently broken.

@henrybear327 henrybear327 self-assigned this Jun 13, 2024
@henrybear327
Copy link
Collaborator

henrybear327 commented Jun 15, 2024

Latest toolchain image and sail reference model image have been updated to DockerHub

I will create 2 PRs

  • push the updated dockerfiles that are used to generate DockerHub images
  • push all the updated elf binaries affecting the benchmarking CI in the build directory

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants