v1.0.2-ae (Artifact Evaluation Release)

🚀 Overview

This is an updated Artifact Evaluation (AE) release of our paper artifact. The repository provides a complete, containerized environment and automated scripts to reproduce the performance comparison experiments across multiple verification methodologies and hardware targets.

In this update, we improved the AE workflow for reviewers by:

• refining artifact-size accounting for python-mem_direct to include the generated YAML offset file,
• adding grouped matplotlib figures for both Group A and Group B,
• integrating figure generation into the one-click scripts,
• improving Docker support for headless plotting,
• and simplifying reviewer-facing output collection with explicit file paths and docker cp commands.

🛠 Key Features

• Experiment Group A:
  Comprehensive comparison among cocotb, python-dpi, python-vpi, and python-mem_direct.

• Experiment Group B:
  Multilang performance analysis across raw-verilator, python, cpp, java, and golang.

• Hardware Targets:
  Supported DUTs include Rocket (SimTop), coupledL2 (TestTop), and XS (SimTop).

• Optimized for AE:

  • Dockerized Environment:
    Fully containerized setup via ghcr.io/makiras/unitychipexp:latest or the local docker/Dockerfile.
  • Headless Plotting Support:
    The Docker image now includes matplotlib configured with the Agg backend, so PNG figures can be generated directly inside the container without any GUI/X11 environment.
  • One-Click Outputs:
    The wrapper scripts now automatically:
    • run the experiments,
    • extract metrics,
    • render CLI summaries,
    • generate overview PNG figures,
    • print the in-container output paths,
    • and print example docker cp commands for copying results back to the host.

📊 Result Outputs

After running the one-click scripts, the following outputs are generated automatically:

• Extracted metrics:

  • results/extracted/group_a/...
  • results/extracted/group_b/...

• CLI summary plots:

  • group_a_cli_plot.txt
  • group_b_cli_plot.txt

• Matplotlib overview figures:

  • results/plots/.../group_a_overview.png
  • results/plots/.../group_b_overview.png

Notes on plotting:

• Group A is rendered as a single figure with 4 subplots:
  • Build CPU Time
  • Binary Size
  • Simulation Speed
  • Simulation Memory
• Group B is rendered as a single figure with 2 subplots:
  • Simulation Speed
  • Simulation Memory
• Within Group A, python-mem_direct is shown as the leftmost variant for consistency across plots.
• Bars are normalized within each DUT group, matching the CLI view.

📋 Quick Start for Reviewers

sudo docker pull ghcr.io/makiras/unitychipexp:latest
sudo docker run -it ghcr.io/makiras/unitychipexp:latest

cd /home/xyl/exp

# Quick Check
./scripts/A_without_XS.sh
./scripts/B_without_XS.sh

# Full Check (more than 1 day)
# ./scripts/A_with_XS.sh
# ./scripts/B_with_XS.sh

📦 Copying Results from Docker

At the end of each one-click script, the container prints the generated file paths and example docker cp commands.

If you want to copy files manually, replace with your container ID or name:

#### IN ANOTHER SHELL ####
  docker cp <container>:/home/xyl/exp/results/plots/group_a/group_a_overview.png ./
  docker cp <container>:/home/xyl/exp/results/plots/group_b/group_b_overview.png ./
  docker cp <container>:/home/xyl/exp/results/extracted/group_a ./group_a
  docker cp <container>:/home/xyl/exp/results/extracted/group_b ./group_b

✅ Reproducibility Notes

• The repository is intended to support both quick validation and full reproduction.

• Existing build artifacts are reused when possible.

• Required artifacts are rebuilt automatically if missing.

• Metrics are extracted from the current run logs.

• The scripts use fixed taskset CPU sets for reproducibility.

• Group A expects at least 8 physical cores available.

• If your environment cannot honor the pinned CPU set, rerun with:

  PIN_RUNTIME=0 ./scripts/A_without_XS.sh

📚 Recommended Reviewer Path

1. Pull and run the Docker image.
2. Execute:
   • ./scripts/A_without_XS.sh
   • ./scripts/B_without_XS.sh
3. Inspect:
   • results/extracted/.../metrics.csv
   • results/extracted/.../group_*_cli_plot.txt
   • results/plots/.../group_a_overview.png
   • results/plots/.../group_b_overview.png

This AE release is designed to minimize setup effort and provide both raw metrics and reviewer-friendly visual summaries in a single reproducible workflow.

v1.0.1-ae (Artifact Evaluation Release)

🚀 Overview
This is the initial release for the Artifact Evaluation (AE) of our paper. This repository provides a complete environment and automated scripts to reproduce the performance
comparison experiments between different verification methodologies.

🛠 Key Features

• Experiment Group A: Comprehensive comparison among cocotb, python-dpi, python-vpi, and python-mem_direct.
• Experiment Group B: Multilang performance analysis across raw-verilator, python, cpp, java, and golang.
• Hardware Targets: Supported DUTs include Rocket (SimTop), coupledL2 (TestTop), and XS (SimTop).
• Optimized for AE:
  • Dockerized Environment: Fully containerized setup via ghcr.io/makiras/unitychipexp:latest or local Dockerfile for a "one-click" experience.

📋 Quick Start for Reviewers

1. Pull Docker Image: sudo docker pull ghcr.io/makiras/unitychipexp:latest
2. Run Experiments:
   • Follow the Reproduction Guide in README.md.
   • Use ./scripts/run_experiments.py to automate the benchmarking process.
3. Analyze Results: Metrics and logs are automatically generated in the results/ directory.

sudo docker pull ghcr.io/makiras/unitychipexp:latest
sudo docker run --rm -it ghcr.io/makiras/unitychipexp:latest

cd /home/xyl/exp
# It may be necessary to adjust the taskset CPU core number.
# Quick Check
./scripts/A_without_XS.sh
./scripts/B_without_XS.sh
# Full Check ( more than 1 days )
# ./scripts/A_with_XS.sh
# ./scripts/B_with_XS.sh

v1.0.0-ae (Artifact Evaluation Release)

🚀 Overview
This is the initial release for the Artifact Evaluation (AE) of our paper. This repository provides a complete environment and automated scripts to reproduce the performance
comparison experiments between different verification methodologies.

🛠 Key Features

• Experiment Group A: Comprehensive comparison among cocotb, python-dpi, python-vpi, and python-mem_direct.
• Experiment Group B: Multilang performance analysis across raw-verilator, python, cpp, java, and golang.
• Hardware Targets: Supported DUTs include Rocket (SimTop), coupledL2 (TestTop), and XS (SimTop).
• Optimized for AE:
  • Dockerized Environment: Fully containerized setup via ghcr.io/makiras/unitychipexp:latest or local Dockerfile for a "one-click" experience.

📋 Quick Start for Reviewers

1. Pull Docker Image: sudo docker pull ghcr.io/makiras/unitychipexp:latest
2. Run Experiments:
   • Follow the Reproduction Guide in README.md.
   • Use ./scripts/run_experiments.py to automate the benchmarking process.
3. Analyze Results: Metrics and logs are automatically generated in the results/ directory.