This document provides detailed instructions for reproducing the experiment presented at ACM Symposium on Access Control Models and Technologies 2026.
./
|-- assets/ # Entities, policies, and obligations for the experiment
|-- crates/ # Rust source code for all benchmarked components
| |-- tinytodo-cedar # TinyTodo implementation using Cedar (from the Cedar authors)
| |-- tinytodo-strobilus # TinyTodo implementation using Strobilus
|-- .dockerignore
|-- .gitignore
|-- benchmarks.py # Benchmarks execution script
|-- Cargo.toml
|-- Cargo.lock
|-- docker.sh # Bash helper script for build and run Docker container
|-- Dockerfile # Docker container definition
|-- graphs.py # Graphs generation script
|-- LICENSE
|-- Makefile # Makefile commands
|-- NOTICE
|-- README.md
|-- requirements.txt # Python dependencies
In this section, we describe how to reproduce the results presented in Section 6 ("Strobilus: Performance Evaluation") of the paper.
The experiments were conducted on an Intel Ultra 7 155U (14 cores) at 4.80 GHz with 16 GB DDR5 SDRAM (5600 MT/s), running Linux 6.14.0 on Ubuntu 24.04.4 LTS.
Our evaluation procedure consists of M = 1000 experimental runs. Each run executes a sequence of N = 400 requests:
- 100 calls to
create_list; - 100 calls to
get_list; - 100 calls to
update_list; - 100 calls to
delete_list.
These parameters are defined in the benchmarks.py file (lines 27-28). The default values correspond to those used in the paper.
With this configuration:
- On bare metal, the experiment takes approximately 2/3 hours.
- Using Docker, it takes approximately 3/4 hours.
To reproduce the experiment, clone the repository using the following command:
git clone https://github.com/strobilus-lang/strobilus-tinytodo-example
You can prepare the system for the experiments in two ways: with a Docker container or directly on your OS installation. The two methods are equivalent, though Docker is preferred for reproducibility.
A ready-to-use Docker container is provided.
To build and start the container, run: ./docker.sh.
This script installs all required dependencies and creates the Python virtual environment.
- Install the following system packages:
makegcccurlwgetrust(i.e.,rustcandcargo; follow the instructions here: https://rust-lang.org/tools/install/)python3(i.e.,python3,pipandvenv)
- Create the Python virtual environment:
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
To execute the experiment, run: make experiments.
This command executes the benchmarks.py script to measure execution times and the graphs.py script to generate the corresponding plots.
Note: You may encounter some Rust warnings related to unused variables and functions.
These do not affect the experiments or execution. They have been left in place deliberately rather than removed in order to keep the TinyTodo code as close as possible to the original version from Cedar repository.
Below is an example of the expected output:
python benchmarks.py
> Starting Tinytodo Cedar benchmark # Running the Cedar implementation
> Progress: 1/M # Current experiment run
>> Iteration Create: 1/N # Current iteration for specific function
>> Iteration Retrieve: 1/N
>> Iteration Update: 1/N
>> Iteration Delete: 1/N
>> Killing PID 30106 # Process terminated after completion
...
> Done # Cedar benchmark completed
> Starting Tinytodo Strobilus benchmark # Running the Strobilus implementation
> Progress: 1/M
>> Iteration Create: 1/N
>> Iteration Retrieve: 1/N
>> Iteration Update: 1/N
>> Iteration Delete: 1/N
>> Killing PID 30319
...
> Done
python graphs.py
>> Prepare "create_list" data # Parsing raw data from the `results` directory
...
>> Generate "create_list" plot # Generating graphs for each function
>>> U statistic: ...
>>> P-value: ...
>> Average improvement: ...
>> Best improvement: ...
>> Worst improvement: ...
>> Done
...
After execution, two directories will be generated:
...
|-- results/ # Raw benchmark outputs (CSV files)
| |-- tinytodo-cedar
| | |-- create_list.csv
| | |-- delete_list.csv
| | |-- get_list.csv
| | |-- update_list.csv
| |-- tinytodo-strobilus
| | |-- create_list.csv
| | |-- delete_list.csv
| | |-- get_list.csv
| | |-- update_list.csv
|-- graphs/ # Generated graphs (image files, e.g. PNG)
| |-- create_list.png
| |-- crud.png
| |-- delete_list.png
| |-- get_list.png
| |-- update_list.png
...
Specifically:
- The
./results/folder contains the raw benchmark data; - The
./graphs/folder contains graphs generated from the benchmark data.
Each run produces N measurements, and the experiment is repeated M times.
Therefore, each result file contains M × N rows corresponding to the execution times measured during the experiment.
The graphs directory contains five images:
create_list: comparison between Strobilus and Cedar for thecreate_listoperation;get_list: comparison for theget_listoperation;delete_list: comparison for thedelete_listoperation;update_list: comparison for theupdate_listoperation;crud: combined visualization of all operations, corresponding to Figure 6 in Section 6 of the paper.
The results should reflect the performance trends described in the paper.
In particular:
- In the
create_listscenario, Strobilus shows an average performance improvement of approximately 67% over Cedar; - In the remaining scenarios, the average improvement exceeds 80%, with the lowest observed improvement around 71%.
Due to system variability, deviations of approximately ±10-20% are acceptable. However, the overall behavior should remain consistent with Figure 6.
-
Rust installation fails
Ensure that your internet connection is active and follow the official Rust installation guide. -
Benchmarks terminate immediately
Some requests may be blocked by the host system. Try running the experiment again. -
Graphs are empty
Ensure that theresults/directory contains valid data and that all files are correctly generated.
This project is licensed under the Apache-2.0 License.