Skip to content

fiograph

Jump to bottom
Vincent Fu edited this page Sep 19, 2023 · 2 revisions

Adam Manzanares provided helpful feedback on this blog post.

I grew up as an avid reader and usually find the written word to be more useful than charts or diagrams. However there are instances where even I find diagrams genuinely useful. One such instance is the job file diagrams drawn by fiograph. Here is an example job file with an accompanying diagram:

[global]
ioengine=null

[readjob]
filesize=1G
numjobs=4
rw=read
group_reporting=1

[writejob]
filesize=1T
io_size=1G
rw=write
exec_prerun=echo "exec_prerun example"

[rwjob]
stonewall
rw=randrw
filesize=1M
time_based
runtime=1m
exec_postrun=echo "exec_postrun example"

example

Fio job files can be difficult for humans to parse because the sequencing of jobs can be easily missed. fiograph provides a display that graphically communicates the main features of fio job files. The jobs in the first grey box all run simultaneously and when they complete the job in the second grey box begins running. Each grey box denotes a separate execution group. The red x 4 indicates that fio will actually be running four instances of the job named readjob. Generic options are displayed in green and ioengine-specific options are displayed in blue. exec_prerun and exec_postrun options are highlighted in red and respectively positioned above and below other options. All of the options relevant to a given job are bound within rounded blue boxes headed by the job's name. The top two jobs stop when a specified number of bytes is written but the bottom job is time_based as indicated by the self-arrow and specified run time. The colors and styles used for this diagram are the defaults specified in fiograph's configuration file described below.

This blog post will provide a brief introduction to fiograph which was contributed to the fio repository in June 2021 by Erwan Velu via a GitHub pull request. This discussion is based on the version of fiograph distributed with fio 3.33.

How to use fiograph

fiograph is a Python script. It generates a graphviz file to render the plot and hence requires graphviz to be installed. We need both the graphviz executable as well as graphviz Python support. Below is an example of installing the needed dependencies on a Debian-based system. For good system Python hygiene let us use a virtual environment for the Python package:

root@ubuntu:~/fio-dev# apt install graphviz -y
Reading package lists... Done
...
root@ubuntu:~/fio-dev# python3 -m venv venv
root@ubuntu:~/fio-dev# source venv/bin/activate
(venv) root@ubuntu:~/fio-dev# pip3 install graphviz
Collecting graphviz
  Downloading graphviz-0.20.1-py3-none-any.whl (47 kB)
     |################################| 47 kB 5.6 MB/s
Installing collected packages: graphviz
Successfully installed graphviz-0.20.1
(venv) root@ubuntu:~/fio-dev# ./fio-canonical/tools/fiograph/fiograph.py --help
usage: fiograph.py [-h] --file FILE [--output OUTPUT] [--format FORMAT] [--view] [--keep] [--config CONFIG]

optional arguments:
  -h, --help       show this help message and exit
  --file FILE      the fio file to graph
  --output OUTPUT  the output filename
  --format FORMAT  the output format
  --view           view the graph
  --keep           keep the graphviz script file
  --config CONFIG  the configuration filename

The one required argument is a fio job file for fiograph to process. This works best when using fio job files with the .fio file extension. The default output will be an image file with the same name as the job file but with a .png file extension. Because of a bug in the script, job files without a .fio file extension will be deleted if an output filename is not explicitly specified. See the notes at the end for details.

The example diagram above was constructed via this command:

(venv) root@ubuntu:~/fio-dev/fio-canonical/tools/fiograph# python3 fiograph.py --file example.fio

By default the resulting diagram is placed in the same directory as the specified job file. As described by the help text, the other options for the script are:

  • The --output option can be used to specify an alternative destination for the output image.
  • By default fiograph will generate PNG files. Other formats such as bmp, gif, and jpg can be selected with the ---format option (available choices are at https://graphviz.org/docs/outputs/).
  • On platforms with support it is also possible to directly open the generated diagram for display with the --view option.
  • The --keep option will retain the graphviz script instead of deleting it.
  • Finally, a configuration file other than the default can be specified with the --config option.

fiograph configuration file

The fiograph configuration file specifies the colors and styles to use for the different diagram components as well as the list of engine-specific options. Click below to see an excerpt:

fiograph.conf 
[fio_jobs]
header=<<B><font color="{}"> {} </font></B> >
header_color=black
text_color=darkgreen
shape=box
shape_color=blue
style=rounded
title_style=<<table border='0' cellborder='0' cellspacing='1'> <tr> <td align='center'> <b> {} </b> </td> </tr>
item_style=<tr> <td align = "left"> <font color="{}" > {} </font> </td> </tr>
cluster_style=filled
cluster_color=gainsboro

[exec_prerun]
text_color=red

[exec_postrun]
text_color=red

[numjobs]
text_color=red
style=<font color="{}" > x {} </font>

[ioengine]
text_color=darkblue
specific_options_color=darkblue

# definitions of engine's specific options

[ioengine_cpuio]
specific_options=cpuload cpumode cpuchunks exit_on_io_done

[ioengine_dfs]
specific_options=pool  cont  chunk_size  object_class  svcl
...

The first several sections specify the colors and styles for components of the diagrams. The final set of sections list engine-specific options for each ioengine.

Another example

Here is a more complicated example using the e4defrag2.fio example job file from the fio repository. This ioengine uses the e4defrag ioengine to simulate defragment activity. It shows seven different fio job configurations arranged into four execution groups. Each of the execution groups simulates a different type of defragmentation activity. Carefully studying the job file and its comments and noting the placement of the stonewall directives can produce a mental picture of how the job configurations are arranged into execution groups but the diagram makes the job arrangement easier to grasp. The ioengine-specific options are also helpfully highlighted in blue. The amount of data to be written and runtime limits are also highlighted via self-arrows for each job configuration. The value for numjobs is also included in the title for each job configuration when its value is specified.

e4defrag2.fio 
#################################################
# Hardcode defragmentation patterns
# Please be careful, it can trigger kernel panic
#################################################
[global]
ioengine=e4defrag
group_reporting
directory=/scratch
nrfiles=1
filesize=100M
size=100M
donorname=file.def
bs=32k

###########
# Run several defragmentation threads for different files, but
# use shared donor file
[parallel-e4defrag]
buffered=0
inplace=0
rw=write
numjobs=4

########
# Run two defragmentation threads, each thread use another's file
# as donor file

[e4defrag-1]
stonewall
inplace=0
rw=write
donorname=e4defrag-2

[e4defrag-2]
inplace=0
rw=write
donorname=e4defrag-1

###########
# Run random defragment activity
[e4defrag-fuzzer-4k]
stonewall
inplace=1
bs=4k
rw=randwrite
filename=file
donorname=file.def

########
# Run random e4defrag and various aio workers in parallel
[e4defrag-fuzzer-4k-bis]
stonewall
continue_on_error=all
inplace=1
bs=4k
donorname=file3.def
filename=file3
time_based
rw=randwrite

[buffered-aio-32k]
continue_on_error=none
verify=md5
buffered=1
ioengine=libaio
iodepth=128
bs=32k
filename=file3
rw=randrw
runtime=30
time_based
numjobs=4

[direct-aio-32k]
continue_on_error=none
verify=md5
buffered=0
direct=1
ioengine=libaio
iodepth=128
bs=32k
filename=file3
rw=randrw
runtime=30
numjobs=4

e4defrag2

Summary and notes

I have found fiograph to be a useful tool and hope this discussion allows more users to benefit from it. Fio's job options can produce complicated configurations and the diagrams drawn by fiograph make the job flow easier to comprehend.

Below I conclude with a series of notes.

  • As mentioned above, if fiograph is instructed to process a job file that does not end with the .fio file extension it will overwrite the job file with the graphviz script file and then (unless the --keep option is specified) delete this file afterwards. fiograph should instead use a temporary file for the output. Fixed by this patch with discussion on GitHub.

  • fiograph does highlight size and runtime as stopping criteria for fio jobs but should also do the same for io_size, io_limit, and number_ios.

  • Updating the default configuration file with ioengine options currently must be done manually. It would be helpful to automate this.

  • How can we keep the fiograph diagrams committed to the fio repository in the fio repository's examples directory updated? We should consider some sort of makefile for the diagrams.

  • To create the default output file name, fiograph takes the input file name and replaces all instances of .fio in the name with an empty string. This should only be done if .fio is at the end of the input file name. Addressed via this patch

  • If no configuration file is specified on the command line fiograph assumes that the configuration file is in the current directory. So if the current working directory is not the source directory and a configuration file is not explicitly provided, runs will fail. fiograph should consider the script location directory as a potential location for the default configuration file. Addressed via this patch.

  • The help text should provide a link to the list of available output formats for the --format option https://graphviz.org/docs/outputs/. Added via this patch.

Clone this wiki locally