pbench-fio dependency on python-pandas

[bengland2]

Karl Cronburg wrote fiologparser_hist.py to postprocess latency histogram logs collected by fio --client so that we could get latency percentile variation over time for an entire cluster, but he wrote it with a dependency on python-pandas, which he thought greatly improved the efficiency and simplicity of the code. When I go to install pbench-fio RPM that contains it, pbench-fio RPM fails because of this dependency. When I install python pandas from EPEL, it pulls in 75 RPMs which include everything from X-windows to tex to you-name-it.

I propose that we remove the dependency on python-pandas and simply put a test into fiologparser_hist.py such that if python-pandas is not installed (i.e. the "import pandas" fails, then it tells you to install it. This tool is not intended to be run on every workload generator host, just on the one that fio --client runs on.

Do people agree with this approach? If not, what approach should we take? This change would have to be pushed upstream to fio repo.

[ndokos]

How about this:

• Delete the dependency in the RPM.
• Modify the pbench-fio script to check_install python-pandas (as well as the pbench-fio RPM) on the controller only.
• No change to upstream fio: if somebody tries to run the fiologparser_hist.py script without the module, they get the standard "cannot import pandas" python error message from the exception.

WDYT?

[bengland2]

sounds good to me. By the "controller" you mean the host where pbench-fio is launched, not the workload generators. I don't think there is much problem having the controller with some extra RPMs, but I get nervous about having all those RPMs required everywhere that pbench runs.

[ndokos]

python-pandas is available from the standard Fedora repos, but not from the standard RHEL repos: the EPEL repos have to be installed (on the controller only) in this case. The release notes for v0.40 contain the following instructions:

on your RHEL7 controller (and similarly for RHEL6)

cd /tmp
curl --output ./epel-release-latest-7.noarch.rpm  https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install ./epel-release-latest-7.noarch.rpm

After that, the pbench-fio script should be able to find and install the python-pandas package.

Apart from this complication, PR #383 implements this; closing the issue.