Calling from PoCL's LLVM pass chain to instrument OpenCL SPIR-V/C kernels (for CPU devices)

[pjaaskel]

Hi,

I'm investigating if Verificarlo's instrumentation could be integrated to PoCL's work-group generation passes so that it could be used to explore FP arithmetic simplifications for OpenCL and SYCL kernels. My initial idea is to simply call the instrumentation module pass from the PoCL's WG generation pass chain (the final pass). I noticed that the verificarlo script also uses a "vfcwrapper", which is a generated .c file. What is it needed for and why it's compiled in from the source, not linked in, to the final program? Any other tips for the best way to proceed with this trial?

Thanks,
Pekka

[pjaaskel]

It seems to be built to .ll so the instrumenter can get LLVM function handles to it, but is that really needed, could it just construct the function declarations on the fly?

[pablooliveira]

In verificarlo, we compile the wrapper as a bitcode file to avoid issues between different code-generation flags between the user code and the wrapper. In some architectures we had ABI problems with vector types when different options were used between the user code and the wrapper (see #244).

[pablooliveira]

Hi Pekka,

I have helped @vincentAlba to use Verificarlo passes within the MLIR code generation chain; so maybe something similar is possible within PoCL. It sounds interesting !

Initially I think you do not care about function instrumentation; so you just need to call the libvfcinstrument.so -vfclibinst pass as you propose. Since we are still using the legacy pass manager, you need to pass -enable-new-pm=0 for now.

You are correct that verificarlo requires a wrapper library. The wrapper is compiled (at instrumentation time) by the verificarlo script for two reasons:

1. The interface uses the LLVM C vector types extension which needs to be compiled with clang. Moreover, the wrapper uses library constructors for its initialization. When we first developed it, clang did not support well .so library constructors; therefore we decided to compile it as an object file .o and link it directly against the user program.
2. Some passes such as delta-debug, require enabling some compile time macros on the wrapper. This allows us to have costly code pathes removed at compile time when they are not needed (instead of paying a costly branch for every instrumented instruction).

What @vincentAlba did in MLIR is compile the wrapper once as a .o object file and include it in the final link command. This worked well for him.

I hope this helps you a bit...

Cheers,
Pablo

[pablooliveira]

Currently, I think the driver script does not apply verificarlo dedicated pass to .bc files. But I think it would not be too hard to change.

Nevertheless, you will have to link the final executable with verificarlo driver script or manually add the required libraries to your linker command.

[pjaaskel]

Yep, this worked. I added .bc support to verificarlo and called it before our final codegen step. At least does something sensible-looking for the basic OpenCL example1:

(venv) pjaaskel@pjaaskel-mobl1:~/src/pocl/build-cpu$ VFC_BACKENDS="libinterflop_vprec.so --precision-binary32=3 --range-binary32=5" examples/example1/example1 
Info [verificarlo]: loaded backend libinterflop_vprec.so
Info [interflop_vprec]: load backend with precision-binary32 = 3, range-binary32 = 5, precision-binary64 = 52, range-binary64 = 11, mode = ob, error-mode = rel, max-abs-error-exponent = -1022, daz = false, ftz = false and instrument = none
(0.000000, 0.000000, 0.000000, 0.000000) . (0.000000, 0.000000, 0.000000, 0.000000) = 0.000000
(1.000000, 1.000000, 1.000000, 1.000000) . (1.000000, 1.000000, 1.000000, 1.000000) = 4.000000
(2.000000, 2.000000, 2.000000, 2.000000) . (2.000000, 2.000000, 2.000000, 2.000000) = 16.000000
(3.000000, 3.000000, 3.000000, 3.000000) . (3.000000, 3.000000, 3.000000, 3.000000) = 36.000000
OK
(venv) pjaaskel@pjaaskel-mobl1:~/src/pocl/build-cpu$ VFC_BACKENDS="libinterflop_vprec.so --precision-binary32=2 --range-binary32=5" examples/example1/example1 
Info [verificarlo]: loaded backend libinterflop_vprec.so
Info [interflop_vprec]: load backend with precision-binary32 = 2, range-binary32 = 5, precision-binary64 = 52, range-binary64 = 11, mode = ob, error-mode = rel, max-abs-error-exponent = -1022, daz = false, ftz = false and instrument = none
(0.000000, 0.000000, 0.000000, 0.000000) . (0.000000, 0.000000, 0.000000, 0.000000) = 0.000000
(1.000000, 1.000000, 1.000000, 1.000000) . (1.000000, 1.000000, 1.000000, 1.000000) = 4.000000
(2.000000, 2.000000, 2.000000, 2.000000) . (2.000000, 2.000000, 2.000000, 2.000000) = 16.000000
(3.000000, 3.000000, 3.000000, 3.000000) . (3.000000, 3.000000, 3.000000, 3.000000) = 40.000000
FAIL

[pjaaskel]

I added a feature to PoCL that can be used to hook the verificarlo script: pocl/pocl@ed7ed16#diff-daa32eea6957d0ffd4917a63c90fd3b99c2e18b2b5b2394c02005cef860e4e69R113 seems to work nicely. When running PoCL 'make check' with verificarlo instrumentation on 79% tests passed, 36 tests failed out of 170. There are double frees and issues with 3-element vectors, quickly spotting, which might be something to check at some point.

[titeup]

Thanks! If the double free probelem is on verificarlo side, we are definitly interested to fix it. You said 36 of your test are not passing, is it related to Verificarlo, I cannot find the corresponding test output on your (fork) pocl git with the branch adding support for verificarlo.

[pjaaskel]

(Sent privately.)