Debug with python interface

[SYkirk]

Hello all!
I really appreaciate it for your great work on FLAME GPU2. I have been learning details on it and transferring other agent-based model into this astonishing framework with python interface. But I am wondering how can I debug or figure out whether my code works as expected since python raw strings are used widely (e.g. define agent function behaviour)? I can not get any error notations or exceptions from pycharm now. Could you show me some approaches?

[mondus]

This is a tricky one. You certainly can't debug from a python debugger as the code is run on the GPU device. The stringified agent functions are compiled (as cuda c++) on the fly and executed so a Python debugger would not have any way to understand this. The best way to debug would probably to be to add print lines within your agent functions (@Robadob will follow up with an example). We discussed this and it is technically possible to use the cuda debugger to attach to your agent functions but I would expect this to be difficult (to the point we probably wouldn't attempt this!). We are looking at improving our Python interfaces and I am currently drafting the option for pure python functions (although this would undertake source to source compilation so wouldn't actually allow you to debug them).

[Robadob]

Firstly you probably want to build a recent copy of pyflamegpu with SEATBELTS enabled. This runtime error-checking, will throw an exception if you are accessing variables that don't exist and similar. I don't believe this is enabled in the official release builds (@ptheywood can confirm when he's back from leave?), similarly those are now 7 months old so alot of improvements will be missing. (We've been holding off a further release until an academic paper is accepted...)

Three options (assuming you're referring to debugging agent behaviour, rather than compilation errors) in order of ease, these all apply to non-python models too.

1. printf()

   You can use the C/C++ printf() method to print data directly from agent functions.
   This works similar to Python formating e.g.

   print("foo: %d, bar: %f"%(my_int, my_float))

   from Python, would look like this in C/C++/CUDA

   printf("foo: %d, bar: %f\n",my_int, my_float);

   A couple of additional notes

   • printf() unlike Python's print() doesn't automatically terminate with a line break, so you probably want to add \n.
   • If you have thousands of agents, having them all print in parallel is a bad idea. You should selectively print from a subset of your agents. You could arbitrarily do this using either the thread index (FLAMEGPU->getThreadIndex()) or the agent's index (FLAMEGPU->getID()). Both return type unsigned int, the first index being 0 and 1 respectively.

2. Agent Variables/Logging
   For higher level modelling investigation you probably want to look at population level dynamics. This could be tracking an existing agent variable over time, or a validation specific variable you may temporarily add to the model.
   The documentation has a thorough page on our logging API for models. We use this extensively when developing, validating and calibrating models internally. It allows environment properties and agent variable reductions (sum/average/standard deviation/min/max) to be logged.
   The logging API however won't return the raw population data (as it's assumed this is impractical for large models). You can manually grab the agent data if you require that for some reason. We mostly make use of this in our test suite.

   An additional note:

   • Environment macro properties are a new feature to FLAMEGPU2. These have been useful for logging, as you can have agents increment values in parallel based on complex internal state information. They're a bit more involved to use as they can't be logged directly, but really useful in some cases.

3. CUDA debugging of Python models should be possible, but CUDA debugging is inherently complex (due to highly parallel and SIMT architecture), runs very slowly and made more challenging to setup by the runtime compilation within the Python API. As such, we generally leave it as a last resort for investigating problems with large models.
   Requirements:

   • cuda-gdb on Linux, Visual Studio + Nsight (the VS CUDA debugging plugin, not the standalone profiling tools) on Windows
   • You may want a debug build of pyflamegpu. This will run much slower, but prevents compiler optimisations from causing the compiled code to not match source.
   • You will need to move your agent functions to standalone source files, and load them using agent.newRTCFunctionFile("agent_fn1", "agent_fn1.cu"); (This appears to be missing from the userguide!)

   With all this setup, it should be possible to run your Python model, and attach the Cuda debugger. Additional configuration will be required to allow the debugger to find the source files, but I haven't tried this recently so don't know off-hand.

Actions For Me

• Add a note about newRTCFunctionFile() to the userguide.
• Create an issue/plan to discuss model debugging in the userguide. I'm on leave from tomorrow till the end of the month, so this may not get addressed until August unless someone else picks it up.

[ptheywood]

Firstly you probably want to build a recent copy of pyflamegpu with SEATBELTS enabled. This runtime error-checking, will throw an exception if you are accessing variables that don't exist and similar. I don't believe this is enabled in the official release builds (@ptheywood can confirm when he's back from leave?), similarly those are now 7 months old so alot of improvements will be missing. (We've been holding off a further release until an academic paper is accepted...)

The released python wheels are built with SEATBELTS=ON (source), with users wanting more performance instructed to build their own wheels (this should be in the release notes).

We could provide both, but this would double the number of wheels produced, pypi/wheels don't have a conventional way of supporting this (conda has mechanisms to support this iirc).