Repository navigation
Kaden McCartney's Notebook
I'm leaving this final entry as a guide for the networking team for the Spring 2026 semester. If you have any questions, feel free to ping me in the Discord server or send me an email at kaden.mccartney@gmail.com.
Currently, we have a dnp3-example repo and the main cps-cosimulation-env repo. The dnp3-example repo should be considered deprecated since all of the work has been migrated to the cps-cosimulation-env repo.
The cps-cosimulation-env repo has way too many branches. The main ones for the networking team's purposes are the Networking_team branch, the piperr branch, and the Device_modeling_subteam01 branch.
These three branches are all strictly ahead of the main branch (i.e., not branched from a previous main commit). The main branch is thus the starting point for the features these branches add.
The Device_modeling_subteam01 branch started as the networking team's branch in the Spring 2025 semester, and turned into the device modeling sub-subteam's branch in the Fall 2025 semester. As such, it contains some networking code for DNP3, but this is very old and outdated. However, it contains the most up-to-date device models, and also has extremely useful documentation for DNP3 integration with device models. For example, this rtu.py file contains a large starting comment explaining the DNP3 points for the RTU device. This information will be used by the networking team when integrating the finished DNP3 Python extension module in device models. Put more simply, this branch has up-to-date device models and the blueprints for DNP3 integration, but very old DNP3 code.
The Networking_team branch is the main branch that the networking team used in the Fall 2025 semester. As such, it contains all of the DNP3 code that was carried over from the dnp3-example repo. In addition to all networking-related changes, it contains significant changes to the project's directory structure, build system, and dependency management. The up-to-date documentation for building/running the branch is contained in the top-level README.md. Note that the branch uses Docker, unlike all other branches. The Docker container is also different from the ROS-HELICS team's container.
The piperr branch was made from the Networking_team branch for a final demo. The main difference is that the piperr branch works off the older Nix build environment instead of the newer Docker environment, meaning it is missing these two commits (fa3c17c and f4c06ec). This was done because Docker has issues on MacOS (due to Apple Silicon), so Piper was still using Nix for the final demo. However, this branch contains three extra commits adding DNP3 outstation database functionality (d425252, 2c97a33 , and f8e3dc4), meaning it contains the most up-to-date DNP3 code.
The first task will be merging the changes from the three branches into main. This will not be easy, and I think the best way would be to forgo using Git to do the merge, and instead have somebody manually compile the changes together.
The first step will be deciding whether to use Docker, Nix, or something else. Something to consider is that for the Spring 2026 semester, the instructors plan on bringing in standardized Linux machines for students to use. Personally, I would use the Docker build environment because it's more ubiquitous. However, if the team feels more comfortable with Nix, that's also fine. I would advise against no dependency management (i.e., just using the OS's packages) because that introduces variability in versions across machines and also makes it harder for non-Linux users to build and run locally.
From there, the changes from the piperr and Networking_team branches can easily be resolved. If going with Nix, just create a pull request for the piperr branch into main and merge it. If going with Docker, take the extra commits from the piperr branch and apply them to the Networking_team branch, then create a pull request and merge into main.
After that, someone will need to do the painful process of integrating the Device_modeling_subteam01 branch. There's no easy way to get out of doing this manually. The approach I would take is creating a new branch off the now updated main, and manually adding in all of the changes from Device_modeling_subteam01. Then create a pull request for the new branch and merge. After that, the DNP3 library can be finished and device model integration can be started.
The main work left here are Alex's and Bermet's parts. They should know what needs to be done, and I've mentioned this work some in my notebook, and also in the Discord server here and here. Additionally, we need to grab HELIC's time in the DNP3 C code (see here).
This can happen while the DNP3 library is being finished since the database update methods are already done. The work here is to translate the comments from the Device_modeling_subteam01 branch into actual calls to the Python DNP3 library. All classes that don't already inherit from the Outstation class should be made to do so. This gives access to methods like def analog_output_status_transaction(self, value: float) to update the outstation's database. It will also give a way to set callback handlers for events from the Master once Alex finishes his work.
Lastly, the RTU and Master will need to be finished. These will be significantly more challenging than regular Outstations since they are responsible for controlling all the substation devices. The RTUs are in charge of more granular controls within a substation, and are deployed at the substation. They serve as intermediaries between all the substation devices and a higher level Master. The Master will be located at a centralized SCADA control facility and may be in charge of multiple substation. See the diagram below.
There's much more nuance than this simple explanation though, since there are different ways this can be done. See the Wikipedia entry for example. Since I don't fully grasp how this looks in practice, it would be best to ask the instructors for guidance before seriously starting on this endeavor. Finishing the RTU and Master will likely take the whole semester, and will constantly be updated as the project continues.
If I were to start working on the RTU/Master blind, right now, without giving any more thought, the approach I would take is having the RTU also be a DNP3 master (by inheriting from this class) so it can issue commands to all the outstation devices. Additionally, I would change that Master class to inherit from the Outstation class and make any modifications necessary to make that work, so that a Master can serve as an outstation for another Master. With this setup, the RTU will be both a Master for the substation devices, as well as an Outstation to the main SCADA Master.
- Finish final paper
As the semester wraps up, the only thing left is the final paper. I updated my part to include details on the build system and environment. Since I graduate in a few weeks, I will try to leave a large entry detailing where progress left off for the project for future students to reference.
Thanksgiving break
- Finish the final presentation
The final presentation was on Thursday. I didn't really make any changes from the last week since I already had plenty to talk about (the presentation ended up going a bit over time). I tried to coordinate with the team members to get a demo of master-outstation two way communication, but only Piper was able to finish her part in time (updating the outstation database). Bermet's part (master read handler callbacks) and Alex's part (select-operate callbacks) will hopefully be finished in time for the final paper, but if not, they will have to go to next semester.
The demo can be found on Piper's branch. It allows a user to input new database values from the command line and uses the Python dnp3 library to update the database values. We also used Wireshark to show that an actual DNP3 packet is formed - when you run a capture while running the program, you will see a packet with the description "Unsolicited Response" containing the actual data that you update.
We also assigned parts for the final paper at the Friday meeting. I will just update my parts to say that we fixed the segfault issues, and that we are moving to sharing physical machines next semester since Docker/Nix/VMs all have issues. Piper is updating the summary, Bermet will hopefully be able to write about the read handler callbacks, and Alex will hopefully be able to write about the select-operate callbacks. He is also adding some information on HELICS since he has also started trying to get timing consistent between HELICS and the dnp3 library.
Since this is my last semester before I graduate, I will try to leave a big final notebook entry after I graduate for people to reference next semester and beyond. I will also remain in the Discord server.
- Get everyone on Docker
- Start final presentation
At our Thursday meeting, everyone was struggling to get the Docker solution working. Almost every person had a different issue - Piper was hanging on the CMake step "Detecting C compiler ABI" while building the dnp3 extension module, but also hung when trying to run cmake for HELICS. Bermet and Chungman were able to get past this, but would eventually fail while building HELICS because all system memory was being used. Increasing the amount of memory helped the problem, but not enough - even with 16 GB dedicated to Docker, the OOM reaper would still kill the build. Maria and Alex also ran into some issues on Windows.
My conclusion from this is that cross-platform development is a nightmare, and I wasn't totally wrong to go with a QEMU VM at first. However, if I remember correctly, even the VM couldn't build everything without running into OOM issues. The instructors said they will just get some machines for students to use in future semesters.
We also started working on the final presentation. Our goal is to have a demo showcasing everyone's work. I also added a slide showing my efforts at making a good development environment + build system.
- Migrate from Nix to Docker
It seems like everybody has unique issues when trying to work with the new Nix setup. So, I decided at our meeting on Friday to ditch Nix and move to Docker. The commit is here. It was fairly simple to get working since I have experience setting up Docker for development environments. All the documentation is in the README, so I won't repeat it here in my notebook. Theoretically, this fixes the Gridlab-D issue since Apple Silicon users can enable Rosetta 2 to emulate x86_64 instructions. However, Rosetta will apparently be phased away starting in 2027. Hopefully Docker figures out an alternative by then.
The lift wasn't too bad. I have the Docker container function very similarly to the Nix shell by using the cps-cosimulation-env directory as a shared volume (specified in the docker run command). I just translated the flakes that Nix included into Debian packages in the apt-get step of the Dockerfile. I also had to use the uv install script since a package doesn't exist for uv in Debian.
The only tricky parts were related to the home directory. When building the dnp3 extension module, there were some warnings about uv not being able to copy files from its cache. This was because uv created its .cache directory in the home directory, which is only in the container, not in the shared volume. This caused permission conflicts since the Docker container runs as a non-root user. I fixed this by hardcoding an environment variable to tell uv to create its cache in the cps-cosimulation-env directory: ENV UV_CACHE_DIR="$REPO_ROOT/.cache".
Additionally, I created a setup script (setup.sh) to take the place of the nix-shell-setup.bash. I split the functionality in two parts: I added a line to the container user's .bashrc to source the virtual environment (in the Dockerfile), while setup.sh installs the correct version of Python, creates a virtual environment, and runs a uv sync to download/build any dependencies. I also added the make all-install step to setup.sh.
- Get teammates migrated to the Nix environment
- Figure out solution to gridlab-d issues
I didn't have much time this week due to two exams. I mostly helped the teammates with getting the new build environment working. With some slight tweaking, we were able to get the device modeling and dnp3 components completely working on Windows and Macs. The main troubles were installing Nix on Macs and figuring out the Git submodules, but this was solvable with some trial and error. These are also only issues if trying to update the existing repo (in the case of the submodule issues) and if Nix is partially installed (i.e., config exists, but not the Nix executable). After getting these issues fixed, things mostly went smoothly (after pushing a few quick fixes), except for Maria. On her Windows machine, she was getting a weird error about not being able to find the nix-shell-setup.bash, which I wasn't able to figure out in class.
However, there is another bigger issue that I didn't realize existed with the main repository: gridlab-d doesn't support Apple Silicon. They have a few branches which make some fixes, but there are still issues. The solution posted on that GitHub issue is to use Rosetta 2 to allow Apple Silicon to emulate the x86-64 architecture. As such, I told one of the people with a Mac (Arya) to try to get Rosetta 2 working. I gave this article as a starting point, which shows there is integration possible with Nix.
- Get to a better solution for building
I made a crazy amount of progress on Monday since Canvas was down, so I had nothing better to work on. The TLDR is that I was able to migrate all of our work the past two semesters from the dnp3-example repo to the cps-cosimulation-env repo. To do this, I moved from using Poetry to uv, updated all the project's dependencies, and added the code for the Python extension module from dnp3-example. All this work was done as part of this rather large commit.
The code for the dnp3-example extension module was copied from this commit, with one minor modification: I learned that the free-threaded build means I can get rid of the awkward is_shutting_down check in the C logging function, explained in this entry. The new code lives here, and our work should probably continue here instead of the dnp3-example repo.
Now, why did I make such a major change? Well, I was considering integrating the ROS-HELICS Docker container with both the cps-cosimulation-env repo and dnp3-example repo. This may seem elegant at first, but the more I thought about it, the more I realized what a pain it would be, and that it would be completely unnecessary. The cps-cosimulation-env repo already has a development environment, and one arguably even better than a Docker container - Nix! For a list of pros and cons, see this discussion.
The only fundamental point of conflict between the dnp3-example repo and (old) cps-cosimulation-env repo was everything involving Python. I recently moved the dnp3-example repo to use uv and scikit-build-core, while the cps-cosimulation-env uses Poetry. Since uv is the modern solution, I decided it wouldn't be a bad idea to migrate the cps-cosimulation-env repo to uv. This part was surprisingly easy thanks to the standardization of the pyproject.toml format. The main magic is changing the top-level pyproject.toml to list dnp3 as a dependency and add these lines:
[tool.uv.sources]
dnp3 = { path = "./dnp3", editable = false }This tells uv that the dnp3 package that the main cps-cosimulation-env project depends on can be found locally at the path ./dnp3. In the ./dnp3 directory is the pyproject.toml from the dnp3-example repo used to build the dnp3 extension module. With this, uv is smart enough to automatically build and install the dnp3 extension module for us! One note - I set editable to false. This is usually desired when developing a Python package, but since we also have a compiled part, I figured it would cause more confusion to have some parts of the package need rebuilding on change (the C parts) and some parts not need rebuilding (the Python parts). Usually, this would be a pain, but uv is blazing fast, so the effect will barely be noticeable.
I also moved from using our own forks of HELICS and GridLAB-D to using the main repos. I was running into some major versioning issues trying to get Nix to cooperate, so updating everything turned out to be easiest.
To try out the new code:
- Update to the latest version of Nix. Most likely, you installed it with the install script, so run the command from here.
- Nuke your existing cps-cosimulation-env repo. Trying to update everything will just be more of a pain, trust me.
- Clone the repo again:
git clone https://github.com/cps-vip/cps-cosimulation-env.git - Switch to the branch with the changes:
git switch Networking_team - Run
nix flake update- this will pull in the latest the Nix packages - Update the Git submodules with
git submodule update --init --recursive - Run
nix developto get into the Nix shell - Now build HELICS and GridLAB-D with
make all-install. This step will take a long time. - Test out simple_gridlabd_example:
cd simple_gridlabd_example
# If run_example hangs, press Ctrl+Z, then run `kill -9 %1`, and try the command again
./run_example.sh
./kill_all.sh
# Ideally, you should see a graph updating in real time while you run the example
# If you don't, that's still fine as long as the there are logs in the `results` directory
ls results
cat {log file names here}- Test out the DNP3 functionality
# cd to the root directory of the project
cd device_modeling/src
# This is the command to use while actively working on the dnp3 extension module. Set PYTHON_GIL=1 to disabled free-threading.
PYTHON_GIL=0 uv run --python 3.14t --reinstall-package dnp3 run_simulation.py
# This is the command if you aren't touching the extension module code (aka you aren't on the networking team)
python -Xgil=0 run_simulation.py
# To check if it worked, look at simulation.log
cat simulation.log
# The run_dnp3_demo.py is run.py from here: https://github.com/cps-vip/dnp3-example/blob/main/c_api_example/run.py
PYTHON_GIL=0 uv run --python 3.14t --reinstall-package dnp3 run_dnp3_demo.py
# Or
python -Xgil=0 run_dnp3_demo.py
# Check if it worked
cat simulation.logIn the future, I will make the extension module say that it supports free-threading. For now, I find it useful to have the dialog popup telling me when I'm not using free-threading since uv sometimes likes to do weird stuff with free-threading. For example, running PYTHON_GIL=0 uv run --reinstall-package dnp3 run_simulation.py will give an error since as of the time of my writing, uv doesn't automatically detect that the existing virtual environment is free-threaded, and will instead install a new regular Python 3.14 virtual environment.
One more technical note on the change I made to Makefile. When I tried to run make all-install with the new versions of everything, I was getting an error because I was now using CMake version 4.1.1.
[steel@cerberus cps-cosimulation-env]$ make all-install
mkdir -p HELICS/build && cmake -DCMAKE_INSTALL_PREFIX=/home/steel/git/cps-cosimulation-env/install/HELICS -DHELICS_BUILD_CXX_SHARED_LIB=ON -DHELICS_BUILD_EXAMPLES=ON -DHELICS_BUILD_TESTS=ON -S HELICS -B HELICS/build
-- SYSTEM INFO -> Unix Makefiles Linux-6.17.3-3-cachyos:GNU-12.3.0
-- got heads/main-0-g71f375635 from third try
-- helics version tag description is main-g71f375635
CMake Error at ThirdParty/cmake/ucm.cmake:13 (cmake_minimum_required):
Compatibility with CMake < 3.5 has been removed from CMake.
Update the VERSION argument <min> value. Or, use the <min>...<max> syntax
to tell CMake that the project requires at least <min> but has been updated
to work with policies introduced by <max> or earlier.
Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.
Call Stack (most recent call first):
config/cmake/commonBuildFlags.cmake:12 (include)
CMakeLists.txt:183 (include)I tried installing an older version of CMake to use for compatibility, but that didn't end up working because the version of CMake was so old that I would need to bring in other old dependencies (I was getting some glibc version errors just running cmake --version). Instead, I just added -DCMAKE_POLICY_VERSION_MINIMUM=3.5 like the error message suggested.
At our Thursday meeting, I realized the issue with the Rust dnp3 library not supporting MacOS wasn't fixed by Nix - a major oversight by me. However, while the official releases for the library don't include Mac support, it does provide unofficial support by building from source. By doing this, Piper and I were able to run the C example with modifications to the CMakeLists.txt. Specifically, this involved removing the code that checked for platform support and the code that pulls the release from GitHub. Instead, when building the library manually, libdnp3_ffi.dylib is available in the "targets" directory, and this can be linked against. After the meeting, I was able to integrate these changes in this commit.
- Try to get the ROS-HELICS team Docker container working
I didn't do much this week because I was busy with other classes. I was able to build the ROS-HELICS Dockerfile and get it running. Turns out that the dnp3-example repo works perfectly fine; the only addition needed is installing uv. Meetings this week didn't involve much.
- Work on midterm presentation
Fall break was this week, so all I did was add a slide to the midterm presentation updating our current progress.
- Work on midterm paper
I added a brief update on how Python 3.14t has fixed our issues from last semester. I also moved the two large diagrams (DNP3 flow diagram and class diagram) to the appendix. I also had Alex create a new flow diagram for the movement of data (the current one is showing the separation of code). I didn't make any more technical work due to exams before Fall Break.
- Work on improving the build system
For an assignment in another class (any project worked, but I conveniently already had the Debian image for this VIP), I measured the time to build dnp3-example (specifically this commit) in different environments.
Once Jude finishes the Docker image, the rest of the team should be able to move a lot faster. To further support this, I have ported from setuptools to uv and scikit-build-core (commit). uv is the newest Python virtual environment solution on the block and runs extremely fast compared to the default Python tooling. uv is a replacement for pip, so any pip commands become uv pip. scikit-build-core is the build backend - that is the system building the extension module - and will replace the old backend, setuptools. Currently, sci-kit-learn is the only available option for use with uv, as the uv documentation states the following:
Creating a project with an extension module requires choosing an alternative build system. uv supports creating projects with the following build systems that support building extension modules:
- maturin for projects with Rust
- scikit-build-core for projects with C, C++, FORTRAN, Cython
Conveniently, scikit-build-core integrates with CMake, which is already being used to bring in the dnp3 Rust library. To use this new build system, the interface remains the exact same:
./install_module.sh
source venv/bin/activate
python run.py
# To test everything worked, look at simulation.log
cat simulation.logIf you want to see the final build outputs (as opposed to only seeing that the module installed), run uv build in the dnp3-module directory. The sdist and wheel files will be in the dist directory. For those unfamiliar with Python wheels, they are just special zip files that Python recognizes as an importable module. The sdist is a tar of all the source files used to make the module, and doesn't have any use for our project. We don't easily see these file with ./install_module.sh, but if you look carefully at the end of the output of the command, you'll notice "Created dnp3-1.0.0-cp314-cp314-linux_x86_64.whl." Since uv build doesn't respect the build-dir specified in pyproject.toml (which is used to cache the dnp3 library), we use uv pip install instead, which creates the wheel somewhere in uv's cache instead of in a nice dist directory like uv build.
Some technical notes:
- libdnp3_ffi.so is copied into the wheel by a CMake install command at the bottom of the CMakeLists.txt. This manual copy was the best way I could find of including the .so in the wheel. To facilitate proper linking with this, I set the runtime path for the master/outstation/tcpserver binaries to $ORIGIN, which is the current directory.
- The CMAKE_BUILD_TYPE can be changed to Release when we integrate DNP3 with the cosimulation repo.
- CMAKE_EXPORT_COMPILE_COMMANDS is set to generate compile_commands.json in the build directory, which gives LSPs the info they need.
- As mentioned above, the build directory is necessary to avoid reinstalling the dnp3 library every time. To see what I mean, just run
uv buildmultiple times, and you will notice a lot of downloading. If you runuv pip install --verbose .multiple times, you will only see the downloading on the first run. - scikit-build-core uses ninja instead of make by default.
Our Thursday meeting was taken up by a Wireshark walkthrough. At the Friday meeting, we discussed the midterm paper. The roles are the following:
- I'm updating our technical progress in the "Integrating DNP3" section
- Alex is creating a diagram to add to that section
- Piper will write the text to explain that diagram
- Bermet will rewrite parts of the "Substation Network Architecture" section to talk about future TLS integration
- Jude will update the executive summary
Also, Alex mentioned he wasn't having the same logging issues I was having (mentioned in the previous entry) with client_state_on_change. He was totally right - somehow, I forgot to test the code with free-threading enabled. Now there are no known issues with integration!!! We have a working example of calling outstation functionality from Python in Piper's branch (showing a database transaction), a working example of a callback written in Python (in the now outdated name "callback-not-working" branch), and a much faster build system in main. It's all coming together finally!
- Work on free-threading
Warning - This notebook entry is a bit all over the place, partly because I was switching between different tests very rapidly and updating the entry while doing this. I'll have a TLDR at the end.
Since I was segfaulting on Py_BEGIN_CRITICAL_SECTION, I figured there was something fundamentally wrong with what I was doing. As such, I created a new project without any of the dnp3 baggage to try to get free-threading working. As expected, even a barebones version of the project was segfaulting with Py_BEGIN_CRITICAL_SECTION. After that, I tried going back to a regular Python threading build, and here I realized something deeper was wrong - I couldn't even get things working using the GIL. Here's what the simplest test code looks like:
PyObject *global_binary_input_callback = NULL;
void internal_binary_input_callback() {
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
printf("Acquired GIL!\n");
PyObject_CallNoArgs(global_binary_input_callback);
PyGILState_Release(gstate);
}
static PyObject* add_binary_input_callback(PyObject *self, PyObject *args) {
if (!PyArg_ParseTuple(args, "O", &global_binary_input_callback)) {
return 0;
}
global_self = self;
if (!PyCallable_Check(global_binary_input_callback)) {
return 0;
}
Py_RETURN_NONE;
}
void *thread_func(void *arg) {
internal_binary_input_callback();
pthread_exit(NULL);
}
pthread_t tids[100];
static PyObject* start_master(PyObject *self, PyObject *args) {
pthread_mutex_init(&lock, NULL);
for (int i = 0; i < 100; i++) {
pthread_create(&tids[i], NULL, thread_func, NULL);
}
Py_RETURN_NONE;
}
static PyObject* stop_master(PyObject *self, PyObject *args) {
for (int i = 0; i < 100; i++) {
pthread_join(tids[i], NULL);
}
Py_RETURN_NONE;
}Yet for some reason, the call to PyGILState_Ensure() never finished on any of the threads! So, I went searching online for deadlocks related to PyGILState_Ensure, and I found this comment. The difference from that example and my code is Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS. And somehow, adding these in start_master and end_master fixed the problem. According to the documentation, Py_BEGIN_ALLOW_THREADS is used before blocking I/O that doesn't access Python objects, which I guess pthread_create and pthread_join fall under.
But... I don't know if this is particularly helpful for the real project since the threads are created by the Tokio runtime.
After that, I went back to experimenting with free-threading. In a rather baffling case of bad naming, you still need to call PyGILState_Ensure when the GIL is disabled (link). That fixed the Py_BEGIN_CRITICAL_SECTION segfaulting, and the test was working properly.
Naturally, I tried applying this to the (callback_not_working branch)[https://github.com/cps-vip/dnp3-example/tree/callback_not_working], and to my surprise, it worked! However, this confused me a lot... now that I reverted to using PyGILState_Ensure, this is effectively the same code I used for this video. So why work now?
Well, in a very unlucky coincidence, it turns out that the code from that video would've worked perfectly fine if I had written it a couple months later... Python 3.14 fixed whatever issue was causing the segfaulting and deadlocks. The exact same code segfaults on Python 3.13. I could've saved the last few weeks of work if I had just tried the same code with a newer Python version.
Some of the issues still remain though with logging. As mentioned in this notebook entry, trying to use the logger in client_state_on_change still doesn't work (deadlocks). This may be the actual situation where free-threading is useful, since I believe this one may be a logic-based deadlock (rather than a Python version issue like with the binary input callback).
Our Thursday meeting was uneventful. At our Friday meeting, I let the team know of my progress. There wasn't any other news, and Jude wasn't there to give an update on the Docker work. After the meeting, I fixed the problem in Piper's branch (commit) related to a null pointer dereference. Now, we have an example of making an update to the outstation's database, and the master station seeing the update.
TLDR; Python 3.14 fixed the biggest issue, no free-threading necessary. Some logging issues still remain though.
- Work on free-threading
I had an exam and projects this week, so I wasn't able to do any technical work. At our Thursday meeting, I helped the team members with pulling the new changes to the dnp3-example repo. We also decided to have the networking team meeting at 2:00 PM on Friday, and I will go to the 1:30 meeting as a representative for the networking team.
At the Friday meeting, Maria mentioned openPDC, a featureful data concentrator. I found that openPDC does support dnp3 (see Dnp3Adapters.XML). The repo seems to includes pre-built dnp3 executables coming from OpenDNP3 (the deprecated library discussed in many of my previous entries). The releases page is where I found that they were using OpenDNP3. I'm not sure how useful this is at the moment, but it could be useful in the future if we want to integrate other protocols (e.g., openPDC also supports Modbus and IEC-61850, which are other protocols we were initially looking at).
- Finish kickoff presentation
- Start researching Python free-threading feature
I mentioned two notebook entries ago that new versions of something (Python or setuptools) had broken the c-api-example build system. I managed to get it working again (commit), but that's only a temporary fix. If I can get the free-threading to work, I'll transition to using scikit-build-core, which is meant for the type of building our project does (with CMake).
I also plan on using uv as much as possible. One benefit is it makes creating a virtual environment with a free-threaded Python version very simple: uv venv --python 3.14t.
My next step is to try getting the bug fixed with Python free-threading. I'm no longer able to replicate the deadlocks I was getting, but that isn't free-threading specific (seems to be a newer version of Python in general). Now, I just get segfaults on the PyObject_CallNoArgs call. I tried adding a Py_BEGIN_CRITICAL_SECTION around the call, but for some reason it segfaults on that also, specifically in _PyCriticalSection_BeginMutex. See the stack trace below.
$ gdb -args python -Xgil=0 run.py
(gdb) r
Starting program: /home/steel/git/dnp3-example/c_api_example/venv/bin/python -Xgil=0 run.py
This GDB supports auto-downloading debuginfo from the following URLs:
<https://debuginfod.archlinux.org>
<https://debuginfod.cachyos.org>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[Detaching after fork from child process 545610]
[New Thread 0x7ffff6fff6c0 (LWP 545611)]
[New Thread 0x7ffff6dfe6c0 (LWP 545612)]
[New Thread 0x7ffff6bfd6c0 (LWP 545614)]
[New Thread 0x7ffff69fc6c0 (LWP 545615)]
[New Thread 0x7ffff67fb6c0 (LWP 545616)]
[New Thread 0x7ffff65fa6c0 (LWP 545617)]
[New Thread 0x7ffff63f96c0 (LWP 545618)]
[New Thread 0x7ffff61f86c0 (LWP 545621)]
Ptr is: 0x22520b6bc00
Calling callback
Thread 2 "tokio-runtime-w" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff6fff6c0 (LWP 545611)]
_PyCriticalSection_BeginMutex (m=<optimized out>, c=<optimized out>)
at ./Include/internal/pycore_critical_section.h:103
warning: 103 ./Include/internal/pycore_critical_section.h: No such file or directory
(gdb) bt
#0 _PyCriticalSection_BeginMutex (m=<optimized out>, c=<optimized out>)
at ./Include/internal/pycore_critical_section.h:103
#1 _PyCriticalSection_Begin (c=<optimized out>, op=<optimized out>)
at ./Include/internal/pycore_critical_section.h:115
#2 PyCriticalSection_Begin (c=<optimized out>, op=<optimized out>) at Python/critical_section.c:129
#3 0x00007ffff7e798b0 in internal_binary_input_callback (info=..., it=0x7ffff6ff3bd0, arg=0x0)
at src/dnp3/master.c:25
#4 0x00007ffff710e78c in dnp3_ffi::handler::<impl dnp3::master::read_handler::ReadHandler for dnp3_ffi::ffi::ReadHandler>::handle_binary_input ()
...Revisiting the code after the summer, I'm hopeful I can get something working. I think Py_BEGIN_CRITICAL_SECTION should be very effective at stopping deadlocks, so if I figure out this segfault, there might be a chance it works. I also figured out how to turn off optimizations when building the extension module, which makes debugging easier: CFLAGS="-O0 -g3" python -m build. My current work is on this branch.
At our subteam meeting, we worked out tasking:
- Piper - working on outstation database methods
- Alex - continuing HELICS work, looking at the free-threading issue
- Jude - getting a Docker container with x86-64 emulation instead of qemu
- Bermet - looking at the free-threading issue
- Start on kickoff presentation
- First subteam meeting
In class, we broke into subteams. There are only two new people, and most people from previous semesters stayed. We also set up a subteam meeting time, 1:30 PM on Fridays. I talked to Sam after class about direction for the bug described in my previous notebook entry. He agreed with my assessment: first, try getting free-threading to work, then fall back to writing the callbacks in C while working on a more permanent solution.
For the Discord meeting, my mic was broken, so I didn't go over anything technical yet. We just broke up the kickoff presentation slides between team members. I assigned myself to make a slide explaining the problem described in my previous notebook entry.
After that, a few of the device modeling members (including me) had a meeting with Sam about doing cybersecurity in the project. He mentioned two avenues: 1. network protocol security, 2. device firmware security. Since I'm on the networking team, I'll probably be doing the network protocol security at some point (since our DNP3 library has TLS support). I'm also interested in the firmware security though, and Sam mentioned a common device that we might want to look at: an SEL-751 Feeder Protection Relay.
I tried finding some firmware that we could reverse engineer, but I didn't find any. I tried expanding my search to any SCADA device firmware, but I wasn't able to find any. I did find this useful resource though: awesome-industrial-control-system-security.
- Set up dev environment (again)
- Brainstorm solutions to the Python-related problems
Since I changed my OS (Arch -> CachyOS), I set up my dev environment again. This should've been as simple as pulling down the two repos (this one, cps-cosimulation-env, and the dpn3-example repo) and running the steps in the README. However, I found out that a newer version of Python messed up the c_api_example due to updates of how the Python egg is built.
I could fix this example, and I likely will next week, but there is a deeper problem with the project that I have described in previous notebook entries - calling Python from the dnp3 C API in a thread-safe way. As such, I think it may be worth re-thinking our approach to DNP3 integration.
The three obvious ideas are:
- Rewrite everything in Rust (abandon C and Python)
- Rewrite device models in C (abandon Python integration)
- Try a more creative method of calling Python from C (async? Cython?) and hope it works
Idea 1 is the obvious solution. I don't know Rust, so I would be spending the rest of my time in the VIP learning Rust, but this is probably the correct long-term approach for the project.
Idea 2 would be my personally preferred approach, but that comes out of a selfish reason - I like C. As far as the bigger picture, C is an unsafe language, and thus inappropriate if our goal is to create a simulation of a "secure" substation. However, a counterargument is that real-world substations probably use C despite its many pitfalls, so it wouldn't be that inaccurate of a model.
Idea 3 is what I'm leaning towards for now, but I also have significant sunk-cost bias there. We wouldn't have to rewrite anything, but we still have the very sketchy base of calling Python from C in a Rust runtime. I think the only situation in which this approach is justifiable is if using the experimental no-gil version of Python fixes the issue. This is called free threading: link 1, link 2. Cython has had the ability to run code outside of the GIL, but only if not interacting with any Python objects.
I will consult the instructors and do some more research on how actual substations implement DNP3 before making a decision.
- Finish final paper
- Make a videos for next semester
Final paper: https://www.overleaf.com/project/67a67658b1b60e082fb6efc2
I pushed my existing work attempting to get the Python callbacks working to the repo at this branch: https://github.com/cps-vip/dnp3-example/tree/callback_not_working
Sorry for the poor audio quality, but here are some videos to help explain the current progress.
- QEMU VM + dnp3-example guide: https://youtu.be/SfDIqouPzRE
- Explanation of c_api_example: https://youtu.be/PnenFxppkgs
- Current c_api_example issue: https://youtu.be/egK3IAOdUaE
- Finish final presentation
- Start work on final paper
- Peer evals
Final presentation slides: https://docs.google.com/presentation/d/1inUh7Owz4n2c59d4-hSA0mpdCEPt4-Qzhgzp9YmLKJw/edit
Not much to say other than finished the final presentation. I also added a bit to the DNP3 documentation.
- Get working example of callbacks
- Start on final presentation
At our Thursday meeting, I helped some of the team members with their tasks again. I tried getting a simple example of the Master callback functionality working, but I wasn't able to. I strongly suspect a similar issue to the logging bug, which means it may not be fixable. When I do a simple example of setting a callback function in the C code from Python and calling it from a startup method, it works fine. But when I try to call the Python function from a handler method (e.g,. handle binary input), I get a segfault. I tried acquiring the GIL in the callback, but this led to race conditions (some runs were segfaulting, and others were deadlocking).
Sadly, this might mean what I've been trying to achieve isn't possible. We may just have to write handler functions in C, instead of being able to write them in Python. This makes the networking team's job easier, but the device modeling team's job harder.
Additionally, Piper is having issues with a crash when making a database transaction from Python. This would be even worse, since there is no aspect of calling Python from C, indicating the issue might be worse than anticipated.
- Provide support to teammates
Besides documentation, I haven't given myself a task since I'm focused on other classes. However, I have been spending some time writing up more details about the tasking for my teammates, and learning more myself in the process. I'll paste one of the messages at the end of the entry.
On Thursday and Friday, I mainly helped out Piper and Nicholas with their tasks, explaining the callback mechanism to Nicholas and helping Piper with testing out her code. Also, Maria and Chungman made good progress on the networking document. I gave them some additional tasking (documenting IPC and HELICs in addition to DNP3).
This was the Discord message for Nicholas and Bermet's tasks of outstation callbacks (select/operator) and master callbacks (read handler) respectively:
What you and Bermet are working on is something more complicated than what Piper is. Piper's code just directly calls C methods that update database values in the outstation. You on the other hand want to write functions that add Python callback functions to the outstation (master in Bermet's case). What I mean by this:
When a device receives values, it needs to know how to act. Currently, the code just logs what value it got, but in a real-world substation, the device will need to appropriately act upon that info. The way this is done in the dnp3 library is that you pass function pointers to the functions that will get called when a new value is received. These are called callback functions. Here's what this looks like in the code (outstation.c:596 in c_api_example):
dnp3_outstation_t* add_outstation(dnp3_outstation_server_t *server, dnp3_address_filter_t *filter, dnp3_outstation_config_t *config) {
dnp3_outstation_t *outstation = NULL;
dnp3_param_error_t err = dnp3_outstation_server_add_outstation(
server,
*config,
get_outstation_application(),
get_outstation_information(),
get_control_handler(),
get_connection_state_listener(),
filter,
&outstation
);
...This code instantiates the main dnp3_outstation_t struct with a dnp3_control_handler_t, given by the get_control_hander method. This looks like:
dnp3_control_handler_t get_control_handler() {
return (dnp3_control_handler_t){
.begin_fragment = &begin_fragment,
.end_fragment = &end_fragment,
.select_g12v1 = &select_g12v1,
.operate_g12v1 = &operate_g12v1,
...
.select_g41v4 = &select_g41v4,
.operate_g41v4 = &operate_g41v4,
.on_destroy = NULL,
.ctx = NULL,
};
}This is making a struct of function pointers (e.g., select_g12v1 is a function name, which we get a pointer to via &). Then whenever the outstation receives a new SELECT command for a group 12 variation 1 (Binary Command) from the master, it will call the select_g12v1 method, passing in values like the actual command.
Your job is then to allow the device modeling team to write these callback functions in Python so they can handle new data/commands in a higher-level language. The way to do this will be by creating methods in the C extension module that will set these callbacks functions. The Python code should be able to call these methods to set the callback to a Python function. You will need to require this to happen before the outstation object is created (since the dnp3_control_handler_t is passed to the dnp3_outstation_server_add_outstation method). You can do this by adding a similar Python class to the dnp3_control_handler_t as a parameter of the Outstation class constructor.
Bermet, a very similar thing happens in the Master. Instead of a control_handler, it is called a read_handler. In master.c, you can find very similar code such as this:
dnp3_read_handler_t get_read_handler()
{
return (dnp3_read_handler_t){
.begin_fragment = &begin_fragment,
.end_fragment = &end_fragment,
.handle_binary_input = &handle_binary_input,
...
.handle_octet_string = &handle_octet_strings,
.handle_string_attr = &handle_string_attr,
.on_destroy = NULL,
.ctx = NULL,
};
}
int add_association(dnp3_master_channel_t *channel, int outstation_addr, dnp3_association_id_t *association_id) {
dnp3_param_error_t err =
dnp3_master_channel_add_association(
channel,
outstation_addr,
get_association_config(),
get_read_handler(),
get_association_handler(),
get_association_information(),
association_id
);
...- Figure out issue with on_log_message in threads
- Document DNP3Outstation API changes for device modeling team
Thursday we had a guest lecturer, so no progress there. At our Friday meeting, only Nicholas and Piper from the networking team showed up. Maria and Chungman also showed up, and I was able to walk them through next steps for network integration (mainly, diagramming and documenting what data should be sent to/from each device). They created a Google Doc for it. I also walked Nicholas and Piper through their tasks in more detail.
The on_log_message issue with the outstation and master running in the same threads actually turned out to be very simple... it's not possible.
From dnp3.h (available at dnp3-example/c_api_example/dnp3-module/src/dnp3/include once built):
/// @brief Set the callback that will receive all the log messages
///
/// There is only a single globally allocated logger. Calling this method a second time will return an error.
///
/// If this method is never called, no logging will be performed.
/// @param config Configuration options for logging
/// @param logger Logger that will receive each logged message
/// @return Error code
dnp3_param_error_t dnp3_configure_logging(dnp3_logging_config_t config, dnp3_logger_t logger);I confirmed that the call to this function in master.c fails, giving the error message "logging_already_configured". So the outstations and master will just have to run in two separate processes, which is already implemented in run_fork.py.
I also took a look at implementing the handler function callbacks to make sure I wasn't assigning an impossible task to my classmates. I found these resources, which show how it's possible:
- https://stackoverflow.com/questions/53215786/when-writing-a-python-extension-in-c-how-does-one-pass-a-python-function-in-to
- https://stackoverflow.com/questions/74899431/python-c-api-send-a-python-function-pointer-to-c-and-execute-it
- https://docs.python.org/3/c-api/call.html#object-calling-api
Spring break
- Provide support for teammates
Still busy with exams and classes. In class on Thursday, I helped Piper with getting her work running inside the VM, but somehow her progress got deleted in the process. No meeting on Friday due to the upcoming spring break.
- Provide support for teammates
Still busy with a few large projects in other classes. During our Thursday meeting, Nicholas Reed joined the networking team from the device modeling team, so I got him up to speed with the VM and dnp3-example repo. On Friday, I discussed with Christina and Chungman about about starting the DNP3 integration in the cosimulation repo. Working through that also helped me gain a better understanding of how the outstation portion works, which I'll document here.
DNP3 outstations have two key purposes:
- Exposing internal data (measurements and values) to the master station
- Handling commands given by the master station
I'll focus on the first since I'm still not fully clear on the exact mechanism for the second.
Outstation data is handled by the database abstraction. Each outstation has a "database" (in reality, it's just a data structure protected by a mutex). The master station usually sends periodic requests called polls to the outstations asking for data from this database, to which the outstation responds with the appropriate data.
The database holds device data (measurements/values such as voltage and temperature) in what are called "database points." Points consist of five components:
- Index
- Value
- Measurement Type/Group
- Variation
- Class
The index and value are fairly self-explanatory. You use different indices to represent different values (for example, you might assign voltage to index 0, and temperature to index 1). The value would then be the actual value of the voltage or temperature - this is the main thing a device will be frequently updating.
The measurement type, aka group, is the type of the value being stored - for example, a binary value, an analog (integer or floating point) value, a counter, etc. There is further breakdown between read-only/input types and output types, such as a binary input (a single bit value that the master station can only read) vs a binary output (a single bit value that the master can read and write). These types are assigned group numbers - for example, a binary input is group 1, and an analog output is group 40.
A variation is a slightly different variation of the group type. For example, group 30 (analog input) has seven variations:
- Group30Var0: Analog Input - Any Variation
- Group30Var1: Analog Input - 32-bit With Flag
- Group30Var2: Analog Input - 16-bit With Flag
- Group30Var3: Analog Input - 32-bit Without Flag
- Group30Var4: Analog Input - 16-bit Without Flag
- Group30Var5: Analog Input - Single-precision With Flag
- Group30Var6: Analog Input - Double-precision With Flag
This documentation gives a full list of groups and variations.
Each point is also created in one of four class. Class 0 is for static data that doesn't change (for example, one time config). Classes 1, 2, and 3 are for regular data that does change, where class 1 is the highest priority, then class 2, and lastly class 3. This means the master station should poll class 1 data the most frequently and class 3 the least frequently. A poll to class 0 is specifically called an "integrity poll."
The outstation database is modified in database transactions, which give us a uniform way of specifying database point creations and updates. A transaction specifies a handler (the function to call to execute the transaction) and a context (the data to be used by the handler). The transaction handler will contain a call to a method like dnp3_database_update_binary_input() to actually update the database. A transaction is applied with a call like this: dnp3_outstation_transaction(outstation, transaction).
These methods to create database transactions are what need to be exposed to Python from the C API.
References:
- https://www.trianglemicroworks.com/docs/default-source/referenced-documents/DNP3_Overview.pdf
- https://www.dnp.org/Portals/0/AboutUs/DNP3%20Primer%20Rev%20A.pdf
- https://docs.stepfunc.io/dnp3/1.6.0/guide/docs/api/outstation/database
- Provide support for teammates
Going to take a break from technical work this week. Got behind in other classes since I spent a lot of time last week working on the logging issues, midterm paper, and midterm presentation. We didn't have a normal Friday meeting, but I did spend two hours helping Jude get up to speed with the rest of the team members. He still isn't able to get a shared directory working due to some weird permissions issue, so we will try sorting that out in class on Thursday.
- Fix issue in logging
- Write my part of the midterm paper
- Write networking team parts of midterm presentation
Finished writing my part of the midterm paper on DNP3 integration. Here is the main addition describing how the new extension module based architecture works:
I also finished up my part of the midterm presentation. As part of this, I assigned the first coding tasks to the team members.
Also, I found the bug in the logging code. It's a particularly nasty bug involving three different programming languages and multi-threading... I had to pull out all the debugging tools I know of to even figure out where it was happening at. Below is a description of the bug and how I solved (most of) it.
The Rust dnp3 library gives us a simple example for logging: just printf.
void on_log_message(dnp3_log_level_t level, const char *msg, void *arg) { printf("%s", msg); }
dnp3_logger_t get_logger() {
return (dnp3_logger_t){
.on_message = &on_log_message,
.on_destroy = NULL,
.ctx = NULL,
};
}Nice and simple, and the current code works perfectly fine with logging like this.
However, just printing to stdout isn't a nice solution, especially when we want to use a proper logger. Even if we use a solution like Wurlitzer to redirect stdout to Python's logger, we will still have no control over how it shows up in the Python logger.
Ok, so let's call the Python logger from C. That's why I chose to use a Python extension module, after all. How hard can it be?
void log_message(const char *msg, dnp3_log_level_t level) {
static PyObject *logging_library = NULL;
static PyObject *logging_object = NULL;
if (logging_library == NULL || logging_object == NULL) {
logging_library = PyImport_ImportModule("logging");
if (logging_library == NULL) {
PyErr_Print();
return;
}
logging_object = PyObject_CallMethod(logging_library, "getLogger", "s", "master");
if (logging_object == NULL) {
PyErr_Print();
Py_DECREF(logging_library);
return;
}
}
// Just to be 100% safe...
char *dup_msg = strdup(msg);
if (dup_msg == NULL) {
fprintf(stderr, "Failed to allocate memory for log message\n");
return;
}
PyObject *logging_message = PyUnicode_FromString(dup_msg);
free(dup_msg);
if (logging_message == NULL) {
PyErr_Print();
return;
}
const char *method_name;
switch (level) {
// Python's logger doesn't have a trace level
case DNP3_LOG_LEVEL_TRACE:
case DNP3_LOG_LEVEL_DEBUG:
method_name = "debug";
break;
case DNP3_LOG_LEVEL_INFO:
method_name = "info";
break;
case DNP3_LOG_LEVEL_WARN:
method_name = "warning";
break;
case DNP3_LOG_LEVEL_ERROR:
method_name = "error";
break;
default:
method_name = "info";
break;
}
// Returns None, which is an immortal object since Python 3.12 (https://peps.python.org/pep-0683/)
// So no need to Py_DECREF the return value
PyObject_CallMethod(logging_object, method_name, "O", logging_message);
Py_DECREF(logging_message);
}Wow... that's a lot more code for sure. Lots of footguns also - we need proper NULL checking and manual reference counting.
But what if I told you there's more?
Remember that get_logger function from earlier?
dnp3_logger_t get_logger() {
return (dnp3_logger_t){
.on_message = &on_log_message,
.on_destroy = NULL,
.ctx = NULL,
};
}The reason this exists is so the Rust dnp3 library can call on_log_message... and it can call it from multiple threads. Fun, right?
Turns out that this requires special handling since the Python interpreter isn't fully thread-safe. Since our threads aren't being created by Python, they require an even more special handling. I spent a while before realizing this - debugging multi-threaded segfaults is NOT fun.
Thankfully, the fix is fairly simple. Just wrap the function in the ensure and release:
void log_message(const char *msg, dnp3_log_level_t level) {
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
...
// Make sure to release on any early exits
...
PyGILState_Release(gstate);
}Fixed, right?
Not quite. This code works fine, but at the very end of running a simple example, you get a very cryptic error:
FATAL: exception not rethrown
fish: Job 1, 'python run.py' terminated by signal SIGABRT (Abort)
Let's consult gdb:
$ gdb -args python run.py
...
Enable debuginfod for this session? (y or [n]) y
...
(gdb) r
...
End fragment
FATAL: exception not rethrown
Thread 17 "tokio-runtime-w" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff47f06c0 (LWP 858946)]
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6,
no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6,
no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x00007ffff78a56d3 in __pthread_kill_internal (threadid=<optimized out>, signo=6)
at pthread_kill.c:89
#2 0x00007ffff784bba0 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3 0x00007ffff7833582 in __GI_abort () at abort.c:73
#4 0x00007ffff78343bf in __libc_message_impl (fmt=fmt@entry=0x7ffff79c101d "%s")
at ../sysdeps/posix/libc_fatal.c:134
#5 0x00007ffff7898e69 in __GI___libc_fatal (
message=message@entry=0x7ffff79c3e00 "FATAL: exception not rethrown\n")
at ../sysdeps/posix/libc_fatal.c:143
#6 0x00007ffff78ac2d4 in unwind_cleanup (reason=<optimized out>,
exc=<optimized out>) at unwind.c:114
#7 0x00007ffff69da6ec in __rust_panic_cleanup ()
from /home/steel/git/dnp3-example/c_api_example/venv/lib/python3.13/site-packages/dnp3/lib/./libdnp3_ffi.so
#8 0x00007ffff664bcfd in std::panicking::try::cleanup ()
from /home/steel/git/dnp3-example/c_api_example/venv/lib/python3.13/site-packages/dnp3/lib/./libdnp3_ffi.so
#9 0x00007ffff669f054 in tokio::runtime::task::harness::Harness<T,S>::poll ()
from /home/steel/git/dnp3-example/c_api_example/venv/lib/python3.13/site-packages/dnp3/lib/./libdnp3_ffi.so
#10 0x00007ffff6995b4d in tokio::runtime::scheduler::multi_thread::worker::Context::run_task ()
from /home/steel/git/dnp3-example/c_api_example/venv/lib/python3.13/site-packages/dnp3/lib/./libdnp3_ffi.so
#11 0x00007ffff6994f05 in tokio::runtime::scheduler::multi_thread::worker::Context::run ()
from /home/steel/git/dnp3-example/c_api_example/venv/lib/python3.13/site-packages/dnp3/lib/./libdnp3_ffi.so
...That stack trace goes on longer, but something is missing... where is our code? The only non-libc/libpthread code seems to be something called Tokio. This provides the runtime that the Rust library uses to manage threads.
Since the error is happening at the end of execution and involves the Tokio runtime, a logical conclusion is that the issue lies with the runtime destructor.
The C wrapper makes extensive use of Capsules. This provides a convenient way to wrap the dnp3 types without having to replicate their internal structure (like done in some of the types for the ctypes example).
Even better, when creating a capsule, you can specify a destructor! This is great for allowing Python's reference counter automatically do its thing, effectively eliminating the need for a destroy() method in the Device base class. Naturally, when implementing the wrapper, I made use of these destructors.
Here's what that looked like for the init_runtime method:
static PyObject* py_init_runtime(PyObject *self, PyObject *Py_UNUSED(ignored)) {
dnp3_runtime_t *runtime = init_runtime();
if (runtime == NULL) {
Py_RETURN_NONE;
}
return PyCapsule_New((void*)runtime, "dnp3_runtime_t", runtime_destructor);
}But of course, lifetimes are always tricky. Getting rid of the capsule destructor and manually calling the destructor method from Python got rid of the FATAL: exception not rethrown error for some reason still unknown to me.
And since nothing is easy, it led to a new error - the code now hangs. After some painful debugging and rr magic, I was able to find the actual lines of code it was hanging on:
void log_message(const char *msg, dnp3_log_level_t level) {
...
// One thread hangs here
logging_object = PyObject_CallMethod(logging_library, "getLogger", "s", "master");
...
}
static PyObject* py_destroy_runtime(PyObject *self, PyObject *args) {
...
// And another hangs here
destroy_runtime(runtime);
Py_RETURN_NONE;
}So it wasn't a regular hang, it was a deadlock.
Why? No clue. There's three programming languages, multi-threading for a language that doesn't support threading well, and weird object lifetimes. I cannot be bothered to figure out why.
I'm not the only one who's had the issue though, and there doesn't seem to be a particularly good solution.
- https://stackoverflow.com/questions/29100082/python-gil-and-threads
- https://stackoverflow.com/questions/47167251/pygilstate-ensure-causing-deadlock
- https://bugs.python.org/issue38680
- https://stackoverflow.com/questions/77243572/python-gil-and-deadlock-detection
- https://github.com/PyO3/pyo3/issues/1305
There are actually two ways to fix this issue. The first is very simple - just set a timeout on the runtime shutdown. Looks like the Rust dnp3 library creators might've seen an issue similar to this at some point since the API provides a method to do just this. Obviously, this is unsavory since it might leak memory.
The second fix is also hacky: manually check if the runtime is shutting down before logging through a global variable, is_shutting_down. This at least doesn't leak memory, so it's worth using both methods (this as the primary, and the timeout as the backup).
Here's what the final result looks like:
atomic_bool is_shutting_down = ATOMIC_VAR_INIT(false);
static void on_log_message(dnp3_log_level_t level, const char *msg, void *ctx) {
static PyObject *logging_library = NULL;
static PyObject *logging_object = NULL;
if (atomic_load(&is_shutting_down)) {
return;
}
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
if (logging_library == NULL || logging_object == NULL) {
logging_library = PyImport_ImportModule("logging");
if (logging_library == NULL) {
PyErr_Print();
PyGILState_Release(gstate);
return;
}
logging_object = PyObject_CallMethod(logging_library, "getLogger", "s", "master");
if (logging_object == NULL) {
PyErr_Print();
Py_DECREF(logging_library);
PyGILState_Release(gstate);
return;
}
}
PyObject *logging_message = PyUnicode_FromString(msg);
if (logging_message == NULL) {
PyErr_Print();
PyGILState_Release(gstate);
return;
}
const char *method_name;
switch (level) {
// Python's logger doesn't have a trace level
case DNP3_LOG_LEVEL_TRACE:
case DNP3_LOG_LEVEL_DEBUG:
method_name = "debug";
break;
case DNP3_LOG_LEVEL_INFO:
method_name = "info";
break;
case DNP3_LOG_LEVEL_WARN:
method_name = "warning";
break;
case DNP3_LOG_LEVEL_ERROR:
method_name = "error";
break;
default:
method_name = "info";
break;
}
// Returns None, which is an immortal object since Python 3.12 (https://peps.python.org/pep-0683/)
// So no need to Py_DECREF the return value
PyObject_CallMethod(logging_object, method_name, "O", logging_message);
Py_DECREF(logging_message);
PyGILState_Release(gstate);
}
void destroy_runtime(dnp3_runtime_t *runtime) {
dnp3_runtime_set_shutdown_timeout(runtime, 5);
atomic_store(&is_shutting_down, true);
dnp3_runtime_destroy(runtime);
}
static PyObject* py_destroy_runtime(PyObject *self, PyObject *args) {
PyObject *runtime_capsule;
if (!PyArg_ParseTuple(args, "O", &runtime_capsule)) {
return NULL;
}
dnp3_runtime_t *runtime = PyCapsule_GetPointer(runtime_capsule, "dnp3_runtime_t");
if (runtime == NULL) {
PyErr_SetString(PyExc_TypeError, "Invalid runtime capsule");
return NULL;
}
destroy_runtime(runtime);
Py_RETURN_NONE;
}Sadly, a consequence of this is needing a destroy() method in the Device base class. If anybody wants a particularly tough challenge, you could try to get the capsule destructor working, but good luck.
While that solution works for just the on_log_message callback, once I tried calling on_log_message from other functions in the default example that print (such as client_state_on_change), I ran into yet another deadlock. Ugh.
Some functions work fine though - for example, replacing the printf in begin_fragment with a call to the logger works fine:
void begin_fragment(dnp3_read_type_t read_type, dnp3_response_header_t header, void *arg)
{
char* string;
if (asprintf(&string, "Beginning fragment (broadcast: %u)\n", header.iin.iin1.broadcast) < 0) {
fprintf(stderr, "Failed to allocate memory for log message\n");
return;
}
on_log_message(DNP3_LOG_LEVEL_INFO, string, NULL);
free(string);
}What this tells us is that the issue isn't with on_log_message. The issue is some form of interdependence between the on_log_message calls and the other callback functions like client_state_on_change.
Using GDB to see which calls were deadlocked shows three different threads stuck on the following functions:
- dnp3::tcp::master::client::spawn_master_tcp_client_2 (source code)
- dnp3::tcp::outstation::server::Server::bind_no_spawn (source code)
- dnp3_ffi::master::functions::master_channel_add_association (source code)
The first is the one actually calling the client_state_on_change callback that triggers the log call. The second is logging about the outstation accepting connections, and I actually found two threads on this for some reason. The third is creating the association for the master channel.
I'm not 100% sure, but I'm fairly convinced that there is some deadlock here due to interaction between the async operations, thread spawning, and blocking in the Rust code plus locking the GIL in the C code. I don't believe this is a solvable issue without more investigation (and potentially modifications to the Rust library), and I have already spent WAY too much time on this issue. As such, there will be some callbacks that just don't log.
THE FOLLOWING IS SUBJECT TO CHANGE
Currently, I'm still struggling with getting both the outstation and master to write to the log when the outstation (technically the TCP server) and master are handled by the same process. It's not an issue with my logging code - it doesn't even work with printf. When I run the master and outstation separately (through fork or in a separate terminal), it works fine. But when I run both in the same process, only the first one that gets initialized has the on_log_message called. In fact, the on_log_message doesn't even show up in the GDB disassembly.
- Finish new example in dnp3-example repo using Python C extension
- Finish my part of the DNP3 documentation (implementation)
It works!!! After over a week of struggling, I have DNP3 integrated into Python as a C extension module. However, my code likely has some memory issues, and I'm still not able to get logging working. Because of this, I'm still not able to start integrating it into the cosimulation repo and I'll also hold off on writing documentation until I get everything working.
Some resources that helped me:
- https://setuptools.pypa.io/en/latest/userguide/ext_modules.html
- https://stackoverflow.com/a/63837811
- https://docs.python.org/3/c-api/init.html#non-python-created-threads
- To debug:
gdb --args python somefile.py
- Write guide for QEMU
- Finish new example in dnp3-example repo using Python C extension
I wrote up the guide for QEMU below. My other goal was to finish the DNP3 example. However, I ran into some technical issues with linking the dnp3 library into the Python extension module. I'm close to getting it working, so I will write the entry detailing the issues and solution next week.
At our Thursday and Friday meeting, I helped the other members with the VM guide, and also set up a Google Doc for our internal DNP3 documentation.
The files are hosted on my personal GT OneDrive since nowhere else has >2GB for free. Download the files there.
First install qemu: https://www.qemu.org/download/
- If you downloaded the files as a zip, unzip them and delete the zip file. Open a terminal in the directory with the unzipped files.
- Open a second terminal in the same directory
- Make scripts executable:
chmod +x run.sh connect.sh - Make a directory for sharing files between the host and VM:
mkdir share - In the first terminal, type
./run.sh share/ - In the second terminal, type
./connect.sh - Wait for the second terminal to display a login. It might take a while. If it starts taking >1 minute, type Ctrl+c to cancel the command and retry it.
- You should see that you are logging into the VM with username
networking. I made the password the same as the username:networking. The password won't display while you are typing. - You should now be logged into the VM. Type
ls -laand you should see some files already there named setup_linux.sh, setup_mac.sh, and setup_windows.sh. - If on Linux, run
./setup_linux.sh. If on Mac, run./setup_mac.sh. - It will prompt for your password. Enter
networking. - If nothing happens, then you are good. That setup script should've activated the share directory, so test it out by doing
touch share/test. - Now, on your host machine, navigate to the same directory you've been working out of (the one with the directory named share that we created in step 4). Look inside that share directory, and you should see a file named
test. This is the same one we created in step 12, which means that the shared directory between the host and VM is working. - Everything is good to go. Keep the first terminal open (the one that says "Welcome to Grub!") in the background. This is running your VM in the background, while the second terminal is ssh'd into it.
- If you downloaded the files as a zip, unzip them and delete the zip file. Open Powershell and navigate to the directory with the unzipped files
- Open a second Powershell in the same directory
- Open File Explorer in the same directory
- Right click on run.ps1, click on Properties. At the bottom of the new popup, check
Unblock, then press Apply and OK. - Repeat step 4 for connect.ps1
- Make a directory for sharing files between the host and VM: from Powershell, run
mkdir share, or create it in File Explorer. - In the first terminal, type
./run.ps1 - In the second terminal, type
./connect.ps1 - Wait for the second terminal to display a login. It might take a while. If it starts taking >1 minute, type Ctrl+c to cancel the command and retry it.
- You should see that you are logging into the VM with username
networking. I made the password the same as the username:networking. The password won't display while you are typing. - You should now be logged into the VM. Type
ls -laand you should see some files already there named setup_linux.sh, setup_mac.sh, and setup_windows.sh. - Now, we need to edit setup_windows.sh to point to the correct share directory. Type
vi setup_windows.sh. - If you aren't familiar with vi/vim, follow these next steps very closely. Press the following keys: jjci" Then type in the path to the share directory you created on your Windows machine. For example, User/{YOUR_USERNAME}/Documents/share. Press escape, then j, then ci" Now type your Windows username. This will probably be the same as the username in the path you just entered. Press escape, then :wq, then press enter
- Run the command
./setup_windows.sh. - When prompted for a password, enter
networking. - When prompted for another password, enter the password to your Windows machine.
- If nothing happens, then you are good. That setup script should've activated the share directory, so test it out by doing
touch share/test. - Now, on your host machine, navigate to the same directory you've been working out of (the one with the directory named share that we created in step 4). Look inside that share directory, and you should see a file named
test. This is the same one we created in step 15, which means that the shared directory between the host and VM is working. - Everything is good to go. Keep the first terminal open (the one that says "Welcome to Grub!") in the background. This is running your VM in the background, while the second terminal is ssh'd into it.
To shutoff the VM, you have two options. First is powering it off from the ssh session (second terminal) by typing sudo shutdown -h now, and entering networking for the password when prompted. Second, with the first terminal in focus, press Ctrl+a, then press x after. This will completely shutdown the VM. To restart it, run ./run.sh share/ again.
If instead you want to disconnect from ssh while keeping the VM running in the background, press Ctrl+d from the second terminal. To reconnect, run ./connect.sh.
Any work you do on this VM will persist in the debian.qcow2 file, so as long as you don't delete that, your work will be fine. Remember though, run the ./setup_{os}.sh every time you start up the VM (step 10 for Linux/Mac, steps 12-14 for Windows).
There are two ways you can use this VM:
- Do your development work inside the VM
- Keep your work in the share directory. Edit the files there on your host machine (so you can use the text editor and environment you are comfortable with), and just use the VM for running the simulation.
If you choose to go with the first option, you can modify run.sh to have it open as a graphical session. This would eliminate the need to run ./connect.sh, and you could also install an entire desktop environment.
If you choose the second option, just clone all your repos and keep all your files in the share directory on the VM. Do all text editing from your host machine, and whenever you need to run something, do it from inside the VM.
The VM has very, very minimal tools right now. The only thing I've preconfigured is Nix since it can be a bit finicky. The VM is running Debian Linux, so the package manager is apt. Follow along with these commands to get the dnp3-example repo running. If prompted for the password at any point, type networking.
sudo apt install git -y
# Clone the repo and change directory into it
git clone https://github.com/cps-vip/dnp3-example.git
cd dnp3-example
# Install required dependencies. Look at the README file for the updated dependency list.
cat README.md
# Follow the instructions in the README
cd default_example
# Create a build directory for CMake to place its files
mkdir build
cd build
# Build the project
cmake ..
make
# From here, you will need to open another terminal through ssh. This means repeating steps 2, 6, 7, and 8 to get another terminal open.
# In that newly connected terminal, run these commands:
cd dnp3-example/default_example/build
./outstation_example tcp
# Now in the other terminal, run this:
./master_example tcp
# If you see a bunch of text on both terminals, then the example is working.
# Press Ctrl+c on both terminals to stop the example.Now for the cosimulation repo (the one this Wiki is in). You can close one of the terminals (doesn't matter which). If you feel comfortable, you can also follow the README from step 2 and on, but here are the exact commands to run. Warning: all of these commands take a while, so be patient.
# Clone the repo
cd
git clone https://github.com/cps-vip/cps-cosimulation-env.git
cd cps-cosimulation-env
# This step may or may not be needed depending on how powerful your computer is
# Follow these next inputs very closely and exactly. If at any point you mess up, press <Escape>:wq<Enter>.
# You don't type the <>, just press the escape and enter key.
vi Makefile
# Once inside vi, type the following
23ggf$cE1<Escape>:wq
# Enable nix develop
sudo vi /etc/nix/nix.conf
# Follow these next steps closely.
Press o
Type this: experimental-features = nix-command flakes
Press escape
Type :wq
# The git repo points to other GitHub repositories that we need to bring in
git submodule update --init --recursive
# This will take a while - you may have to press enter at some point if it looks like it has stalled
nix develop
# This will also take a while
make all-install
cd simple_gridlabd_example
./run_example.shFirst, install the packages for xfce4, a light-weight desktop environment:
sudo apt install xfce4
sudo apt install xfce4-goodies
sudo shutdown -h nowNow edit the run file (run.sh on Mac, run.ps1 on Windows). Just remove the line that says "-nographic " and run the script again. This time, it will open a new graphical window. You don't need to run ./connect.sh anymore in a second terminal.
When the VM boots up, it should show a graphical login. Username is "networking" and password is "networking". Now you should be logged into a desktop like the one in the screenshot I attached. The button on the top left that says "Applications" is how you launch things.
- Help get new members get up to speed
- Create QEMU image
- Figure out best way to do DNP3 logging
Now that the kickoff presentation is done, I'll be able to spend time helping the new members of the networking team get everything running (namely the dnp3-example repo, simple_gridlabd_example, and the WIP integration of DNP3 into the main repo). From there, they can hopefully start on the DNP3 documentation task, while I spend my time finishing some of the work that we didn't get to last semester (like logging). Under the goals I laid out last week, I will be working on #2, and the other members will be working on #1.
At our Thursday meeting, I gave a demo of our current work to the new members, and helped them with getting the dnp3-example repo. However, we encountered an issue because they are using Macs. I believe the issue is inherent to the Rust dnp3 library, so it isn't something fixable on our side. This leaves two options: Docker or a VM.
Traditionally, Docker isn't used for hardware emulation, although it does seem to support that. Instead, I created a QEMU image, since it's a tool I'm familiar with, and would guarantee that it works on all devices.
I created a Debian VM with a minimal install, and a corresponding script to run and ssh into. Unfortunately, I wasn't able to get the image below 2GB for uploading to GitHub or Dropbox, so for now I'm storing it on my GT OneDrive. The login username is networking, and the password is also networking. Root's password is root. I may also create another image that has the cps-cosimulation-env repo pre-installed, built, and ready for development. I haven't done it yet because it would deprive new members of the learning experience, plus the image would probably get upwards of 8GB.
I'll explain how to use the VM and provided scripts in the QEMU guide, so I'll write here about how they work.
run.sh:
#!/usr/bin/env bash
# These next few lines just make sure the argument passed in is a valid directory
if [[ $# -ne 1 ]]; then
echo 'Please specify the path to the shared directory on the host'
exit 1
fi
if [ ! -d "$1" ]; then
echo "Specified shared directory $1 does not exist."
exit 1
fi
qemu-system-x86_64 -hda debian.qcow2 \
-m 8192 \
-smp 4 \
-netdev user,id=net0,hostfwd=tcp::5555-:22 \
-device virtio-net,netdev=net0 \
-fsdev local,id=fs0,path=$1,security_model=none \
-device virtio-9p-pci,fsdev=fs0,mount_tag=share \
-nographic \There's a lot going on in that qemu command. I'll break it down:
- qemu-system-x86_64 - this is the binary to emulate a full x86_64 processor.
- hda debian.qcow2 - the virtual disk image that will be ran.
- m 8192 - amount of RAM for the guest VM in MB, 8 GB
- smp 4 - number of CPUs for the guest VM
- netdev user,id=net0,hostfwd=tcp::5555-:22 - opens up port 22 (ssh) on the guest for port 5555 on the host
- device virtio-net,netdev=net0 - creates a virtual network device in net0 (specified in line above)
- fsdev local,id=fs0,path=$1,security_model=none - specifies a filesystem device with the path specified by the argument to run.sh
- device virtio-9p-pci,fsdev=fs0,mount_tag=share - creates a virtio-9p-pci device on fs0 (specified above) with a mount_tag share (to be used by mount)
- nographic - runs qemu in the terminal. Removing this will open an interactive GUI.
The flags that can be modified are m, smp, and nographic. For additional speed, look if your machine supports any of these accelerators: https://www.qemu.org/docs/master/system/introduction.html
The most common would be KVM for Linux, in which case you add -accel kvm, or Hypervisor Framework for MacOS, in which case you add -accel hvf. In my experience, whpx didn't work.
connect.sh:
#!/usr/bin/env bash
# TERM=xterm fixed a bug in Linux. Not sure if it's relevant in Windows or Mac, but it works so I kept it.
# Rest is a normal ssh on the port 5555 we specified in connect.sh
TERM=xterm ssh networking@localhost -p 5555The biggest challenge was getting shared directories working. On Friday, I guided Piper through using the VM, and we ran into a permissions issue with Mac. Thankfully, this GitHub comment has the fix.
Even my Linux machine was having some issues with the shared directory when I mounted it through /etc/fstab, so to solve this issue, I created three scripts on the Debian VM to set up the shared directory (for Linux, Mac, and Windows).
setup_mac.sh:
#!/usr/bin/env bash
# We need two directories for this to work. One will have the wrong permissions, so I name it .bad_share to keep it hidden.
mkdir -p /home/networking/.bad_share
mkdir -p /home/networking/share
# This is the only command needed on Linux - just a standard mount using 9p with virtfs. The file permissions on Mac get messed up though.
sudo mount -t 9p -o trans=virtio share /home/networking/.bad_share
# To fix the permission issue, we remap the .bad_share mount with the incorrect permissions (uid=501, gid=@dialout)
# to the correct persmission (uid=1000, gid=1000 which correspond to the networking user/group)
sudo bindfs --map=501/1000:@dialout/@1000 /home/networking/.bad_share /home/networking/sharesetup_windows.sh (modified from https://gist.github.com/neelabhg/8c6bfcdb44c0aa3f86bed00fe1f6da49)
#!/usr/bin/env bash
# With Windows, we can use the sharing feature of folders to specify a host folder from the guest VM.
# WINDOWS_DIR_PATH is the path to the folder we want to share on the Windows host
# WINDOWS_USERNAME is the username of the owner of that folder (so whatever you are logged in as)
WINDOWS_DIR_PATH="Users/Kaden/Documents/myshare"
WINDOWS_USERNAME="Kaden"
# This mounts using cifs, which is able to work between Windows/Linux
# We set the uid and gid of the mount to the current uid and gid (which will be 1000/networking)
# 10.0.2.2 is specifically set up by QEMU for a use like this to refer to the host machine
sudo mount -t cifs -o user="$WINDOWS_USERNAME",uid=$(id -u),gid=$(id -g) "//10.0.2.2/$WINDOWS_DIR_PATH" /home/networking/shareOf course, because the guest is now the one referencing the host's shared directory path, the run script is also modified to remove the shared directory argument.
run.ps1:
& "C:\Program Files\qemu\qemu-system-x86_64.exe" -hda debian.qcow2 `
-m 8192 `
-smp 4 `
-netdev user,id=net0,hostfwd=tcp::5555-:22 `
-device virtio-net,netdev=net0 `
-nographicFor the rest of the VM, I kept things very minimal. The only things I installed are the tools for shared directories (bindfs for Mac and cifs for Windows) and Nix (since the setup can be a bit complicated).
I also made a bit more progress on logging. I was able to get a C extension able to use Python's logger, and I started writing a new example in the dnp3-example repo that uses a C extension instead of ctypes. I'll write more about this once I'm finished with it.
- Finish kickoff presentation
- Get logging to work in main repo
Only real goal for this week is finishing up the kickoff presentation. These are the four tasks I laid out in the presentation:
- Gain better understanding of DNP3 protocol, library, and usage. Outcome: Beginner-friendly document describing DNP3 internals and its typical use in substations
- Finalize Python Outstation and Master API and Implementation. Outcome: Create bindings for all relevant methods from the Rust library; integrate HELICS (for timestamps)
- Create demo of Master to Outstation communication. Outcome: Demonstrate Master station detecting change in a device and operating on updated info
- Start integration of DNP3 into device models. Outcome: Work with device modeling sub-subteam to start using DNP3 API in Python models
I also remembered that last semester, I started some work on integrating logging. I made a few changes to the Networking_team branch to get run_simulation.py working, and have logging work across the different devices. I still have to figure out how to get DNP3 logs to go to the same log though, since that code is written in C. I looked at two solutions: wurlitzer, a way to redirect std out in C to a Python logger, and Python extension modules, which would allow me to get access to the Python logger directly in C.
Each have some downsides. Wurlitzer works fine, but it makes all DNP3 logging messages appear as INFO logs, which is very undesirable. Creating an extension module is much more difficult - I tried for a few hours, but couldn't get it working.
Specifically, I'm able to get Python calling C (through ctypes), and C calling Python (as a Python extension module), but when I try to combine both, I get a segfault. This indicates that I need to stick with one or the other. Currently, all Python-and-C interop is done through ctypes, so I might have to undergo the task of converting all existing code to be a Python module. On the upside, this makes using the C code from Python much easier, exposing easier to read code for the device modeling team.
On Thursday, the kickoff presentation went well. We decided to not meet on Friday, and it didn't seem like there would be a large turnout anyways.
Model how networking works in a substation (and potentially between multiple substations). The long term goal is to do security research on substation networking, but that will come after we've established an accurate simulation.
Python, C, basic networking concepts (OSI model). Please read up on the OSI model if you aren't familiar with it. The bulk of our work will be in Python.
DNP3 is a specialized networking protocol made for communication within a substation. We use DNP3 over the TCP/IP stack, but DNP3 can also run independently of TCP/IP, using its own link layer and transport layer. DNP3 packets still contain this link and transport layer information as a historic relic. For example:
When looking up other communication protocols, you will see references to Modbus and IEC 61850. We chose DNP3 over Modbus since DNP3 is more efficient, modern (although still outdated), and feature-rich. IEC 61850 is even more modern and feature-rich, but this comes with additional complexity that we didn't want to deal with. IEC 61850 is also used more in Europe, so it isn't as relevant.
Last semester, we went from no networking at all to a very basic example in the main repository on the Networking_team branch. Nothing about the example is finalized.
The main files to look at are the Master and Outstation, as well as the C files in the dnp3 directory.
I'll give a basic rundown of what we did last semester. Since our substation devices are written in Python, we need a way to use DNP3 from Python. There already exist libraries for this (e.g., dnp3-python and pydnp3), but these are both based on opendnp3. However, opendnp3 went end-of-life in 2022, so we had to find an alternative.
Thankfully, the maintainers of opendnp3, Step Function I/O, created a new library in Rust just called DNP3. The work of our previous semester has been to create a Python wrapper around this Rust library, similar to what dnp3-python and pydnp3 did for opendnp3.
There are two ways to call the Rust library from Python:
- Use a third-party library like PyO3 and Maturin to directly call Rust from Python, or
- Utilize one of the existing foreign function interfaces (FFI)
Since I don't know Rust, I decided to go with the second option. The Rust library exposes functions that you can call from C, which we then call from Python. I created an example repo to demonstrate this working. (Note that the C code is just a slightly modified version of the example C code, so don't feel the need to understand all of it.) Feel free reach out to me if you have any questions on how to get the example running or how it works. More details are available earlier in my notebook.
After getting the example running, we started integrating DNP3 into the cps-cosimulation-env repo. What currently exists is very work-in-progress, since it was quickly cobbled together for last semester's final presentation.
- While not necessary for what we are doing, get the simple_gridlabd_example working (see the instructions on the main README for this repo)
- Read up on the OSI model and TCP/IP basics if you aren’t already familiar
- Learn the basics of DNP3
- Take a look at the Rust library's documentation
- Understand what's happening in the DNP3 example repo
Understanding DNP3 better - I know only the very basics of how DNP3 works. Most of the stuff we’ve worked on so far hasn’t required knowing how you use DNP3 in practice since the Rust library handles all of that for us. Having someone proficient in DNP3 would be extremely useful, even if it’s just to explain details to the people who will be implementing it.
Python API and Implementation - We need to determine how a substation device will interface with the Rust library. This involves creating more methods in the Outstation class that individual devices can call.
C work - Since we call the Rust library through the C FFI, we need to do some work in C. This mostly involves wrapping the existing C functions in a way that’s easier to call from Python.
HELICS integration - Somehow, the substation needs to communicate with the rest of the cosimulation through HELICS. We need to determine how to do this. For example, does every substation device publish to HELICS, or does everything go through the Master station? Another problem here is timing - we want DNP3 timestamps to be consistent with the rest of the cosimulation.
Human-Machine Interface - Real substations have GUIs to interact with the devices in the substation. Currently, we have no progress or even a starting point for this, but it will need to be done at some point.
Inter-substation and inter-protocol communication - So far, we are building code around one substation and one communication protocol. In the future, we may want to be able to model multiple substations and other protocols. The current design accommodates inter-substation communication by having the Master inherit from the Substation class, and inter-protocol communication can be done through a data concentrator device. This is long-term and so lower priority, but if you are interested, you can still work on it.
Let me know which of these areas you are interested in working on.
- Set up subteam meeting time
- Get started on kickoff presentation
- Get new members of networking subteam up to speed
This week was mostly finalizing subteams and starting the kickoff presentation. I also created the above guide for new team members.
First week of the semester. Nothing to be done yet.
- Finish final paper
The only goal for this week is finishing the final paper, due on 12/03. I'll leave some notes here for our current state of progress to pick up on next semester.
Currently, all the work on DNP3 integration is in the Networking_team branch. So far, this includes:
- Initial implementation of DNP3Outstation
- All devices now inheriting from DNP3Outstation
- Initial implementation of DNP3Master
- A working dnp3 demo
Future work:
- Bring in all relevant database transaction methods to the DNP3Outstation class
- Get rid of the run() methods in DNP3Outstation and DNP3Master (these exist only for the demo)
- Implement proper logging (I have made some progress on a local branch for this, but not enough to push)
- Start integrating DNP3 communication into more devices
- Create some methods in DNP3Outstation and DNP3Master for changing the default config
- Figure out how the DNP3Master should fit in with the StationGateway and HumanMachineInterface
- Create a demo of the master receiving some data from a device, then acting on that data to send a command to a different device
- Integrating DNP3 with HELICS (e.g., timing, what devices should be exposed to HELICS, etc.)
- Long term: multiple substations, security, Modbus/IEC 61850 devices
Most of the week was Thanksgiving Break. We briefly met on Sunday (Chungman, Maria, Alex, and I) to talk about the final paper. We are mostly just updating and extending what we wrote for the midterm paper. Here is the final paper.
- Finish final presentation slides
This week, my only real focus was on the final presentation. I updated my slides on DNP3 integration from the midterm presentation, which is mostly the same material, but with a few more technical details about the implementation in the cps-cosimulation-env repo. Here's the diagram I added:
This is just a more compact overview of the flow of code than the class diagram. I've explained this exact design in previous notebook entries.
Adam was able to finish a DNP3 example in the cps-cosimulation-env repo, and I was able to get it working also. Here are the steps (these may change since this work is still in an unfinished/unmerged branch):
cd device_modeling
mkdir build
cd build
cmake ..
make
cd ..
cd src
./run_dnp3_demo.sh
# Logs are stored in the results directoryOur final presentation was on Thursday. Here are the slides.
After our final presentation, we decided to not have a meeting on Sunday since there wasn't much to be discussed.
- Determine architecture of DNP3 integration with the cps-cosimulation-env repo
- Start integration of DNP3 in cps-cosimulation-env
Chungman and I made good progress on the class diagram for the DNP3 integration. Chungman did the diagramming while I helped with the specification. This is mostly a formalization of my thoughts from the last entry.
The following is a brief explanation of the different hierarchy levels:
- There is a Device base class that all substation devices will inherit from. The purpose of this is fairly obvious.
- All of the currently existing devices will inherit from a DNP3 Outstation class. If in the future we want other protocols, the devices will inherit from a Modbus/IEC Outstation class. This is because abstracting away the underlying protocol is very difficult.
- Devices within a substation are broken down into three types: station, process, and bay level. Station level devices are the top level devices involved in substation-wide tasks, like the master station and the substation gateway that facilitates external communication. Bay level devices monitor and control devices at the process level. Process level devices are those performing traditional substation functions (e.g., a transformer).
- Within the process level are various devices which may communicate with each other through direct serial connections, which we will be modeling using inter-process communication (IPC), likely using pipes.
At the Thursday meeting, Chungman and I worked together on this. Adam was also there. He hopes to finish up the master part of the DNP3 example.
For our Sunday meeting, we discussed upcoming plans for the next week. The final presentation is next week, but we still decided to try to get a simple example of DNP3 working in the cps-cosimulation-env repo. Since the Thursday meeting, Adam mostly finished getting the Master working in the DNP3 example repo.
After the meeting, I had to fix up a few things in the example repo (I never actually called the run_outstation input loop). The example repo should be finished for the time being. After that, I worked on implementing the class diagram Chungman and I worked on into the cps-cosimulation-env repo [commit]. I created a DNP3Outstation class in cps-cosimulation-env/device_modeling/src/dnp3/dnp3_outstation.py, and modified how the Device base class lifetime works. Now, instead of just active or inactive, there are states for INITIALIZED, ACTIVE, INACTIVE, and DESTROYED. This allows for more flexibility and better integration with DNP3.
I assigned Chungman to update documentation related to these changes, and Adam to finishing the integration (specifically getting the Master added to the repo and changing all the existing devices to inherit from the DNP3Outstation class).
- Determine architecture of DNP3 integration with the cps-cosimulation-env repo
Adam wasn't able to make it to the Thursday meeting, so we weren't able to discuss more about the master implementation. Instead, I chose to pivot focus on how DNP3 will fit into the actual cosimulation.
The repo currently has a very bare-bones layout of how communication would work between devices. Since none of the communication is implemented (for example, the communications directory contains empty files), we decided to completely redo any existing progress (almost nothing). However, the infrastructure for devices themselves is there; notably, there is a Device base class that all devices inherit from.
As such, the first step is determining the new architecture. Here are some design decisions:
- Do we want the communication interface to be abstract enough to allow integration of other protocols (Modbus, IEC 61850) in the future?
- Should the Device class inherit from an Outstation, or should it hold an Outstation object? (Composition vs Inheritance)
- Are there any methods that a device should have to implement for communicating?
- Do we expose an interface for individual devices to modify the default DNP3 settings?
- How do we handle the lifecycle of DNP3 outstations/TCP servers?
- What methods should the Device class expose to children? (This is the big one)
I asked Chungman to move away from his current task of normal device modeling (that should be handled by the device modeling sub-subteam anyways) and to help with this task, specifically for creating diagrams of the architecture we land on.
Here's my takes on these questions:
- We should stick with just DNP3 for now. Other protocols can come in the future, and if necessary, we can use the idea of a concentrator to facilitate inter-protocol communication.
- Since we are tightly coupling to DNP3, it's fine for our class hierarchy to be tightly coupled. Inheritance makes things marginally simpler here.
- I think there should be an abstract method for initializing the values in the Outstation database.
- Not until there is a need to. Obviously, a few options must be exposed, like the DNP3 address, but besides those, I don't see an immediate need for more flexibility. Adding methods for this in the future should be fairly trivial.
- The Device class will expose methods to start and stop communication. It is on the creator of the device to ensure the stop method is called before the object is destroyed; otherwise, memory will leak. I think the easiest way to achieve this is with Python's context manager.
- At the very least, there needs to be the aforementioned start and stop methods, as well as method(s) for making outstation database transactions.
The Sunday meeting was very short. Chungman and I will continue working on this next week. Adam didn't show up, so I'm not sure what the status is on the master station implementation
- Figure out master implementation of multiple outstations
A master station communicates over a master channel. A master channel can be created from any underlying transport, but we will be using TCP. To establish a connection to an outstation, you add an association to the master channel. An association is defined as "'a representation of a logical connection between a master and an outstation'" (DNP3 Documentation).
Initially, I thought we could easily add multiple associations per channel through passing in multiple TCP server addresses to the endpoints list that gets passed into dnp3_master_channel_create_tcp. This is even backed up by the documentation with this image:
However, if you try adding multiple addresses to the endpoints list, what happens is "the channel will round-robin through this list until it establishes a connection" (DNP3 Documentation).
There is also a method to add an association to a master channel, named dnp3_master_channel_add_association. It takes in a DNP3 address through, and since the endpoint list only contains the IP/port of one TCP server, there is no way for the master channel to figure out the IP/port of the other TCP server. This is backed up by the note on the above image.
So, I think the only solution is each master channel only having one association. This means that each outstation has its own TCP server, its own association, and its own master channel. I'm not sure why this is the design that the library/DNP3 took, but I'm guessing its related to DNP3 starting out as its own serial protocol, independent of the TCP/IP stack.
This is most of what we discussed at the Thursday and Sunday meeting.
- Figure out the multiple outstation issue
- Finish wrapping the master functionality
I also wasn't able to do much this week because of the same large project. On our Thursday meeting, we had the lecture on DNP3, so there wasn't too much time to discuss afterwards. Adam, Alex, and I were able to talk a bit about the DNP3 implementation challenge with multiple outstations. We are fairly confident that each outstation needs its own TCP server, so Adam and I will be working on changing that up in the example. Adam is also working on wrapping the master now.
We also talked about timing with HELICS and DNP3, which was a task that Alex had. We are hoping this shouldn't be bad, since we are able to provide a function to the Rust backend that determines the current time. Our hope is we can just return the HELICS time in that function, and everything will work fine.
On the Sunday meeting, there wasn't too much progress to discuss. The main topic was the questionnaire that the ROS-HELICS team gave us, so we divvied out responsibilities for that. Other than that, everyone is still working on the same topics.
- Get DNP3 example running
The first half of this week was fall break, and the second half I had a large project due, so I wasn't able to get much done.
On the Thursday meeting, I pushed for us to get some actual progress made by ensuring everyone had proper tasking. Namely, Maria and Chungman were assigned to figure out what devices we want to model. This really should've been done by the device modeling sub-subteam, but they don't show up to many meetings, so it's hard to communicate this with them.
Since Adam and I are working on the DNP3 example, I gave him an issue I found. You seemingly can't add multiple outstations to the same TCP server if the outstations connect to the same master. I thought the purpose of the TCP server was to have multiple outstations communicate through it, but my assumption seems to be wrong. I gave Adam the task of figuring out how this is supposed to work, specifically by trying to get multiple outstations working with the default example.
On our Sunday meeting, we had some actual progress. The turnout was quite low though - only Maria, Adam, Chungman, Nicholas, and I were present. Maria and Chungman did good research on substations and created some documents describing the devices we should be modeling. Adam broke a finger, so he understandably wasn't able to make much progress.
I proposed the next two tasks for Chungman and Maria:
- Look at the existing device models. Create any that you found that aren't already there.
- Figure out how we want the communication architecture to look like.
Talking a bit more about #2, the current device communication is non-existent. There are a few files in the communication directory for DNP3, Modbus, and IEC 61850 interfaces, and each device has a field for communication protocol. However, none of the communicate() functions are filled out, and the protocol interface files are completely empty. Since we aren't totally sure what the original author had in mind, we will just redo the small amount of existing work. We discussed a few possible implementations, but we didn't decide on anything concrete.
- Get DNP3 example running
- Finish midterm presentation
I finished my slide on DNP3 integration for the midterm presentation. Since I've worked on the example more since the midterm paper, I updated the main figure showing the dependency chain.
Note that this figure is high level and incomplete - there's some more coupling between the TCP server and outstation that isn't shown here.
As for actual implementation, I completed the code for the outstations and TCP server. However, when it came to getting the code to run, I encountered some issues. One of the more tricky parts of the project is building, since the goal is using Python to call a shared library written in C wrapping a shared library written in Rust. The default DNP3 C example comes with a CMake file, but I discovered an issue with it. If you move the executables that it creates (master_example and outstation_example), they stop working! This is because the executables are linked incorrectly to libdnp3_ffi.so (the FFI generated from Rust). This issue also happens for our use case, which is creating a shared library that depends on libdnp3_ffi.so.
When using CMake to create a shared library, it automatically adds a soname (shared object name) to the ELF. Let's see an example:
mkdir test
cd test
echo -e "#include <stdio.h>\nvoid main() {\n printf(\"Hello, World!\");\n}" > hello.c
echo -e "cmake_minimum_required(VERSION 3.9)\nproject(hello)\nadd_library(hello SHARED hello.c)" > CMakeLists.txt
mkdir build
cd build
cmake ..
make
readelf -d libhello.so | head -5Taking a look at the readelf output, we can see the dependency on libc, and also a soname of "libhello.so". For whatever reason, libdnp3_ffi.so doesn't have a soname. This can happen when manually compiling a shared library, like through gcc -shared -o hello.so -fPIC hello.c. I'm not sure if that's what happened in this case. However, not having an soname is an issue when importing a shared library.
For whatever reason, when importing a shared library with no soname, we have to set the IMPORTED_NO_SONAME property in CMake. Otherwise, what happens is the rpath is set correctly (the path that the linker looks for the shared library at), but the name for the shared library in the ELF is put as a relative path to the file (instead of just the soname). This leads to the error when an executable (or another library) depending on the shared library is moved to a different directory.
Here's an example of the outstation library compiled without IMPORTED_NO_SONAME:
Notice the shared library's name is a path, which will clearly cause issues when the linker is looking for it.
Here's what it looks like with IMPORTED_NO_SONAME set:
Notice the shared library name is now correct, and the shared library will work.
Outside of that issue, development went fairly smoothly, and I was able to get the TCP server and outstation bindings working. I won't go into too much detail this notebook entry because it's very possible I make some changes, and I'm still figuring out how the core functionality actually works.
- Finish midterm paper
I completed my part of the midterm paper on DNP3 integration. I included information on the architecture of the code, not on the actual substation architecture we will be emulating. Our Sunday meeting was a very quick discussion about the midterm presentation for next week.
- Structure example repo
- Get the DNP3 example running
- Get the midterm paper started
I made some progress on structuring the example repo (just a very basic split between Master, TCP Server, and Outstation). I also added some documentation for getting the existing C example working. However, due to a large project in another class, I didn't have time to work on the Python bindings.
The Thursday meeting was online due to the hurricane. We were able to ask some good clarifying questions about the project. For me, the biggest takeaway is that we should be establishing a baseline simulation for a substation so we can test cybersecurity in the future. Knowing this, we are focusing solely on DNP3, since we also got some feedback that IEC 61850 would be difficult to implement.
On the Sunday meeting, we started on the midterm paper. There were only 5 people who showed up, so we spent most of the time assigning other people certain sections of the paper to work on. Adam and I will be working on the research section for DNP3 integration. We also decided to meet on Wednesday night in case we have any last minute changes we want to make to the paper. Additionally, Adam and I will be meeting on Monday to discuss out part of the paper.
- Look at DNP3 libraries; get an example running
- Get tasking for everyone in the networking team
Because of this, I started looking at DNP3 implementations. I noticed the current DNP3 example in Adam's branch uses pydnp3, which provides Python bindings for opendnp3. However, opendnp3 reached its end-of-life in 2022, with the Python bindings being even worse (no commits for the past 5 years).
The developers of opendnp3 moved on to a new Rust based implementation, succinctly called dnp3. From here on, I will refer to the protocol in uppercase (DNP3) and the library in lowercase (dnp3). For this week, I tasked myself with looking into this repo.
I found there are bindings for C, C++, C#, and Java, and support for TCP, TLS, and serial channels for the transport layer. The documentation is available here, and the examples for the C/C++ bindings are here.
To use the example, first download the example directory (whole repo not required).
# Navigate to where you downloaded the file
unzip 'stepfunc dnp3 main ffi-bindings_c.zip' -d dnp3_example
rm 'stepfunc dnp3 main ffi-bindings_c.zip'
cd dnp3_example
mkdir build
cd build
# Exporting compile commands allows for proper LSP integration
cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=1 ..
cmake --build .
# At this point, open another terminal in the same directory
# In the first, run this:
./outstation_example tcp
# In the second, run this:
./master_example tcp
# In the terminal running the outstation, try entering 'co' (which stands for counter)
# In the terminal running the master, you should see output that looks like 'Counter 7: Value=1 Flags=0x01 Time=0'
# Enter 'co' again and the value will increment as expectedThere are also options for serial, tls-self-signed, and tls-ca, but those require some extra config.
If we are going to integrate this into cps-cosimulation-env, it would need to interface with Python, so I also looked into calling C from Python using ctypes. It doesn't seem too bad, but it will be a while before I can get a working example since there is a lot of API considerations. I'm thinking it would be easiest to write most of the logic in C/C++, then have a few bindings for the devices to call from Python.
To be a bit more clear on how the binding will work:
- The dnp3 repo contains a CMake file that automatically brings in the shared library needed to call dnp3 functionality from C/C++, located here
- We will write C code that uses this shared library to implement DNP3 initialization and handlers
- Then, we compile the C code as a shared library
- That shared library will be what we are using with ctypes in Python
At the in-class meeting, I showed my findings to the group and explained what my plans are. Everyone seemed satisfied that this was a good path forward, so I'll be continuing by finalizing an API. We also decided to create a test project instead of working out of cps-cosimulation-env since the device modeling work isn't ready for networking yet.
On the Sunday meeting, we got some tasking for everyone on the networking team other than Kyle who didn't show up. Adam and I will be tackling the technical details, while Maria, Chungman, and Alex will be handling design decisions. Adam and I will be prototyping on this repo.
- Get simple_gridlabd_example running
- Familiarize with existing networking work
- Get tasking
I followed the main instructions on the GitHub page to setup Nix and build the project. Everything went smoothly following the instructions, and I was able to get simple_gridlabd_example working, along with Neovim LSP through source .venv/bin/activate.fish
I found some issues with the example, but since it's an example, I'm guessing it's not too big of an issue. For example, destroy_federate references broker, a variable that doesn't exist in it's scope.
def destroy_federate(fed):
h.helicsFederateDisconnect(fed)
# status, state = h.helicsFederateGetState(fed)
# assert state == 3
while h.helicsBrokerIsConnected(broker):
time.sleep(1)
h.helicsFederateFree(fed)
h.helicsCloseLibrary()
So, I decided to look at what the actual simulation in device_modeling/simulations/run_simulation.py, and indeed it doesn't have this issue.
def destroy_federate(fed):
h.helicsFederateDisconnect(fed)
h.helicsFederateFree(fed)
h.helicsCloseLibrary()
However, the rest of this file seems unfinished, so I wasn't able to run this simulation.
I also took a look Adam's notebook/branch to get an idea of what the we (the networking sub-subteam) will be working on. To get some more context, I also looked into the difference between Modbus and DNP3.
During our Thursday meeting, we got Trello set up for assigning tasks, and created two initial task. I took the task for looking at a DNP3 library that I identified, while the other task was for researching how substation networking usually looks like.
On our Sunday meeting, we didn't have many people, so we decided it would be better to split the meetings between the teams in the device modeling subteam. We also tentatively decided to focus on DNP3 for now, but we plan on getting some more direction from the instructors during class. Our reasoning is that Modbus is older, has less features, is less secure, feels less in the spirit of a research project (older technology), and the person who was working on it has graduated.
This week and the weeks before were mostly logistics (learning about the VIP, joining subteams, getting a meeting schedule for the subteams, kickoff presentation, etc.). I joined the device modeling subteam and specifically the networking sub-subteam.