Refactor, Protocol Enhancements, and Device Ecosystem Reorganization #3

MicBoucinha · 2025-07-18T10:03:29Z

Important

We started working on this PR by merging the contributions made in PR#1

Refactoring & Project Structure

Refactored enums, class members, and function naming for consistency with official Harp naming conventions
Changed namespace to harp and sub-namespace reorganization:
- harp.communication for communication-related modules
- harp.protocol for core protocol definitions
- harp.devices for device-specific implementations (using PEP420 for implicit namespace packages)

Features & Enhancements

Updated Harp protocol support with missing features
- Added support for device serial number and missing registers
- Improved message handling (arrays, timestamps, error detection) for different payloads
Updated currently known Harp devices names
Added context manager support to Device
Added static factory method HarpMessage.create
Changed dump file convention to be at the Device level instead of message level
Removed ReadXXXHarpMessage and WriteXXXHarpMessage classes and ReadXXX and WriteXXX static methods from HarpMessage and moved them to Device
Removed existing behavior driver because of new harp.behavior namespace package

Documentation

Added MkDocs documentation for API, and project structure, and per-device documentation
Use numpy style convention for docstrings documentation
Added docstrings to all public methods and classes
Moved and updated existing examples to docs folder
Created GitHub Action to generate documentation (manually triggered)

Documentation here: https://fchampalimaud.github.io/pyharp

Testing

Updated some tests as needed

Technical

Convert project to uv

Contributors

Major contributions by Luís Teixeira and José Grilo

harp.devices

Automatically generated python package for each device and respective documentation
Added harp.devices namespace package for device-specific implementations

Repository here: https://github.com/fchampalimaud/harp.devices

As mentioned in issue #2, we were really interested in developing the Python Harp ecosystem further.

Our starting point was PR #1, which already had pushed pyharp forward (e.g., by implementing a way to deal with the device's events), and try to build on top of that. At first, we mainly focused on completing what was missing regarding the protocol (e.g., add a generic way to deal with the different payload types) and the interaction with a generic Harp device (e.g., add methods for the missing common registers).

Our goal was to have a simple enough but versatile pyharp interface and we based our rationale to be as similar as possible as the Bonsai.Harp implementation. One way to achieve that was to get inspiration from the CreateMessage Bonsai node to implement the HarpMessage.create static method. Additionally, we added methods for the Device class that allow the users to read/write from/to registers that have different payload types.

Another thing we tried to focus on was documentation. It surely can be improved but, at the moment, we documented every function and class of the project in order to facilitate the API usage and adapted/added some minimal examples that should help the user get started with the pyharp package.

Beyond being possible to connect and interact with a generic Harp device, we also wanted to generate device-specific interfaces. On the one hand, we would like that each Harp device had its own package, so that if, for example, a user just wants to use Harp Behavior, it doesn't need to import a package that contains the interfaces for every board. This would also help when there's individual updates on each device. On the other hand, we would like that every package (including pyharp "core") had the same feeling, in the sense the user would get the implicit idea that every package is part of the same ecosystem. In order to achieve that, we organized the project directories according to PEP420, in which the global ecosystem has the following structure:

harp/
├── communication/
├── data (harp-python content hopefully)
├── protocol/
└── devices/
    ├── behavior/
    ├── device_a/
    ├── ...
    └── device_z/

We also wanted to have a way to generate the device-specific packages automatically, so that we could easily add new devices and keep the documentation up-to-date. For that, we used reflex-generator to generate the device-specific packages based on the device's YAML. Currently, the generated packages are available in the harp.devices repository mentioned above, but we believe that the Python interface should reside in each device's repository. PR#86 to reflex-generator has been submitted to support the Python package generation.

Finally, we abandoned the pyharp name for the package because, unfortunately, someone registered a package in PyPI with the same name. The package name is currently harp-protocol. At the same time, and mostly to keep coherence with the namespace naming convention used on C#/Bonsai side, we decided to change the global namespace to harp.

Known limitations of the PEP420 usage

To allow the usage of PEP420, we cannot have members under harp or harp.devices that are not namespace packages. This means that we cannot have a __init__.py file under those directories. As a consequence, we cannot have any code directly in those paths, which means that we cannot have any imports there.

This also means that the harp-python package cannot currently be installed in the same venv until some changes happen there. Our suggestion at the time was to place it's contents under harp.data and then import it from there. This would allow the harp-python package to be installed in the same venv as harp-protocol and the individual harp devices. We also submitted a PR to the harp-python repository to reflect this change in PR#52.

With these new PRs, we aim to expand the current features of the Python implementation and make it more accessible for new users, helping them adopt and engage with the HARP ecosystem. We also see value in consolidating the Python implementation within the official harp-tech repository, and we’re more than happy to help maintain it. This would help avoid fragmented development efforts and lead to a more unified and robust solution for the community. We will continue iterating on these developments and commit new features to this PR while we work toward a consensus on merging it into the harp-tech repository.

…mode Dev/editable mode

Added ReadS32 and refactored WriteS32 to handle array inputs

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

…end method Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

…o account

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

CLAassistant · 2025-09-18T10:23:18Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ MicBoucinha
❌ ZegCricket
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

MicBoucinha · 2025-10-01T09:28:18Z

Hi! We made the following changes since the last meeting:

Separate the previous harp-protocol into 2 different packages (harp-protocol and harp-serial), using the uv workspaces feature
Implement a strategy to handle timeouts when waiting for a reply from the device (harp.serial.TimeoutStrategy)

Notes

harp-protocol & harp-serial are already released as well as new versions of harp.devices using harp-serial directly.
Documentation is online and updated.

@glopesdev @bruno-f-cruz, feel free to give us feedback where improvements can be done.

bruno-f-cruz

Thanks for taking a stab at this. I left some first pass feedback.
I think we should discuss a bit the architecture of the device API synchronously during one of the meetings. I dont have a specific issue with it, but I am just afraid it may become hard to maintain if each register has its own specific read/write method with a name given without any sort of consistency.

bruno-f-cruz · 2025-10-01T14:14:31Z

.github/workflows/build-docs.yml

+      - name: Clone harp.devices repository
+        uses: actions/checkout@v4
+        with:
+          repository: fchampalimaud/harp.devices


I am not sure the package repository should be coupled with the generation of docs for the devices.

In the long term, we could consider to separate the docs generation from the packages in different repos. The current solution is not set in stone, but we would like to keep the documentation for the generic device and for the specific devices together.

bruno-f-cruz · 2025-10-01T14:14:58Z

.gitignore

.vscode should be added to the list

The reason we keep the .vscode directory in the repo is because it has the extensions (and respective settings) we actively use for development and that we would like for contributors to use as well. Do you have an alternative suggestion regarding this matter?

bruno-f-cruz · 2025-10-01T14:19:42Z

LICENSE

This should be changed to harp-tech, similar to what happens with the other software packages

We can change it when the move of the repo to harp-tech is imminent.

bruno-f-cruz · 2025-10-01T14:20:13Z

10-harp.rules

Not sure what this config file is for? But should likely be ignored?

This file was inherited from the AIND fork, but we agree it can be removed.

bruno-f-cruz · 2025-10-01T15:41:12Z

LICENSE

@@ -1,6 +1,7 @@
 MIT License

 Copyright (c) 2020 OEPS & Filipe Carvalho


We should consider making this standard with the copyright in the other software packages.

https://github.com/harp-tech/harp-python/blob/3d97d0b800533e4821896f744e52569c6413c6f7/LICENSE#L3

Sure, same as answered previously,

We can change it when the move of the repo to harp-tech is imminent.

bruno-f-cruz · 2025-10-01T16:55:57Z

src/harp-serial/harp/serial/device.py

+        self,
+        serial_port: str,
+        dump_file_path: Optional[str] = None,
+        read_timeout_s: float = 1,


Simplify the name to: timeout

bruno-f-cruz · 2025-10-01T16:57:02Z

src/harp-serial/harp/serial/device.py

+            _TODO_
+        """


This is not up to date with the signature

Thank you! We will update it.

bruno-f-cruz · 2025-10-01T16:59:47Z

src/harp-serial/harp/serial/device.py

+        # open file if it is defined
+        if self._dump_file_path is not None:
+            self._dump_file = open(self._dump_file_path, "ab")


Not sure I follow how you would want to use this.
I can see cases where users may want to skip querying the device for metadata, but this suggests that a valid use-case is to cache it?

We have to look at this more carefully. We can discuss it further.

bruno-f-cruz · 2025-10-01T17:05:11Z

src/harp-serial/harp/serial/device.py

+        reply = self.send(HarpMessage.create(MessageType.READ, address, PayloadType.U8))
+        return OperationMode(reply.payload & OperationCtrl.OP_MODE)
+
+    def dump_registers(self) -> list:


I think we should discuss a bit this API synchronously during one of the meetings. I dont dislike this but it has the potential to grow a bit out of control.

This was only for this specific register where we don't know yet how to constrain the user's input regarding the OP_MODE to be of type OperationCtrl. In terms of usage, for this particular register, having the separated method made more sense. But we should discuss it further.

bruno-f-cruz · 2025-10-01T17:07:09Z

.pre-commit-config.yaml

@@ -0,0 +1,22 @@
+repos:


I would suggest turning this into an action instead of a pre-commit.

Why an action only?
I might be missing something, but personally, I prefer to have checks locally before committing when possible.
We can discuss this further.

Poofjunior and others added 30 commits April 13, 2022 17:02

can now pip instal in editable mode

be0c7a0

updating readme

ef9936e

fixing readme bug

69e11f0

adding udev rules

4fff7ba

device_names is now a default dict; examples now also work on linux

d974ee4

fixing pdf link; tweaking serial port to use ftdi rule

fbdacbd

Update README.md

33d8cc8

PEP 563 return type hints

0191b23

grand messages.py refactor

6a73a99

more message.py refactor

4108f6b

adding harp rules

e3457bb

adding simple behavior io stuff

455de81

adding prelim behavior driver

4e5e58f

adding a string representation

0e47767

added __str__ and active mode script

43629c0

printing DI event changes works

edfc361

updating how payload is parsed

d5d835a

adding enabling of events

3a122a3

adding inWaiting to reads

2fc5c58

read dumped registers

3f7dfba

expose more core features

401bd36

Merge pull request harp-tech#1 from AllenNeuralDynamics/dev/editable_…

ac57cb3

…mode Dev/editable mode

add WriteU32 and WriteS32

c2bba09

fix signed error in WriteS32

fb54476

add debug level logging.

e7900d4

Added ReadS32 and refactored WriteS32 to handle array inputs

1b1e833

Merge pull request harp-tech#2 from jessy-liao/master

325769c

Added ReadS32 and refactored WriteS32 to handle array inputs

Add threaded serial port

368e346

clean up formatting changes

2381e8f

add event_count method and update test

9e30545

MicBoucinha and others added 25 commits August 5, 2025 09:27

Rename communication namespace to 'serial'

39e3ca3

Remove unnecessary devices folder

4d9a3a4

Fix issue with Visual_en flag

efa5e04

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Fix documentation path for Device

d27c226

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Implement a Timeout handling strategy

77607ae

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Update payload handling taking into account the timeout strategy in s…

11c47df

…end method Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Reorganize project directory structure

a0be5e2

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Create workspaces for harp-protocol and harp-serial

87a8b14

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Update docs

9073f4c

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Chore: add pyrefly extension to extensions.json

8c07844

Chore: add script to run docs locally on Windows

ee5aa32

Add harp reference epoch constant

594399a

Fix issues with types

93aa50c

Docs: fix imports from examples to take the new project structure int…

2481ec1

…o account

Refactor: simplify WriteHarpMessage handling of PayloadType

ea50e70

Chore: add VSCode setting that enables pyrefly to display type errors

713c934

Removed custom message from exceptions

36ec9ad

Pin harp-protocol version on harp-serial

3272e10

Fix: raise HarpReadException for missing timestamp in ReplyHarpMessage

34de54d

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Add LaserDriverController to device_names

a107ebb

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Refactor payload handling in HarpMessage to properly support Python 3.9

ed7388a

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Add operation_ctrl methods to device

fbd7d64

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Update documentation

af7e79f

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Fix tests

5ea52bb

Co-authored-by: José Grilo <jose.grilo@research.fchampalimaud.org>

Merge branch 'feature/workspaces' into develop

25f0445

bruno-f-cruz requested review from bruno-f-cruz and glopesdev October 1, 2025 14:11

bruno-f-cruz requested changes Oct 1, 2025

View reviewed changes

		@@ -1,6 +1,7 @@
		MIT License

		Copyright (c) 2020 OEPS & Filipe Carvalho

Refactor, Protocol Enhancements, and Device Ecosystem Reorganization #3

Are you sure you want to change the base?

Refactor, Protocol Enhancements, and Device Ecosystem Reorganization #3

Conversation

MicBoucinha commented Jul 18, 2025

Refactoring & Project Structure

Features & Enhancements

Documentation

Testing

Technical

Contributors

harp.devices

Known limitations of the PEP420 usage

Uh oh!

CLAassistant commented Sep 18, 2025

Uh oh!

MicBoucinha commented Oct 1, 2025

Notes

Uh oh!

bruno-f-cruz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!