Roadmap

[bilderbuchi]

Benedikt's proposed roadmap:
Define the two protocols: data (PUB/SUB) and control (REQ/REP, exact coordination remains to be determined)
Implement communication methods/classes for the different endpoints (+broker and proxy). These are the building bricks for other people tapping into pymeasure and our own programs
Write working code to connect an instrument to the data publisher and control receiver
Write working code to connect our data sinks (csv results etc.) to data receiver and control receiver
Write (maybe) a simple controller with control emitter, which can send simple commands and show data

[BenediktBurger]

A more extensive / structured roadmap (you might adjust the issue):

• Definition of the goals / scopes, see Goals/scope of the project #6
  • Definition of the vocabulary Glossary #9
  • Definition of the necessary reliability/robustness Reliability #4
  • Analyze existing, public solutions Already existing "big" solutions/frameworks #2
  • Analyze existing software used in individual labs Previous attempts #5
  • Definition of the documentation style Sphinx/ReST or markdown? #1
• Definition of the protocol / protocols (in form of a documentation file)
  • Broadcasting data protocol Data protocol #3
  • Control protocol Control protocol #10
    • Transport layer Control protocol transport layer. #38
    • Message layer Message format #20
• Reference implementation in Python of the protocol
• (Maybe) Ready to use code snippets using the protocol (director, actor etc.)
  • For example the starter Starter #11

[BenediktBurger]

What do you think about doing the reference implementation while we define the protocol? That way we can refine the protocol according to the lessens learnt in the implementation.
For example, once we have defined the data protocol transport layer, we implement that, and I test it in the Lab, while we talk about the Message layer.

I tried to make my current implementation modular, such that I can exchange indidvidual parts according to our ongoing discussion and development without need to change all the programs. So we can get real live feedback quite fast.

[bklebel]

I think it would be very sensible to work on the reference implementation in parallel to the protocol definition, if we only look at the protocol, think something up and then realize that it does not work, that would be just a waste of time. The zmqguide (I read parts of it when I worked on my implementation, but I did not go through it systematically) also advises to build systems piece by piece before designing a very grand scheme of things, and although I did work with zmq already, programming has comes with the almost unique possibility to try out everything immediately, and it would be a waste not to use it.
It might feel like it was slowing us down, to work on both things at the same time, but I think overall we will be quicker!

[BenediktBurger]

Thank you for these ideas

[bilderbuchi]

tl;dr: Yeah why not, but not yet!

I'm a little concerned about "churn" (ideas we pick up and later discard because they don't fit another part of the protocol), if we want the reference implementation to closely follow the specification work that will be a lot of effort and will slow us down.

I feel like we should first get a rough version of all "building blocks" of ECP in place before we start the reference implementation (i.e. "breadth first"). This way we'll have a rough understanding of the shape of things, which will also inform the implementation.
There will be enough work after that, hashing out the details, and getting rid of inconsistencies/design mismatches, where developing the reference implementation can indeed help tremendously.

[BenediktBurger]

We want to separate the transport layer from the message layer (content).
Therefore, we can start the transport layer implementation, once the transport layer is well defined.
In the meanwhile, we can talk about the message layer (which will be more discussions as there are more valid options, I guess).

[bilderbuchi]

Well yeah, but e.g. can the transport layer be well defined before we have an idea what all the Components need (e.g. the Director, Observer) -- cf the current discussion about Coordinator-Coordinator and Coordinator-Actor interaction.

[BenediktBurger]

The transport layer makes sure, that the message finds its way to the target and is (except for Components tasked with routing, that is Coordinators) independent of the Component type.

I see just two different parts: Coordinator - Component and Coordinator - Coordinator.

[bilderbuchi]

I just found this relevant gem - Fundamental truths of networking:

(12) In protocol design, perfection has been reached not when there
is nothing left to add, but when there is nothing left to take
away.

Of course, this is a riff on the popular quote by Antoine de Saint-Exupery.