Add zmq function call to Gmoccapy

[c-morley]

I am working on a qt version of nativecam.
nativecam wants to reload it's program on each change.
This is currently awkward if the calling program is not part of the original process.

In Qtvcp we have the capability to use zmq to subscribe to external programs that then can call the screen's functions.
This pull request adds this capability to gladevcp and gmoccapy.

In Gmoccapy this could be expanded to allow any authorized function to be called. For instance: axes selection or jogging increment. I think if we could add zmq to halui, then this could be a big jump in user usability.

I would suggest an addition to allow the user to select/deselect zmq in the options page.
This pull request is mostly for brain storming - some additional cleanup and docs are required.

[petterreinholdtsen]

How many message busses do we want to implement? I wrote src/hal/user_comps/mqtt-publisher.py to support some things, but did not integrate it into the frontends, while the zmq stuff seem to be more integrated. I guess NML might be seen as a message bus too. Are those the only message busses implemented in LinuxCNC? How many do we need and want.

…

-- Happy hacking Petter Reinholdtsen

[c-morley]

Zmq was already used in Qtvcp for quite some time.
It's well documented with examples and has C, C++ and python bindings.
I can't say which is better Zmq or Mqtt
NML is notoriously hard to use - everyone avoids it like the plague.

[c-morley]

@hansu you have any opinion?

[hansu]

I am working on a qt version of nativecam.

Nice to hear that!

I have no knowledge about mqtt or zmq and what are the strengths of each.

But in general it sounds good to have more capabilities to communicate with the GUI.

For example is that also possible to pass a signal like 'file-reloaded' to an embedded or external (glade) panel?

[c-morley]

As coded the glade panel (or any program that uses zmq) can ask Gmoccapy to reload a fike.
For Gmoccapy to ask is a little bit more code but the basics are there.
There would be decisions to be made about a way to decode who reacts to the message (topics in zmq speak)
I am not an expert either.

[andypugh]

Is this likely to move out of draft status?

[c-morley]

i was hoping for more discussion about the best way to integrate the idea. So whether it comes out of draft or causes another pull request i don't know. I won't let it hang forever if that is what you are asking.

[zz912]

If I understand correctly, zmq is another way of communication between LCNC modules.

Currently we can communicate between modules using NML and HAL. The Python Interface is the defacto communication through NML. Does zmq belong to these communications? Did I understand correctly?

I understand that zmq is already integrated in LCNC. If that's the case, I think it could be used.

BUT!!! It should be documented in the LCNC documentation. Ideally some examples with usage. As the Python Interface is given. If someone wants to learn how to use LCNC, they must be able to:
C
C++
Python
Bash
HALL
G-code
Qty
Glade
.......
There is quite a lot, so the documentation should be as good as possible so that it does not discourage other developers from joining. It is clear to me that the internal documentation for zmq will be more difficult than its implementation.

Will there be any rules when to use NML, zmq, HAL, header file with extern, glade qt signal?

https://linuxcnc.org/docs/devel/html/code/code-notes.html

[c-morley]

Zmq is used for communication between programs using sockets. it a well documented library with many bindings to other languages.
Qtvcp has some 'hooks' in it that can be turned on to use Zmq.
documented here:
http://linuxcnc.org/docs/stable/html/gui/qtvcp-code-snippets.html#_external_control_with_zmq

NML HAL and Zmq are all appropriate for different things, with strength and weakness on each library.

[zz912]

Here is documentation too:
http://linuxcnc.org/docs/stable/html/gui/panelui.html#_zmq_messages

One stuff in many places.

Shouldn't it have a main chapter here?

Explaining for example what is the difference between ZMQ and NML?
Is ZMQ only for GUI?
Is ZMQ for GUI and realtime component?
Can ZMQ replace NML?
Does it make sense to use ZMQ instead of a header file?

The chapter on NML is an example of how it should not look like:
http://linuxcnc.org/docs/devel/html/code/nml-messages.html
There is information only for person, who knows everything about NML.
But at least it's in the right place.

[rmu75]

(off-topic, in case somebody is interested in exploring zmq as replacement for NML: I have a POC running that is using zmq/flatbuffers instead of NML. https://github.com/rmu75/linuxcnc/tree/rs/zmq-experiments / https://github.com/rmu75/cockpit/tree/rs/zmq)

[Sigma1912]

So ZMQ seems to be a promising path forward.

[hydroid7]

I really appreciate the work done by @rmu75. IMHO Flatbuffers combined with message passing it is the future of LinuxCNC.

However, I'd propose the use of Posix message queues. They are easy to use, built in into Linux, have bindings to literally all languages and are well supported.
In my measurements (which I'm not allowed to publish, but here is a simple comparison) ZMQ has about 5x greater latency than plain PosixMQs.

PosixMQs have an absolute maximum message size of about 16MB, but I think that should not be a limiting factor.

[c-morley]

Id there a preferred python library for posix message queues?

[petterreinholdtsen]

[c-morley]

Id there a preferred python library for posix message queues?

A quick search pointed me to <URL: https://pypi.org/project/posix-ipc/ > and <URL: https://pypi.org/project/ipcqueue/ >. The latter also support SysV message queues. Neither of these are in Debian already, as far as I can tell. The closest relative I could find is <URL: https://tracker.debian.org/pkg/python-sysv-ipc >, which provide SysV message queue support. Its home page state "POSIX IPC is a little easier to use than SysV IPC, but not all operating systems support it completely." I could not find a page on <URL: https://openhub.net/ > to check how alive the project is. It is unclear to me how POSIX message queues are better than SysV message queues.

…

-- Happy hacking Petter Reinholdtsen

[andypugh]

I am not sure what the use case is here, though do recall that Machinekit had an ambition to replace NML with ZMQ.

Is intersystem interopability a desirable feature of the system being studied? If so then being tied into POSIX might not be ideal.

[rmu75]

I'm somewhat guilty of derailing this issue, there is probably a more appropriate place or issue to discuss that. I will probably open a draft PR when my code is somewhat more mature and discussion can be continued there. Nonetheless:

@hydroid7 that linked comparison somewhat compares apples and bananas. 0MQ can also use IPC via unix domain sockets (no TCP/UDP or even IP involved), and I doubt that those have to be inherently slower or have higher latency than posix message queues. But I didn't (yet) measure anything. Communications overhead is not that critical, NML is used for GUI stuff, and that doesn't need to be more "current" than what your screen refresh rate allows. Even then more than 30Hz update rate won't gain you much.

IMO the messaging mechanism that replaces NML has to be network transparent. To gain anything over NML, it has to be a well-established and maintained solution that is independent of linuxCNC and in wide enough use so that risk of becoming abandoned is low. Plus points would be cross-platform abilities and availability for a large number or platforms and programming languages, especially javascript/web stuff. Best would be a well-established RPC framework with low overhead, low latency and low complexity (i.e. not CORBA nor SOAP nor derivatives).

That IMO rules out posix message queues and sysv IPC without some interposer that can also talk UDP or TCP like 0MQ.

[c-morley]

This discussion of replacing NML while interesting and important for linuxcnc, it is not really related to the PR.
This PR is for adding a message system to send arbitrary messages between UI programs, something NML does not do.
If linuxcnc had a concept of 'current jog rate/increment/axis' and sent a message for GUIs to update their graphics, then we could avoid the current use case for this ZMQ pr, but I bet in the future there will be more uses.

The nice thing about ZMQ is there are python and C C++ libraries available on the distribution we support.