Thoughts on integrating MODBUS/TCP

[Miq1]

It would be handy to have both MODBUS/RTU and MODBUS/TCP available in a single library. It looks like there could be a lot of code usable on both sides, so I started a closer inspection.

Packet formats

MODBUS/RTU and MODBUS/TCP are sharing the same payload, consisting of slave ID, function code and additional data. The difference are the TCP header for the /TCP variant and the trailing CRC checksum for /RTU:

TCP:  [sender ID][0x0000][length] [slave ID][function code][data]
RTU:                              [slave ID][function code][data] [CRC]

If TCP head and CRC tail were held separately, all functions dealing with the payload may be left as are (but see below).

Functions dealing with setup or manipulation of ModbusMessages will need to know if it is a /TCP or /RTU request or response to add the required header or trailer.

Adressing

While for /RTU the slave ID is sufficient to address a certain slave on the MODBUS, /TCP needs an IP address or hostname, plus port number in addition. In fact the slave ID is necessary nevertheless, as any IP/Port combination may contain a complete MODBUS again with different slaves.

The library would need a way to give IP or hostname plus port additionally to the existing parameters.

Classes

The existing esp32ModbusRTU class takes a HardwareSerial pointer as mandatory constructor parameter. A new esp32ModbusTCP class will have to take a Client parameter instead - Client being the base class for both WifiClient and EthernetClient.

The esp32ModbusRTU class should have the CRC separated from the _buffer to allow functions to be /RTU-/TCP-agnostic. The espModbusTCP class will have a TCPheader member struct instead.

Class methods

The handleConnection, send and receive functions have to be separate for both classes obviously. The checks for valid packets, set up of header or CRC trailer etc. will be different, too. A solution could be to have virtual functions doHeader() and doTrailer() that have to be implemented in both variants differently, but can be called regardless from type-agnostic general functions.

The same applies to the various request calls, as those for the esp32ModbusTCP class will have to have IP and port in addition.

The onData, onError handlers and their tokenized variants can be identical, though, it could even be possible for an application to use the same handler for both variant callbacks.

Preliminary conclusion

It looks like it could be possible to merge the /TCP functionality into the /RTU lib.

1. Prerequisite would be a clean separation of functions and member variables into those interface-dependent and independent and a class scheme implementing this separation.
2. Before integrating any /TCP line of code, the restructured library has to be thoroughly tested for the /RTU functionality to not lose anything on the way ;)

Would you like to jump in here?
3. Finally the /TCP part can be brought in...

[bertmelis]

My thoughts: I'd keep message handling almost totally separated but the API more or less the same. So the class private stuff will be different, the public (apart from constructor) largely the same.
With RTU there is only one connection to multiple slaves, with TCP there is a one-to-one connection the the server.
Message contents however is largely the same. Some refactoring in the class hierarchy is needed offcourse.

In the end, it was also the goal to include the modbus servers but I don't need it so I only did very little work on it. (I do have a testing modbus TCP server for esp in some other repo)

[Miq1]

My thoughts: I'd keep message handling almost totally separated but the API more or less the same. So the class private stuff will be different, the public (apart from constructor) largely the same.

I think the request calls will have to be different due to the IP/hostname and port needed.

I considered having a separate call to set these, but then it would be difficult to check in the request calls if the ip/port are valid. I personally would like to have it in each request call as I need to address several MODBUS/TCP servers in one application. I would suggest to have handleConnection in the TCP variant like

open connection
if ok
  send request
  wait for response or until timeout
  close connection

that is, a single connection for each request. This adds some processing overhead in terms of time, but on the gains side we may talk to any number of servers and will need just one TCP client (with the WW5500 module I got 8, but will need some to do NTP, DNS, push service, serving port 502 etc., so I am a bit scroogy with client slots ;)).

[bertmelis]

Well, the spec recommends against opening/closing a connection on each request. In fact, I had problems in the past with making too many connections to one of my modbus devices. It just refused the connection after merely 5 connections. This also meant I could not connect with my laptop while the esp was making requests.

So if you ask me, I want one persistent connection for one server. It does need to close after some "silent time".

[Miq1]

That would require to maintain a pool of connections and a number of clients to use for the library, right? Then the constructor would need to know how many and which clients the object is allowed to maintain. And those clients will be blocked for the calling application, regardless if there are any requests using them. Uh-oh...

Another idea: what if we added a connected client to use to a /TCP request call and not a unused one to the constructor? Then the application using the lib would keep control over their clients. So the connection can be held, if the application decides it or closed immediately after the response returned. The library simply would not care, unless the connection was broken and an error response will be sent back.

ModbusTCPRequest03(Client *c, uint8_t slaveAddres,.....)
...
if(c->connected()) {
  send()
  wait until receive or timeout
  return response
} else {
  return connection error
}
....

[bertmelis]

But I really don't see the benefit. Maybe we're talking about different things. I have the AsyncTCP lib in mind. You cannot just make a connection and wait. You have to manage all the callbacks. In this case it is way more simple to have the lib manage the connection. I would still make a public connect and disconnect method.

When the Arduino WiFiClient lib is used, it's a whole different story indeed.

Especially when a Modbus Server is implemented I'm inclined to use the async lib. This can handle more connections at once and it becomes imho easier to work with.

[Miq1]

I never dived deep into the AsyncTCP stuff, but I am happily using it for a ESP32-based web server 😄

I will have a closer look at it.

[Miq1]

Hmmm... Looks good from the AsyncClient level up, but how do you connect an EthernetClient on the w5500 module to it? I did not see an example for that yet.

[bertmelis]

Ah yes, that's not supported.

me-no-dev/AsyncTCP#10

[Miq1]

Then I am out, I am afraid. I cannot replace the w5500 by a PHY any more, as the device is completed with PCB and whatnot. 😞
My last hope is that the people all are still referring to the Arduino Core<->Ethernet.h incompatibility, that no one tackled officially so far. I happen to have a fixed inofficial version that is working. But if the AsyncTCP lib is relying on some deeper levels of the ESP stack I am at a loss.

[bertmelis]

AsyncTCP is relying on LwIP indeed.

Now for the client, Async is not a hard requirement for me but for a server it has the advantage that it is faster. The lib still has to be non blocking but that can easily be achieved with tasks.

So we need to think hard about how to do things.

[Miq1]

Yes, indeed. I am doing a little EthernetClient management in my application already, as it is a sparse resource, but mostly I am reserving slots for always-on services.A hanging client will still cost me a slot.If only I could do several connections in one slot...

[Miq1]

Another idea: how about two /TCP variants? One will have a Client only in the constructor and do the connect, send, receive and disconnect routine for every single request. The other will have a Client as well, but in addition a host/port combination, that is fixed for the instance. This variant will connect to the host initially and hold open the connection until told else or destructed.

For the W5500 module this means a limit of up to eight fixed /TCP connections only, but I reckon that will be enough for most applications. My application will use the multi-connection type and spend just one client on it.

It will require to have a triple set of request functions (one set each for /RTU, /TCP without host/IP and port for single connection, /TCP with host/port for the multi-connection type), but as these are on the lighter side, it will not be a huge overhead.

[bertmelis]

I suppose that methods that aren't used in the user application will not be linked into the binary. So from and end user point there's little difference.

One thing to consider: you don't have to wait for a response before sending a new request. So you can send 2 or more request (in separate TCP packets) while the server is still processing them. You don't want a previous request closing the connection while waiting for the response.

Any ideas?

[Miq1]

Right, but...
We have been living perfectly up to date with the RTU request handling being blocking, so why should we handle TCP differently? The multi-host variant at least will just do so. If you would like to have the single-host variant being able to do multiple requests simultaneously, you will have to add a whole new level of complexity. You will need to use the transaction ID in the TCP header to discriminate requests, as two requests may have the identical payload and so on.
Do you see a real need to have that? The most negative impact of not having it would be more timeouts on the requesters' sides in times of high demand and slow servers, since the individual answering times (and timeouts!) will add up until a certain request up the queue is serviced.
l myself see my device being used in smart home monitoring applications, where no millisecond data tracking is required. Targeting for industrial applications would impose a completely different demand, but I am doubting someone would use a library like this for it anyway.

[bertmelis]

Because I wanted to have responsiveness for my wireless devices. For the RTU version I only needed half-duplex and here you have to rely on the request/response routine to finish that's why this part is offloaded to a separate task. The API is non blocking.

Now for TCP, the message ID is specifically intended for that purpose. It's not that hard. You place all requests in a queue (after sending, which is done immediately). When a response comes in, you validate and match the ID with the ID from the requests in the queue. Simple loop over the queue as it is most likely the first item. While waiting for the responses you also have to check for timeouts, but only the first element in the queue as this is the oldest. AsyncTCP gives you a handy onPoll callback which is called every .5s (I believe).

Anyways, when not using AsyncTCP, the inner workings will be different because the API to handle TCP is different. I never looked into WiFiClient.

[Miq1]

Sounds like you would really be better off with the AsyncTCP solution. I am restricted to the 8 sockets the W5500 is providing and do not have that degree of freedom.
May be we should split here, me implementing the more restricted approach I scribbled above and you will be fitting in AsyncTCP?

[bertmelis]

Maybe first things first. what are the common elements in all the options (RTU, TCPvA and TCPvB)?
Can they be designed by a base class and derived classes with specific implementations? (Maybe even without virtual functions?)

And yes, maybe first try with the Arduino WiFiClient. It's the most straightforward.

By the way, should I move this repo to an "organization" or similar? I have this feeling you'll be contributing a lot and I don't want to let it go to waste in the event I'm running completely out of time.

[Miq1]

Maybe first things first. what are the common elements in all the options (RTU, TCPvA and TCPvB)?
Can they be designed by a base class and derived classes with specific implementations? (Maybe even without virtual functions?)

That is what I intended to do next. I already have collected some puzzle pieces and may be able to write the stuff down these days so we get something substantial to discuss. Do you normally use some dedicated notation or tool to design classes etc.? I am oldschool in that respect, the world ended for me at UML and I am feeling most comfortable in vi 😉

And yes, maybe first try with the Arduino WiFiClient. It's the most straightforward.

This is again where my troubles will begin... I got just one device yet, that is on site down in the cellar where no reliable WiFi exists. This is why I originally chose to use Ethernet instead.
I would have to patch together a second on a breadboard. I think I have all the parts in stock, but that would take some time.

By the way, should I move this repo to an "organization" or similar? I have this feeling you'll be contributing a lot and I don't want to let it go to waste in the event I'm running completely out of time.

Nah, no worries! I am doing this completely for myself at the moment (although I got some inquiries for the device). If what I do can benefit someone else: fine, but that is not my original intention. I am retired and have all the time I need - opposite to you who seems to be busy working... 😁

[bertmelis]

Sorry, I meant a regular TCP client instead of WiFiClient.

And lucky me, I'm not working at the moment. I am very busy raising the kids though. I unfortunately don't have any real modbus devices for the time being. So I have to rely on a modbus simulator (the Python implementation) on my laptop.

[Miq1]

Sorry, I meant a regular TCP client instead of WiFiClient.

Much better as seen from my side 👍

And lucky me, I'm not working at the moment. I am very busy raising the kids though. I unfortunately don't have any real modbus devices for the time being. So I have to rely on a modbus simulator (the Python implementation) on my laptop.

I would not choose the term "not working" for raising kids - kudos!

I got one TCP and three RTU devices at the moment - a PV system on TCP, a power meter talking MODBUS/RTU natively, a second power meter with a D0 optical interface and a MODBUS/RTU server I built myself, plus a water meter that currently is lacking a working sensor, so only all-0 data available from there. The bridge device is supposed to poll data from all the devices and push the collected data to a home server to store it in a DB and generate displays on it. The PV data is missing yet - that is why I want to bring in the TCP support. The bridge acts as a TCP server as well and will return the devices' data on MODBUS requests. With TCP support integrated, a bridge may connect to another bridge to further aggregate data for multiple sites. This is why my PV provider is loosely interested in the development, as they could keep track of larger, distributed installations with such a device.

[Miq1]

I grabbed the Violet UML editor to scribble a bit:

Outdated - see below!

So we have the Request/Response classes with the requests splitting up in the serial and TCP variants, due to the different data used there.
On the other side we have three different interface aproaches: serial and TCP for held connection or multiple connections.

The diagram is missing a lot still, but may be we can start with it?

[bertmelis]

OMG, you do realize I'm not a professional programmer do you? I'm actually an engineer (electromechanics). The things I know I learned by myself.

I'll give it a thorough look.

[Miq1]

Well I am a mathematician by profession, so I have a hang on weird things like programming. 🤣

UML has the advantage of being around for quite a while, so there are lots of tools, tutorials and explanations. I did never use the full depth it can provide, but just class diagrams and use cases, and sparsely sequence diagrams for complicated interactions.

Take your time, anyway!

[Miq1]

Another turn. I refined things a bit and added the AsyncTCP variant. Even if we do not implement it right away, it will be good to make it fit from the beginning.

Some explanations below the graphics...

Modbus.class.violet.html.zip

ModbusMessage

This is the base class for requests and responses, as was in your original lib. I put in all data and some methods that are common to any Modbus packet, regardless of transport etc. As all Modbus messages have a slave ID and function code, these are stored separately for easy access (in logic, you may decide to use pointers into data internally of course).

I defined three functions to add 8-, 16- and 32-bit values to data, the way you had add() and hi()/low(). They will add values in the correct sequence for Modbus packets

There will be no bare ModbusMessage, though, but only instances of derived classes. Should we define ModbusMessage virtual?

ModbusRequest

Derived from ModbusMessage, adds the token and an individual timout time to a ModbusMessage to form a request - still independent of transport protocol.

RS485Request

This adds the RTU 16bit CRC to the ModbusRequest it is derived from. This now is a class that will be instantiated for a request if the ModbusRTU object is used. I would like to keep the CRC separate from the packet data and only add it in on a send().

TCPRequest

We will have three TCP object variants (single, multi and async) that share the same request objects. Common to all is the TCPhead struct in front of the request data. I decided to add in the targetHost and targetPort member variables, too, although the will be redundant for the TCPsingle variant. But hence I avoided two more subclasses to be defined.
This is the class instantiated for TCP requests.

ModbusResponse

A ModbusResponse object has in addition to the ModbusMessage member variables two more to hold a reference to the triggering request and a potential error code returned.
We may add another method to generate a valid Modbus response packet for error codes that we define internally (I did so for the new receive function in the lib). Just to set the error variable may force special work arounds in the interface objects.

RS485Response

This adds the CRC, as did the RS485Request class, for the RTU protocol. The response data packet shall be without the CRC.

TCPResponse

We will get a TCPhead struct with every TCP-based response, so for validity checks, length detection etc. we probably will need to keep it.

PhysicalInterface

This is the (virtual?) base class for all transport protocol variants. It has provisions for the separate worker task, the callback handles for the responses and error responses, central timeout and a count of messages processed (we may need that f.i. for the transaction ID in the TCP header).
The callbacks will get the Modbus data, sans the TCPhead or CRC parts. This way the callback will be unaffected from the protocol used. We can not prevent the requests are partly different, but see below.
I still have the queueSize parameter here, but it may be misplaced - see below for the TCPsingle and TCPasync variants.

ModbusRTU

This is basically what we have in your library so far. It takes a Serial pointer in the constructor and optionally the DE/RE opin, plus a queue size, if it should be different from the default.
The object creates and maintains the queue of requests, and implements the handleRequest() virtual function from the base class. This will include the send(), receive() etc. functions needed to communicate over Serial.
It also has a method to calculate the RTU CRC.
The requests are as we know it, parameters are slave ID, function code and request data for the respective function code.

ModbusTCPsingle

This is the class for connections to a single target host/port, that is kept open during the lifetime of the object. You said you thought of firing requests as they come and sort out the responses upon arrival. This would suggest another data structure for the requests than the queue - a linked list or round robin or such.
The handleRequest() here must open (and keep open) the connection to the target host given in the constructor.

ModbusTCPmulti

My favourite TCP variant 😉 Here each request is treated in handleRequest() like those in ModbusRTU - put into a queue and served one by one, each opening the addressed target host/port connection, sending the request, waiting for and processing the response and finally closing the connection again.
The request calls each need to have the IP/port parameters in addition, though, so the calls are different from those in the two variants before.

ModbusTCPasync

This is unsafe grounds for me - I imagine this will be like a mixture of TCPmulti (in that it will address multiple hosts) and TCPsingle (maintaining open connections to the hosts). This will require additional management structures to hold connections independently of requests etc. Left as an exercise for you 😆

[Miq1]

I unfortunately don't have any real modbus devices for the time being. So I have to rely on a modbus simulator (the Python implementation) on my laptop.

I can easily modify my slave sketch to be a test device. You will need an Arduino Nano, a level shifter for Rx/Tx and a few wires to have it run connected to Serial, or a RS485 module and a 120Ohms resistor in addition, if you want to have a real bus. Only function codes 0x03 and 0x04 implemented ATM. Interested?

[bertmelis]

Don't have a nano. I also only test using direct communication, thing the DE pin to a led. The firmware doesn't know.

I'm in the process of salvaging an old laptop to do the development on. My main laptop is being used by the kids...

[Miq1]

Okay, if you get along with your setup - fine for me. I got some spare Nanos, so drop me a note in case you need any.

I am about to patch together a second bridge device. The first is installed in the cellar and in "production", so I will try to keep it unharmed 😁

[Miq1]

I opened an organization ESP32ModbusMasterUnified and invited you to join. Having thought about it, I like it better to discuss things there and not in public.

I will put the files and description etc. there in "TheBase"