Multiple connections to the same plc and handling a pool of connections

[scamille]

This is more of a 2 part question, the first being about Sally7:

Is it possible to have multiple connections to the same PLC with Sally7, assuming of course that some limit on maximum number of connections imposed by the PLC is not reached?
Is it possible by just opening multiple connections, or would that require adding some form of "connection number" before/while establishing the connection? (I know that the DeltaLogic AGLINK has such a concept, but am not sure if that is just internally in their library or if it is related to the S7 protocol).

The second part / the reason why I am asking is:
A single PLC connection with S7NetPlus/Sally7 still needs to be synchronized, even when used with async/await. To truly increase the bandwidth of accessing data I would either have access multiple data items at the same time, or use multiple connections to the same plc concurrently.

The first option with reading/writing multiple data items unfortunately does not fit at all in the existing software I have, and only solves scenarios where you can read multiple date items in a correlated fashion. For performing multiple independent actions on the same PLC, multiple connections would still be preferable.

So to get to the point:
My idea would be to have a pool of available, established connections. A worker grabs a connection, perform a read/write, and gives the connection back into the pool. Some other worker ensured that connections are initially established / re-established.
Alternatively, the pool could contain virtual connection handles and the normal workers are also responsible for establishing the connection itself, besides using it for read/write.

What do you think about that idea? What .NET collections and other mechanisms would you recommend for getting something like that to work?

[mycroes]

Is it possible to have multiple connections to the same PLC with Sally7, assuming of course that some limit on maximum number of connections imposed by the PLC is not reached?
Is it possible by just opening multiple connections, or would that require adding some form of "connection number" before/while establishing the connection? (I know that the DeltaLogic AGLINK has such a concept, but am not sure if that is just internally in their library or if it is related to the S7 protocol).

It's possible, but you need to have the connections defined and loaded into the PLC. You can add additional S7 connections in the network topology in TIA portal between the PLC and a PC station. The IP address needs to match for the PLC to accept the connections using the supplied TSAP's. I might have a small instructional movie from one of our PLC developers somewhere, let me know if you can't figure this out.

The second part / the reason why I am asking is:
A single PLC connection with S7NetPlus/Sally7 still needs to be synchronized, even when used with async/await. To truly increase the bandwidth of accessing data I would either have access multiple data items at the same time, or use multiple connections to the same plc concurrently.

I did some lab testing where I would get something like a 3% improvment (IIRC) when using multiple connections, with a PLC nearby the PC. However I've recently been thinking about a multitude of changes. For one, if multiple requests would be used then there's less impact of higher network latency. At some point network latency might actually be even more than the actual request handling duration, so on slower networks that could have huge impact. The other thing is combining read actions, which results in larger requests and thus also benefits from concurrent requests.

The first option with reading/writing multiple data items unfortunately does not fit at all in the existing software I have, and only solves scenarios where you can read multiple date items in a correlated fashion. For performing multiple independent actions on the same PLC, multiple connections would still be preferable.

I don't follow, maybe you could provide a bit more detail or a concrete example. I do think that it doesn't matter much whether you're using multiple connections or concurrent requests; except for the fact that concurrent requests requires a single connection only so there's less configuration there.

So to get to the point:
My idea would be to have a pool of available, established connections. A worker grabs a connection, perform a read/write, and gives the connection back into the pool. Some other worker ensured that connections are initially established / re-established.
Alternatively, the pool could contain virtual connection handles and the normal workers are also responsible for establishing the connection itself, besides using it for read/write.

What do you think about that idea? What .NET collections and other mechanisms would you recommend for getting something like that to work?

Honestly, I'm not sure. I never did pooling in that way. I'm trying to figure out what synchronization primitives to apply to concurrent requests, it's something I intend to add sooner rather than later. When I add it I'll probably migrate it to S7NetPlus as well.

What you could do is push all the connections to a ConcurrentQueue or BlockingCollection, take a connection, perform some action, add it back to the queue. However, both only have a sync API, so I'm not entirely sure if this is the way to go because there's a fair change you go through all your connections faster than any of them has finished it's request.

Anyway, I intend to address this inside the libraries, it still has to materialize though....

[scamille]

Ok thanks for the information. I definitely need to figure out these Tsap objects anyway.
Why do multiple connection require extra setup in the TIA project, when I can connect with a single one just fine? But maybe that is already done on all the PLCs I have accessed, with other people having taken care of it. I usually try to avoid getting involved with the PLC side and TIA at all, life is too short :-)

Our existing software can maybe be summarized as followed:
IList<Command(int dbNr, int offset, int length, bool ReadOrWrite, ExecutionPolicy> commands;
ExecutionPolicy would either be to read periodically ever x seconds, or poll some "Handshake" Byte as often as possible.

In the past those commands have just been executed in sync one after another, assuming their ExecutionPolicy said that it was their time to do so. Now they are wrapped inside a Task.WhenAll, but are still mostly independent of each other.

Merging multiple of those commands into a single request would require some major rewrite of all this. Merging the polling of the "Handshake" Bytes of all those commands into a single request might in fact be a huge performance gain, but would also require thinking whether I need would reach the upper limit of number of data items too quickly (but I guess that could be solved by having multiple chunks of commands).

Merging periodic commands could have some benefit as well, but they might just have wildly different periods which sync up seldom.

Both things would increase complexity a lot. A pool of connections would be abstracted away completely in the lower level "s7 driver", with no significant architectural change at all.

What you could do is push all the connections to a ConcurrentQueue or BlockingCollection, take a connection, perform some action, add it back to the queue. However, both only have a sync API, so I'm not entirely sure if this is the way to go because there's a fair change you go through all your connections faster than any of them has finished it's request.

I would definitely expect to still be resource-starved, with a empty connection pool and commands waiting for others to release their connection. But it would still be better than just waiting for a single connection.

[scamille]

About the part with multiple connections:
We might have talked about different things here. I do not have plans of having connections with different slot/rack (and thus destination tsap).
The idea is to just have multiple TCP connections to the same PLC address (IP, cpu type, rak, slot).

This seemingly does work with the 3rd party library I use when I just set up 2 different PLC objects living in different threads and just run both at the same time. But I don't know what magic they do under the hood, whether it is truly a separate TCP connection or not. That's why I wanted to ask if you know more about this.

Edit: I should probably just test with an actual PLC once I have Sally7 set up, and see what happens.

[mycroes]

About the part with multiple connections:
We might have talked about different things here. I do not have plans of having connections with different slot/rack (and thus destination tsap).
The idea is to just have multiple TCP connections to the same PLC address (IP, cpu type, rak, slot).

We're talking about the same thing, but what you're suggesting is blocked by the PLC. I started out with Siemens with Simatic Net and a wrapper around the Sapi S7 library (which I wrote myself) to connect to Siemens PLC's. In order to use Simatic Net the PLC programmers needed to setup a PC station in TIA portal and configure connections between the PC station and the PLC's. Then a file can be exported which can be imported into Simatic Net, which exposes connections using yet another abstraction which then again can be resolved using Sapi S7.

At some point we went looking for other solutions, mostly because of the huge amount of issues we had with Simatic Net and because we knew other people could connect to our PLC's without actually needing Simatic Net. So I guided an intern into finding and benchmarking different libraries for S7 communication, with the clear winner among them being S7NetPlus. There were some minor issues with S7NetPlus, but nothing we couldn't fix. So I started with multiple write support, which on itself presented a huge performance improvement. At the same time I also started thinking about Sally7 (or perhaps I had even begun development at that point), because even though I could make some improvements we're constantly reading from PLC to detect changes so the architecture and API of S7NetPlus simply wasn't the best fit. I also discussed that with the maintainer at that time, but he wanted to keep S7NetPlus (mostly) backwards compatible, so that meant I was better off writing an alternative.

In the mean time I had figured that I could only make a single connection using S7NetPlus, while I could actually make multiple connections using Simatic Net. But I was reading the specs for COTP and ISO over TCP and noticed that they mentioned TSAP's and I have seen them mentioned somewhere else, I guess perhaps in S7NetPlus, but maybe in other libraries or even in Simatic Net logging. So I went looking for those in TIA portal and found those on all the S7 connections (and other connections as well I think).

What did amaze me that I was originally told by the Siemens engineer that you need to setup connections for S7 series PLC's, I've once read or heard that it has been the consequence of Stuxnet which was targeting Siemens PLC's, while libraries like S7NetPlus could just connect. And it's actually pretty simple: every PLC has at least a single connection which is freely accessible without even requiring a specific source IP. Apparently (by sample of S7NetPlus) the TSAP's for that connection can be calculated using rack and slot numbers. Of course it depends on GET/PUT permissions and non-optimized datablocks, but other than that it's no different from the configured S7 connections. However, you can only make single connection on that TSAP (at PLC side) at a time. So in order to have multiple connections, you still need to do some setup in TIA portal (or whatever tool, not sure if there's others as well). For the other connections it also holds that you can only have a single active connection using the configured TSAP's.

Basically this answers your question and concern: you can only connect using a set of TSAP's once, and there's only one freely available TSAP for each rack/slot. If you want multiple connections, you really need to configure them. We always configure the connections we use in the production software, so that we can use the freely connectable TSAP to connect with debugging utilities (like my new PlcMonitor).

This seemingly does work with the 3rd party library I use when I just set up 2 different PLC objects living in different threads and just run both at the same time. But I don't know what magic they do under the hood, whether it is truly a separate TCP connection or not. That's why I wanted to ask if you know more about this.

I don't know, but if you want to know you should just emulate a PLC and see what happens. I intend to add this functionality to Sally7, I'm just short on time for everything I want to do.

Edit: I should probably just test with an actual PLC once I have Sally7 set up, and see what happens.

Yes, but I'd be amazed if you find anything else than I explained above 😉

[mycroes]

@scamille I implemented concurrent requests the past few days. That does rely on the concurrency level the PLC permits (negotiated during connection init), but so far it's proven slightly (~3%) faster in a lab test and 50% faster at a customer site. Still need to clean up the code, possibly add some tests, but I expect to have it in Sally7 somewhere this week. Possibly will also migrate it to S7NetPlus (although that won't just be copy-paste, due to the different structure).

[scamille]

I did implement a multi connection wrapper, using a queue from https://github.com/StephenCleary/AsyncEx . It did seemingly work with 2 parallel Sally7 connections without any special setup, but I need more time to test and also measure things.

[scamille]

This is slightly off topic, but I do think it makes sense to just lock things down in S7NetPlus to reduce usage errors, especially since the target audience seems to need it.

For Sally7 I think most clients are capable of rolling their own synchronization. I don't know how much performance adding unused synchronization costs, but it might be a more sensible choice to offer it as an opt-in thread-safe wrapper instead of modifying the existing S7Connection.

[mycroes]

I did implement a multi connection wrapper, using a queue from https://github.com/StephenCleary/AsyncEx . It did seemingly work with 2 parallel Sally7 connections without any special setup, but I need more time to test and also measure things.

We configure our connections and variables, so we used a similar approach where we would just divide the data over different connections, instead of dynamically dividing it at runtime. That does work, I guess it offers performance benefits similar to concurrent requests, but I've only really tested it in lab conditions where concurrent requests also only delivered like a 3% improvement.

This is slightly off topic, but I do think it makes sense to just lock things down in S7NetPlus to reduce usage errors, especially since the target audience seems to need it.

Yes, but no matter what approach is taken it mostly makes sense to implement all of that async. The network communication already is async, waiting for other requests to complete in the meantime would only add to that. I guess for people having just a single source of reads (not necessarily single threaded though) they wouldn't really notice if they wait for the requests to complete unless they actually timeout.

For Sally7 I think most clients are capable of rolling their own synchronization. I don't know how much performance adding unused synchronization costs, but it might be a more sensible choice to offer it as an opt-in thread-safe wrapper instead of modifying the existing S7Connection.

Concurrent requests rely on negotiation during connection init. Although I guess the PLC will never send unsollicited messages (unless you're using cyclic reads, which is not even possible with the current generation of PLC's) it could possibly be an issue to permit a higher level of concurrency than the single request we currently allow. However, I might implement it in a way where a RequestQueue can be supplied to the connection constructor to allow overriding the default queue with something else (like an unsynchronized-sent-straight-away option for instance). OTOH my current implementation uses a System.IO.Pipelines.PipeReader to parse incoming TPKTs and MemoryPool for both request and response handling. I think I still have a few minor improvements to make, but the current effect is that there are no buffer allocations (they're borrowed from and returned to the pool) and that building the requests happens outside of the concurrency barriers (although currently that also requires me to keep the request memory lease until I get a response as well). All in all there's probably not much room for improvement and I'm guessing performance impact is negligible.

[scamille]

May I ask what your rough setup was when measuring the 3% increase?

My current setup makes it a bit hard to get reliable test results, but I definitely expect much larger gains to be had here. Even though we normally operate in a local network only, I believe the network delay should usually outweigh the response time on the PLC itself.

We also usually have large amount of single byte reads, where I would expect this to benefit even more.

[SmackyPappelroy]

I know this is highly unrelated to the topic at hand, but has anyone of you tried imolementing an OPC UA server?

[scamille]

We started using softing opc ua client (net standard version) with the integrated server in s7 1500 plcs, but we haven't started using it in production (project is put on hold).
There are performance problems with addressing all variables as nodes approach, so we probably need to switch to reading whole udts as byte[] (but that does not work on full data blocks, which is a bit unfortunate). Data is in little endian instead of big endian, with .NET compatible types already converted.

[mycroes]

I just release version 0.9.0 with concurrent request support. That means you now no longer need to worry about concurrently invoking read/write actions on the S7Connection (as long as the connection is opened first) and it's blazing fast. Test done at a customer, doing 1000 reads on 5 threads (so 5000 reads total) of short[10]:

• Exclusive reads: 10286ms
• Concurrent reads: 3372ms

Basically it's 3 times as fast, because 3 requests can be send (established during connection init). It might depend on other factors as well, like network speed, PLC cpu, but I was really amazed by the performance improvement. Anyway, it might be possible that opening more connections is even faster, but I won't add connection pooling to Sally7, because I feel it shouldn't be part of the library.

I highly recommend you to try out the concurrent support and see if it satisfies your needs.

[scamille]

That sounds exciting. If the S7 protocol itself allows for up to 3 concurrent request channels, that is definitely the way to go.

Definitely a good reason to get my Sally7 driver into a production-ready state.

[mycroes]

It depends on the PLC, but I believe S7 1500 always allows up to 3 concurrent requests. S7 1200 doesn't allow concurrent use at all AFAIK, I might add an optimization to that vs the current channeled implementation. I didn't notice any negative impact when performing requests sync (using synchronization outside S7Connection) though, and memory allocation was pretty low anyway, so it shouldn't be a blocker... Also, the synchronization barrier now is around socket send and receive only, so there's performance improvements no matter the level of concurrency.