-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allen Bradley Micrologix 1400 PLC Error "Failure (Bad values)" #2
Comments
Hello, I'm not aware of this problem. This node has been developed and published in a partnership with another guy that had the PLC, I never had one myself, so I can't confirm or test this behavior myself. For us to better understand the problem, could you please run your node-red in verbose mode (with the Beside that, could you also please post your flow? Even though I don't have a PLC here, I can try to use SoftLogix and check if it also happens with a SoftPLC. |
Something curious happened. I had to delete the PCCC read and write nodes and the configuration node in order to keep from getting an error. When I put them back in order to do the debug today. I put in the configuration only with N7:0... then I added N7:0/0 when it would not read or write, however, it did say "online". [{"id":"7d21a8eb.0d0578","type":"function","z":"46d5e03a.f74e1","name":"counter","func":"var count = msg.payload;\nif (count>32766)\n{\n count=0;\n}\ncount=count+1;\nmsg.payload=count;\nreturn msg;","outputs":1,"noerr":0,"x":370,"y":1140,"wires":[["b7b90936.ad3588","bbdfd60a.7b0b58"]]},{"id":"bbdfd60a.7b0b58","type":"pccc out","z":"46d5e03a.f74e1","endpoint":"88afadd7.3a97d","variable":"N7:0","name":"","x":890,"y":1140,"wires":[]},{"id":"92772800.ebb978","type":"pccc in","z":"46d5e03a.f74e1","endpoint":"88afadd7.3a97d","mode":"single","variable":"N7:0","diff":true,"name":"","x":150,"y":1140,"wires":[["7d21a8eb.0d0578"]]},{"id":"88afadd7.3a97d","type":"pccc endpoint","z":"","address":"192.168.254.220","port":"44818","userouting":false,"routing":"0x01,0x00,0x01,0x00","cycletime":"500","timeout":"1500","name":"","vartable":[{"addr":"N7:0/0","name":"N7:0/0"},{"addr":"N7:0/1","name":"N7:0/1"},{"addr":"N7:0/2","name":"N7:0/2"},{"addr":"N7:0/3","name":"N7:0/3"},{"addr":"N7:0/4","name":"N7:0/4"},{"addr":"N7:0/5","name":"N7:0/5"},{"addr":"N7:0/6","name":"N7:0/6"},{"addr":"N7:0/7","name":"N7:0/7"},{"addr":"N7:0/8","name":"N7:0/8"},{"addr":"N7:0/9","name":"N7:0/9"},{"addr":"N7:0/10","name":"N7:0/10"},{"addr":"N7:0/11","name":"N7:0/11"},{"addr":"N7:0/12","name":"N7:0/12"},{"addr":"N7:0/13","name":"N7:0/13"},{"addr":"N7:0/14","name":"N7:0/14"},{"addr":"N7:0/15","name":"N7:0/15"},{"addr":"N7:0","name":"N7:0"}]}] |
Verbose output to Terminal.
|
What would cause the timestamp 950882,536746565 to be out of sequence in this log around the time the write packet seems to go out without a reply? That is odd... Or am I just not thinking of something obvious? Is this problem repeatable with many node versions on multiple platforms? |
I have not tried it on any other platforms or versions of the node. I'm not sure who has had good or bad experience, or what kind of data collection they use with this pccc node. I was hoping that we could get an idea what is going on here and debug it so it could be of use. Right now it's useless and I'll have to resort to using OPC-UA to connect to at least our Allen Bradley Micrologix 1400 PLCs. |
@youngkzy thanks for the information provided. I think we've found the issue. @plcpeople regarding the time drift, that seems to be a platform-dependant that affects the Regarding the main issue, you can see that the connection seems to go away around I'm not as familiar with this code as with nodes7, but I'd suggest calling |
That may be a good solution... My thinking in not doing a reset here is that the connection timeout may be temporary, such as a lost packet on a slow/unreliable connection, so a retry might be more appropriate as constantly creating new connections without properly disconnecting can quickly use all the connection resources available if the PLC doesn't release them right away. And if the TCP connection is broken you should EVENTUALLY get a socket error, but this can take a long time on a timeout. I wonder why a retry wouldn't work, but a reconnect would? I will see if I can run a test. I don't mind making the change as it is really simple, I just want to make sure we don't cause issues for any existing code. |
@plcpeople and @gfcittolin , I am eagerly awaiting any progress on a test mentioned above in your reply regarding "reconnect". My test setup is connecting through a Tripplite USB 3.0 Dual Gigabit adapter to the PLC with a 3' patch cable. I'm running Windows 10. I connect with this device and I have not had issues with it in any other programming and testing scenarios. That being said, does not mean that the device is not part of the problem. Right now I don't have the expertise or resources to delve into this deeper. However, I am curious to understand what's wrong and how to fix it. Thank you for your insight and willingness to help me resolve this problem. |
I've just spent some hours reading and re-learning some network stuff and doing simulations, and let me share my discoveries that may impact this project and nodeS7 too. @plcpeople It looks like things can get a bit more complicated when dealing with these dark corners of a TCP socket. Looks like that sending a network packet from a userland process does not necessarily implies that this packet will be sent over the wire. Congestions mechanisms from the OS (and this is therefore OS dependant) may queue them up before receiving the acknowledge of a previous packet. As a proof of concept, I've setup a connection between my computer and a S7-1200 (PLC brand is irrelevant here, as the same logic is being used on nodeS7), but going over two dummy switches. After connection establishment and data exchange for a while, I unplugged the cable between the two switches, so that neither the PLC nor my PC would detect that the physical link is down (and therefore signalize this to the application). Even though we were calling I think your concern regarding resource exaustion in the PLC is valid and very important, that's why I've invested a lot of time on this after all. But from my experience with industrial networks and after the test above, I'm pretty much convinced that we really need either to reset the connection on packet timeouts, or find another way so that this behavior don't happen. After all, if just a single packet was dropped, we'd close the connection and send a FIN packet, releasing the resource. And if a more permanent issue is taking place preventing the packets from being delivered, closing the connection shouldn't make any difference anyway, as the PLC won't receive anything, having us to rely on PLC mechanisms to free up the resource anyway. What we can do as to avoid flooding the PLCs with connections is to increase the timeout before actually dropping the connection and creating a new one to something like 5 seconds. I'll do some more tests and reading about this, but if you agree, I think we should call |
@gfcittolin , How is the test going? Did you modify the code to include your suggestion to call "connectionReset()"? Do you plan on submitting a trial version? Kind Regards, |
Hello, I was in business trip last week and wasn't able to do any tests or modifications. I plan to work on this tomorrow. I'll run some basic tests and then I let you know about it so you can also test in your environment. |
Dear @youngkzy, @plcpeople has kindly published a new version that implements the fixes proposed on this issue. Could you please confirm the issue has been solved? |
@gfcittolin , Thank you for your quick response to this. I've tried to keep up, but I missed checking the forum for a few days. I'll test as soon as I can get time today. |
OK, where was the revision posted? How do I test it? Install ?? Update? 1.1.1 is the newest? Please help. |
I followed the link to "plcpeople/nodepccc". And I found the files which I copied to my existing nodepccc folder. I renamed the files "nodePCCC.js" and "package.json" and copied the content to new files from the repository. I ran a test and am running a test to read N7:0 and then increment the number, and if it is higher than 32766 it will reset the value to 0. Then it writes it out to N7:0. I saw an error when I added an inject node to reset the value to 0 manually, but I had not deployed the change when I clicked on the inject. I had to deploy first. The counter is at 1800 and no problems yet if I just watch and don't make changes. I'll watch and update when an error occurs. |
OK, The behavior is now different. It does not continue to repeat the errors indefinitely. It stops at The error is the same as before, "Failure (Bad Values)". The error points back to the PCCC Configuration Node. I made an attempt to recover by injecting a 0, and/or 1. The result is the input and output nodes report a value instead of "Failure", but will not update the PLC with new values. The value sent to the PCCC output registers on the node, but the input does not reflect any activity. |
I did a lot more testing yesterday with similar behavior seen after the same change on nodeS7 and found that after this new connection reset logic was first triggered, the code would not have a chance to re-establish connection, so I have posted yet another fix. Please test the latest when you have a chance. |
I copied the files from "https://github.com/plcpeople/nodepccc", and applied them to the "nodepccc" folder. I resumed the test which had failed before with similar results. Today's test is going much better! The test comprises of a read node, reading N7:0, an increment/rollover function to the value and then posting the new value with a write node. It usually would bomb anywhere from 1200 or less and that varies on something that is unknown. Today's test is going much better with a count of around 22,000 with 3 hours under the belt. Well, It just bombed at 21,633... The difference is that I had let my PC alone for a couple of hours this morning... Upon returning, I was in the process of checking the functions and writing this response. After I updated it just now, I checked and it was in error, again. "Failure (Bad values)". Reset and it bombed again at 21,900. (sigh...) |
Any chance of getting verbose output of the latest failure? |
[717325,833391040] <Buffer 6f 00 21 00 d6 0f a9 e1 00 00 00 00 00 00 00 01 00 28 1e 4d 00 00 00 00 00 00 00 00 00 04 02 00 00 00 00 00 b2 00 11 00 cb 00 00 00 07 00 00 01 02 03 ... > |
Another weekend, another version, sorry it takes me a few days to get to this each time. The verbose output was really useful in understanding what is happening - in cases where more reads or writes are done in the callback, the way we were doing the reset can lead to a non-response and therefore communication would stop. I am really hoping this is the end of the issue with these changes. I have a test running in Node-RED (previous testing was outside of Node-RED) and I will let it run overnight, but it has recovered quickly from anything I have done so far. I added a bit more output as well, so please post verbose output if you are able to make it fail again. My flow is very simple and hopefully close enough to yours. |
My Node-RED program is taking the input value and increment by 1. If it is the numerical value, 32,767, or higher, it resets to 0. There is no delay, The system is only limited in speed by the read/write functions and the response time from Node-RED. Should I include a 500ms delay? I don't see the means for your program to increment, or change the value. You must be incrementing "TheInt" in the PLC and just writing it to TheIntWrite and those are not the same address. I'll try to monitor it tomorrow 9-17-2019. Thanks for your time to fix this. Kind Regards, |
Correct, the PLC was incrementing the value. I modified my flow with no delay to be more similar to yours, reading/writing the same value and incrementing in Node-RED and it works. I don't think you need a delay as the "cycle time" is set to 500ms which will add a delay between the write and the read. |
UPDATE: |
This is the capture I promised in the last post. I'm hoping it provides insight. The only problem I can identify is a small one. If there is an error either reading or writing, I would need to READ again, or WRITE again after the error in order to ensure the values are complete. Since the ERROR in Node-RED is relative to the "Configuration" Node, the system does not allow usage of the "Catch Error" node in Node-RED. It seems the "Catch Error" Node only will work on the flow tab it resides. Since the node which errors is actually the "Configuration" Node. I don't know how to "Catch" this error and re-do a WRITE or a READ that had failed. I'm assuming the ERROR needs to be passed to the READ and/or WRITE node on the flow page.... [518305,588837047 192.168.254.220] Preparing to WRITE N7:0 |
How can I avoid these errors in the first place? Do I need to adjust the default parameters for a longer cycle time? Timeout? |
Nick O'Leary at Node-RED.org wrote in reply to a question about triggering a Catch Node... It looks like a simple update. "the Catch node will only handle errors that have been logged 'properly'. If the node simply calls node.error("my message"); then that will appear in the sidebar and log, but it will not trigger a Catch node. To trigger a Catch node, the call to node.error() must also provide the message that triggered the error - node.error("my message", msg). If you search through the src of the pccc node - https://github.com/netsmarttech/node-red-contrib-pccc/blob/master/red/pccc.js - you can see it only ever provides an error string to the node.error() call., it never provides the second argument so it will not trigger the Catch node. The node would need to be updated for you to be able to handle those errors with the Catch node." |
I think from looking at the earlier Wireshark captures that there is something unique to your network causing the timeout errors but @gfcittolin looked into this a bit more. If I am direct connected I don't see any errors at all for days unless I create them. But I am testing with a SLC 5/03 and ENI module so response times are much longer (50ms). But I think somehow packets might be getting dropped. Either way, it should recover from any error when communication is possible again and it seems to be doing that now. Is that verbose output truncated? It looks normal now - it looks like it resumed communication after the error. Is there any chance there is anything else you could do in your flow to make it write after an error? I did a simple flow that would do a read and a write without much else and it resumed after an error just fine without anything to catch an error, although you might want to modify the call to node.error as suggested to see if that helps. This simple flow (the function is basically just return msg;) immediately starts printing debug messages after recovering from a failure. Does your flow do this if you put a debug right after your read? |
It does resume after an error. My system is connected directly to a test Micrologix 1400 PLC with no other item in between. The dropping packets could be just from the Tripplite USB 3.0 dual gigabit Ethernet adapter. It usually does not have errors unless I'm working and using my browser. If I don't do anything, it behaves well. Nonetheless, I'd like to get it to update or read properly even if it does have an error. The debug message only shows the message object. From the message, I cannot get the fact it had any errors. The error does show up on the debug pane, but the errors cannot be caught by the CATCH ERROR Node. Reference prior post quote from Nick O'Leary. I think the programming works OK NOW. It's just necessary to capture errors in context to read/write function which did or did not happen due to errors. If my PLC Program increments the value like yours, it would just march on because the read values will catch the next change. However, the problem exists that if data exchange is rare and unique events that need to be caught, We may miss them when they coincide with an error of some sort. Or maybe I'm overthinking this... |
Hopefully it works if you're able to test Nick's proposed changes and the problem is solved after that. I am not sure what to suggest if the flow misses a rare event without knowing more about the application other than we've had to store rare events in the PLC to make sure they weren't missed by a computer, even with other software. But I think you might see an improvement with a different Ethernet adapter because there won't likely be any errors to cause problems then. |
@gfcittolin, "To trigger a Catch node, the call to node.error() must also provide the message that triggered the error - node.error("my message", msg). If you search through the src of the pccc node - https://github.com/netsmarttech/node-red-contrib-pccc/blob/master/red/pccc.js - you can see it only ever provides an error string to the node.error() call., it never provides the second argument so it will not trigger the Catch node." My results of just updating the code with ", msg" adding the second parameter failed because it does not have a reference to "msg". The scope is a problem. Can you please help by taking a look at this in order to help me resolve exposing the errors to the "Catch Error" Node? This would help very much and I think it would be helpful to anyone else using the node(s). Kind Regards. |
@gfcittolin, |
So sorry for not having time for working on this lately, I've been extremely busy at work and outside it. (Thanks @plcpeople for your support here!) I'll try to take a look on this during this week. |
@gfcittolin, |
New Circuit... The system works better if the first detected error, "output", is delayed and the second "input" error should also be delayed from the first resulting error. The input condition and output condition are very close in time. In order to prevent things overlapping delays in the results are needed. You could probably ignore the "input" error. But this shows you can do some logic on it if you need to. |
Holy Moly! Can this get any more frustrating? I had it all working, Then I discovered a problem with some of the injection nodes not able to select false values for boolean datatypes. Nick O'Leary responds back and says "You need to upgrade Node-RED to 1.0.0 instead of your current beta version. OK I upgrade and now we are back with problems with errors that won't reset with 'node-red-contrib-pccc' : ( @gfcittolin and @plcpeople , Can you once again help with this? or... can you please indicate what bug may be causing problems with the reset? I'll post the detail results again for you here. Kind Regards, |
[80007,269296061 192.168.254.220] Preparing to WRITE N7:0 |
I finally had a look at this and I think that somehow when Node-RED 1.0.0 was installed and all your packages were "updated", node-red-contrib-pccc (or something somehow) may have installed an older version of nodePCCC. You should be at 0.1.16. I can tell from this log that it doesn't even have the very first modification that was done to reset the connection on a packet timeout. @gfcittolin assuming I'm not wrong about this, why would a new install not have installed the latest? |
Yes, I had to copy the "package.json" and "nodePCCC.js" files from the website again to my "C:\Users\myname.node-red\node_modules\nodepccc" folder this morning. It's actually working great again. Thanks for your look at this. Is this because I just copied the files before and did not install the system as a unit? |
For some strange reason my system must have updated "node-red-contrib-pccc" node. The files in my C:\Users\myname.node-red\node_modules\nodepccc\nodePCCC.js and package.json were from an older version today when I started my system. I copied the files from nodePCCC again today. How do I prevent this from overwriting newer files with old ones? |
That's on me. I've just published a new version of the node with the dependency updated to the latest I hope that everything should be now fixed! Could you maybe try to install the node on a clean installation and check if everything is working as expected? P.S.: Thank you so much @plcpeople for your help! |
@gfcittolin, Thank you for the time and energy spent on this node. The project I'm starting will benefit from it because we still use Micrologix and SLC5/05 PLC's. Although, I still need to hook up an SLC to test it on... I still get multiple errors "Failure (Bad values)", my test configuration is using the "Status" node to re-inject the last value stored in a variable, and it resumes working. I am not sure if this problem is something you can address. Maybe sketchy hardware... etc... But I was wondering if it would be any benefit to use the "Catch" to trap the error vs. the "Status". That's not a "need". Thanks again for everything. I will try to install it on the "clean" Windows 10 system shortly. Maybe this month. I'll keep you posted. I'll close the issue after I test it further on the SLC and new install. |
I'm connected to the MicroLogix-1400, working just fine, until errors.
192.168.254.220 Port 44818
Routing <--- ?? what's that ?
Cycle Time Adjusted it from 500 -700, No differences noted
Timeout, Adjusted it from 1500 - 3500, No differences noted
Node-RED program is simple.![pccc-node-red-failure](https://user-images.githubusercontent.com/48096208/61980197-02fb8680-afc4-11e9-8664-76e3daf16f20.png)
I have
pccc-node-red-failure
Input - 500 ms delay - json true =1; json false =2; on output 1>> Set msg.payload = 0
on output 2>> Set msg.payload = 1.
This is a simple 50% duty cycle at 500ms. It works perfectly... Until
"Failure (Bad values)" erupts on the debug pane. Usually it works for about 5 minutes. and crashes.
I tried an alternate test with a word vs. bit modification. It lasted about a half hour when left to run by itself. When I returned to my PC and started working on email or internet, it failed with the same problem.
Kind Regards,
kzy
The text was updated successfully, but these errors were encountered: