Connection close on tty: response timeout

[kinkazu]

Hi there.
Let me congratulate you an excelent piece of software that i use on a daily basis.
And now to a problem that I have.

I have a small debian based PC running mbusd and connected to Modbus/CANBUS gateway over RS232.
Modbus/CANBUS gateway is connected over CANBUS to 20 devices.
Modbus queries to debian PC running your software I can read from all 20 devices over Modbus/TCP.

Problem starts is when one of CANBUS devices is switched off:
I am not a programmer or any expert, but it looks like this scenario somehow clogs up mbusd or Modbus/Gateway.

Clogs up, means I that Modbus reading goes fine with some CANBUS devices and doesn't work with others. Timeout setting doesn't seem to have any affect.

Software querying devices is not written by me and i have no idea how it operates, but works fine when all devices ur powered up.

For my test I disconneected all devices and started querying software on one, switched offdevice only.

mbus.log shows:

20 Jul 2018 12:53:01 tty: state now is TTY_PAUSE
20 Jul 2018 12:53:01 conn_loop(): select(): max_sd = 4, t_out = 000000:001000
20 Jul 2018 12:53:01 conn_loop(): select() returns 0
20 Jul 2018 12:53:01 tty: state now is TTY_READY
20 Jul 2018 12:53:01 conn_loop(): select(): max_sd = 4, t_out = 000010:000000
20 Jul 2018 12:53:11 conn_loop(): select() returns 0
20 Jul 2018 12:53:11 conn_loop(): select(): max_sd = 4, t_out = 000010:000000
20 Jul 2018 12:53:15 conn_loop(): select() returns 1
20 Jul 2018 12:53:15 conn_open(): accepting connection from 10.3.8.7
20 Jul 2018 12:53:15 queue_new_elem(): length now is 1
20 Jul 2018 12:53:15 conn[10.3.8.7]: state now is CONN_HEADER
20 Jul 2018 12:53:15 conn_loop(): select(): max_sd = 5, t_out = 000010:000000
20 Jul 2018 12:53:15 conn_loop(): select() returns 1
20 Jul 2018 12:53:15 conn[10.3.8.7]: state now is CONN_RQST_FUNC
20 Jul 2018 12:53:15 conn[10.3.8.7]: read request fc 3
20 Jul 2018 12:53:15 conn[10.3.8.7]: state now is CONN_RQST_TAIL
20 Jul 2018 12:53:15 conn[10.3.8.7]: request: [14][03][00][a2][00][01]
20 Jul 2018 12:53:15 conn[10.3.8.7]: state now is CONN_TTY
20 Jul 2018 12:53:15 tty: state now is TTY_RQST
20 Jul 2018 12:53:15 conn_loop(): select(): max_sd = 4, t_out = 000010:000000
20 Jul 2018 12:53:15 conn_loop(): select() returns 1
20 Jul 2018 12:53:15 tty: written 8 bytes
20 Jul 2018 12:53:15 tty: request written (total 8 bytes)
20 Jul 2018 12:53:15 tty: state now is TTY_RESP
20 Jul 2018 12:53:15 tty: estimated 7 bytes, waiting 4015624 usec
20 Jul 2018 12:53:15 conn_loop(): select(): max_sd = 4, t_out = 000004:015624
20 Jul 2018 12:53:19 conn_loop(): select() returns 0
20 Jul 2018 12:53:19 tty: response timeout
20 Jul 2018 12:53:19 conn_close(): closing connection from 10.3.8.7
20 Jul 2018 12:53:19 queue_delete_elem(): length now is 0
20 Jul 2018 12:53:19 conn_loop(): select(): max_sd = 4, t_out = 000004:015624
20 Jul 2018 12:53:23 conn_loop(): select() returns 0
20 Jul 2018 12:53:23 tty: response timeout

I have no idea what this log means :) All i know that device is not there and when I use standard libmodbus example Modbus/Gateway responds:
./modbus_client --debug -mtcp -t0x03 -a13 -r13 -p502 10.3.8.6 -c1
Connecting to 10.3.8.6:502
[00][01][00][00][00][06][0D][03][00][0D][00][01]
Waiting for a confirmation...
ERROR Connection timed out: select
ERROR occured!

Is there any modification in conn.c I can do for mbusd to drop a connection like that go to the next one in the que without retrying anything?

My mbusd start line is:
/usr/bin/mbusd -v 0 -p /dev/ttyO0 -P 502 -s 9600 -R 1 -T 5 -C 128 -N 3

Kind regards

[3cky]

Hello @kinkazu!

Thank you for the kind words. First of all, mbusd can be instructed to completely disable tty query retrying by specifying -N 0 command line option. Device query timeout can be specified by -W option (defaults to 500 ms), but you mentioned that setting doesn't seem to have any affect, isn't?

[kinkazu]

Hi Victor.

Yes, I played with N and W settings with no success. N and W setting seem to work fine when device is not available in TCP/IP network and modbus_client is throwing out:
Connecting to 10.3.8.20:502
Connection failed: Operation now in progress
But not when device/gateway is in TCP/IP network, but one of the nodes is absent and modbus_client is throwing out:
Connecting to 10.3.8.6:502
[00][01][00][00][00][06][14][03][00][0D][00][01]
Waiting for a confirmation...
ERROR Connection timed out: select
ERROR occured!

I can do all is needed to help to figure out where the problem is. I can set up test environment for you so you could see yourself. I don't want to waste your time.

Thank you.

[3cky]

Just for clarification, those timeouts occurs while querying offline CANBUS devices only, or some of online CANBUS devices are also inaccessible after network membership changes?

[kinkazu]

Scenario:
14 devices powered ON and connected to Modbus/Canbus gateway
Communication using mbusd/RS232/modbus/Canbus gateway works fine
1 of 14 becomes offline/is switched off
Communication to 9 devices remains the same
Communication to 2 devices slows down dramatically (always the same ones)
Communication to 2 devices is completely broken (always the same ones)

[3cky]

That's really weird. Don't have much expirience with CAN bus, but it looks like some hardware-related issue, probably some CAN gateway or network misconfiguration. Anyway, mbusd should handle such situations gracefully using timeout and retries limit, so probably I should check mbusd timeout handling code.

[kinkazu]

I am not sure if it has anything to do with canbus. I wonder how mbusd behaves if connected to more than one device over RS485 link and one of the devices is powered off.
I am hoping you will find some solution. I love mbusd :).

[kinkazu]

Is it possible that if device is missing, mbusd is not responding properly to modbus_close or modbus_free and not closing connection or removing query from queue?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.