In client/coap.py, _wait_for_retransmit_thread may never exit in some cases

[8749236]

My setup:
Server:
I was hosting a coap server with coapthon on Linux Embedded Board with address 127.0.0.1:5683. The server has an observable resource on "data/obs"
Client: I used coapthon HelperClient to observe "data/obs".
My code also starts another guard thread that monitors incoming traffic from observer.

• guard thread will block for response.max_age + 30 seconds
• if blocking was not cancelled, then guard thread will implicitly cancel existing observation and start a new one
• If response is received before blocking finishes, guard thread will remain dormant
• HelperClient is re-used instead of creating a new one.

What happened:
I decided to leave the setup run for 24 hours with a highly unreliable data source. i.e. Coap Server will not honoor the max_age it had sent and causing guard thread to restart observation multiple times.
On the second day I discovered coapthon consumed almost all memory on Linux Embedded Board, and looking at the log from coapthon logger shows it had created thousands of threads (~4850 threads) and majority of them are stuck in "_wait_for_retransmit_thread" from client/coap.py, waiting for retransmit thread to finish.

NOTE: I have yet to find a reliable execution path that duplicate this problem. But it is severe enough that once it happens, it may disable our product entirely.

What I had done:

• Stops re-using HelperClient and create fresh one for every observation

Here is tiny portion of dump from logger "coapthon.client.coap", the log file contains thousands of lines of "Waiting for retransmit thread to finish" which I think is not helpful to post all of them. If you want complete log, please let me know.

02/19/2020 07:39:54 PM DEBUG Thread-1629-Retry-11856 retransmit loop ... retransmit Request
02/19/2020 07:39:54 PM DEBUG Thread-1629-Retry-11856 send_datagram - From None, To (u'127.0.0.1', 5683), CON-11856, GET-lT, [Observe: 0, Uri-Path: data, Uri-Path: obs, Block2: 18, ] No payload
02/19/2020 07:39:54 PM DEBUG Thread-1629 Waiting for retransmit thread to finish ...
02/19/2020 07:39:54 PM DEBUG Thread-1629 Waiting for retransmit thread to finish ...
02/19/2020 07:39:54 PM DEBUG Thread-1629 Waiting for retransmit thread to finish ...
02/19/2020 07:39:54 PM DEBUG Thread-1629 Waiting for retransmit thread to finish ...
02/19/2020 07:39:54 PM DEBUG Thread-1629 Waiting for retransmit thread to finish ...
02/19/2020 07:39:54 PM DEBUG Thread-1629-Retry-11875 retransmit loop ... retransmit Request
02/19/2020 07:39:54 PM DEBUG Thread-1629-Retry-11875 send_datagram - From None, To (u'127.0.0.1', 5683), CON-11875, GET-nK, [Observe: 0, Uri-Path: data, Uri-Path: obs, Block2: 18, ] No payload
02/19/2020 07:39:54 PM DEBUG Thread-1629 Waiting for retransmit thread to finish ...
02/19/2020 07:39:54 PM DEBUG Thread-1629 Waiting for retransmit thread to finish ...
02/19/2020 07:39:54 PM DEBUG Thread-1629 Waiting for retransmit thread to finish ...
02/19/2020 07:39:54 PM DEBUG Thread-1629 Waiting for retransmit thread to finish ...
02/19/2020 07:39:54 PM DEBUG Thread-1629 Waiting for retransmit thread to finish ...
02/19/2020 07:39:54 PM DEBUG Thread-1629 Waiting for retransmit thread to finish ...
02/19/2020 07:39:54 PM DEBUG Thread-1629 Waiting for retransmit thread to finish ...
02/19/2020 07:39:54 PM DEBUG Thread-1629 Waiting for retransmit thread to finish ...