Mediated transfer failing on stopping a mediated node

[agatsoh]

Problem Definition

I am doing mediated transfers through a script which does 10 mediated trasnfers one after the other in a loop. if even for one mediated transfer it does not get the correct response(ie 200) it breaks off.
A --> B --> C
After doing atleast 5 or 6 I stop node B. And my script also breaks off.

(raiden_env) [krishna@krishna-pc raiden_scripts]$ python payments_script_goerli_1.py 
The payment number 0 has the status code 200
The payment number 1 has the status code 200
The payment number 2 has the status code 200
The payment number 3 has the status code 200
The payment number 4 has the status code 200
The payment number 5 has the status code 200
The payment number 6 has the status code 409
The status code is not 200 hence breaking off count is 6
Time elapsed  40.777876138687134

The script which I use

import requests
import time

def payment():
    return requests.post('http://localhost:5013/api/v1/payments/0x95B2d84De40a0121061b105E6B54016a49621B44/0xc453Fa8e3258DF2df7Fca7486c21eCE0795aeD62',
                         headers={'Content-Type': 'application/json', }, json={'amount': 1})

start = time.time()
for i in range(10):
    resp = payment()
    print('The payment number {} has the status code {}'.format(i, resp.status_code))
    if resp.status_code != 200:
        print('The status code is not 200 hence breaking off count is {}'.format(i))
        break
end = time.time()

print('Time elapsed ', end - start)

The failure for the initiating node to wait for the mediating node to come up again and make the transfer may be a transport issue.

raiden debug logs
https://drive.google.com/file/d/1Mod2lBUT2hfv-8FR08J2B15Bx4naxCqY/view?usp=sharing

Initiator nodes logs
https://drive.google.com/file/d/1ScznZTNX9E3rwlrrkC5Z9PAqt3pIFOGe/view?usp=sharing

Edit:
The database of the intiator nodes
node_dad7c957.zip

[rakanalh]

Duplicate of #5043

[rakanalh]

@agatsoh could you please attach the database for the initiator node?

[agatsoh]

@rakanalh updated my original comment with the database.

[christianbrb]

Duplicate of #5059

[rakanalh]

Just to give a bit more details here, the error for this issue was:

2019-10-07 14:33:10.472641 [error    ] Error processing request       [raiden.api.rest] errors=Payment couldn't be completed because: there is no route available status_code=<HTTPStatus.CONFLICT: 409>

This is because the PFS filtered out the route based on the node's presence. The node was UNREACHABLE in this case and we've seen the same issue pop up in other places. #5059's fix (if any) should resolve this problem.