New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Many parallel transfers leads to some becoming stuck #2567

Closed
ulope opened this Issue Sep 24, 2018 · 5 comments

Comments

Projects
None yet
5 participants
@ulope
Collaborator

ulope commented Sep 24, 2018

Problem Definition

This scenario leads to some transfers becoming stuck. Usually no later than the second repeat of the serial task set.

From quickly investigating the logs it appears that at some point messages aren't being processed anymore.

Needs more digging.

@palango

This comment has been minimized.

Collaborator

palango commented Oct 2, 2018

This seems to have happened on the first run as well. Logs are attached.
image
raiden-debug.log
raiden-debug1.log
raiden-debug2.log

@hackaugusto

This comment has been minimized.

Collaborator

hackaugusto commented Oct 2, 2018

@palango The transfer did not get stuck, it expired.

  • Node fcfad555 started a transfer to 2a0374bf with the id 4242318251562065606 at time 13:24:17. It's local view of the block number was at block number 4156434, and the lock's expiration was 4156454.
  • The corresponding message was sent at 13:24:17 and received at 13:24:19 the receiver's block number was at 4156444, but discarded because it was out of order:

2018-10-02 13:24:19 [debug ] Raiden event [raiden.raiden_service] node=2a0374bf raiden_event={"identifier": 4242318251562065606, "secrethash": "0x0c3b8baed28637cc49f10e767845a5a78d0897642483ff7d4c44d6fe67f04daf", "reason": "Invalid LockedTransfer message. Nonce did not change sequentially, expected: 14 got: 15.", "_type": "raiden.transfer.mediated_transfer.events.EventUnlockClaimFailed", "_version": 0}

  • The message was retried at 13:24:24 and received at 13:24:25 and then it failed because it expired. the receiver's block number was at 4156445:

2018-10-02 13:24:26 [debug ] Raiden event [raiden.raiden_service] node=2a0374bf raiden_event={"identifier": 4242318251562065606, "secrethash": "0x0c3b8baed28637cc49f10e767845a5a78d0897642483ff7d4c44d6fe67f04daf", "reason": "lock timeout is unsafe. timeout must be larger than 10, but it is 10. expiration: 4156454 block_number: 4156444", "_type": "raiden.transfer.mediated_transfer.events.EventUnlockClaimFailed", "_version": 0}

The problem here is that the sender is reveal timeout blocks behind the receiver, which basically means the lock was inside the danger zone when it was sent, and the receiver rejected it.

The solution is just to increase the reveal timeout.

@palango

This comment has been minimized.

Collaborator

palango commented Oct 4, 2018

So what would be a reasonable reveal timeout?

Using the medium gas price strategy should result in transaction times of around 5min, which would be roughly 20 blocks. Not sure if we should add some safety to that!? @hackaugusto @LefterisJP

@LefterisJP

This comment has been minimized.

Collaborator

LefterisJP commented Oct 4, 2018

I would go for a safety margin, and not only for Ropsten. 50 blocks sounds good.

@hackaugusto

This comment has been minimized.

Collaborator

hackaugusto commented Oct 4, 2018

So what would be a reasonable reveal timeout?

The reveal timeout is how long it takes for one transaction to be mined under congestion. Our current assumption is that web3.py gas price estimation will chose the price so that the transaction is mined in 20 blocks. So 50 blocks sounds good.

palango added a commit to palango/raiden that referenced this issue Oct 4, 2018

Increase the reveal timeout
We assume at least the medium gas price strategy is used which should
result in most transactions being mined within 5min or roughly 20
blocks. This is increased to 50 blocks ass an additional security
measure.

See discussion at raiden-network#2567

palango added a commit to palango/raiden that referenced this issue Oct 4, 2018

Increase the reveal timeout
We assume at least the medium gas price strategy is used which should
result in most transactions being mined within 5min or roughly 20
blocks. This is increased to 50 blocks as an additional security
measure.

See discussion at raiden-network#2567

@palango palango closed this in #2670 Oct 4, 2018

palango added a commit that referenced this issue Oct 4, 2018

Increase the reveal timeout
We assume at least the medium gas price strategy is used which should
result in most transactions being mined within 5min or roughly 20
blocks. This is increased to 50 blocks as an additional security
measure.

See discussion at #2567
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment