Mainnet stuck because of bad transaction #381

Closed
simonmorgenthaler opened this Issue Jan 8, 2017 · 27 comments

Projects

None yet

8 participants

@simonmorgenthaler
simonmorgenthaler commented Jan 8, 2017 edited

This seems to be the bad transaction: https://explorer.lisk.io/tx/14467342019343228805

Error in pgsql.log:
2017-01-08 15:24:11 UTC ERROR: value too long for type character varying(64)
2017-01-08 15:24:11 UTC STATEMENT: insert into "mem_accounts2u_delegates" ("accountId", "dependentId") values ('202243225L', '8a6d629685b18e17e5f534065bad4984a8aa6b499c5783c3e65f61779e6da06czz');

The string '8a6d629685b18e17e5f534065bad4984a8aa6b499c5783c3e65f61779e6da06czz' is 66 chars long

My nodes are all in the status "Syncing"

@Gr33nDrag0n69

cc001 infos is right (from what i posted in lisk chat) will try to add other things here

@Gr33nDrag0n69

The transaction is linked to the same delegate that got the chain stuck last summer
axlsaddek
https://explorer.lisknode.io/address/202243225L

@Gr33nDrag0n69

The chain isnt forked and everyone is stuck at same height with 100% consensus

@Gr33nDrag0n69

rebuild wont help and clearing memtable too because the tx isnt expired and get rebroadcasted

@Gr33nDrag0n69

a quick fix could be to extend the size of problematic field in the db so it get written in the chain but i fear we could have a biger problem after

@simonmorgenthaler

Question is: why and how is this string too long?

@gregory0219998

The problem starts from the lenght of the wallet... It's all related

@Gr33nDrag0n69

also, i dont know if related but i got a lot of this one (using debug level)

[dbg] 2017-01-08 16:42:07 | Broadcast relays exhausted - {"type":2,"amount":0,"senderPublicKey":"ac395d9a9144768284f29b1aa3f6a0237b03be4b37c4cf526630097540d43ffc","timestamp":19777491,"asset":{"delegate":{"username":"virtum","publicKey":"ac395d9a9144768284f29b1aa3f6a0237b03be4b37c4cf526630097540d43ffc"}},"signature":"9358d80a413a74f127562171c79ad42ec5b2e691014dc4bb9d2e2cfd4074b6f80bf6dd680fc9f29644595a9360c762eac0338f423290a494fb091155ac275800","id":"12145996864416172549","fee":2500000000,"senderId":"12492293221048511653L","relays":2}

@Gr33nDrag0n69

but i think this is normal behavior not related

@Gr33nDrag0n69

the account behind transaction is weird too, so it might be a weird fuck related to that

@Gr33nDrag0n69

i dont know cc001

@viper-tkd

Did anybody notice the word bad in the too long string: The string 8a6d629685b18e17e5f534065bad4984a8aa6b499c5783c3e65f61779e6da06czz

I don't know if it's a coincidence or if it was made like that to break the network.

@Odsejen
Odsejen commented Jan 8, 2017

just looked at the transactions from & to the the account "axlsaddek", one connected acount is the Lisk address "24213L" is this really a common address?

@Gr33nDrag0n69

viper does the lenght is 3 chars too long ? thats odd

@karmacoma karmacoma added the bug label Jan 8, 2017
@karmacoma karmacoma self-assigned this Jan 8, 2017
@viper-tkd

@Gr33nDrag0n69 Dunno. I didn't check the length of the field in the database. But I suspect it is.

@simonmorgenthaler

@viper-tkd @Gr33nDrag0n69 the string is 66 chars long. 2 chars too long

@vrlc92
vrlc92 commented Jan 8, 2017 edited

The bad transaction was made by @AxlSaddek.

Months ago he reported this issue (about Address length <16):
LiskHQ/lisk-explorer#9

screen shot 2017-01-08 at 1 55 20 pm

@1towia
1towia commented Jan 8, 2017

There is some checks missing here I think :
https://github.com/LiskHQ/lisk/blob/master/logic/vote.js#L157

The code doesn't verify if the publicKeys in the transaction are valids.

@simonmorgenthaler

it just expired:
[inf] 2017-01-08 17:09:16 | Expired transaction: 14467342019343228805 received at: Sun, 08 Jan 2017 14:08:56 GMT

@simonmorgenthaler

after reload the nodes seems ok for a short time. I guess the transaction is still broadcasted
and make the nodes stuck again
The transaction is expired only on 2 of my 3 nodes

@simonmorgenthaler

after reload:
OK, then these messages:
[ERR] 2017-01-08 17:27:33 | Failed to get height from peer: 213.136.81.40:8000
[ERR] 2017-01-08 17:27:33 | Failed to apply unconfirmed transaction: 8224227198097948535 - Failed to add vote, account has already voted for this delegate
[ERR] 2017-01-08 17:27:34 | Failed to apply unconfirmed transaction: 1219768254938248202 - Failed to add vote, account has already voted for this delegate
[ERR] 2017-01-08 17:27:34 | Failed to apply unconfirmed transaction: 12145996864416172549 - Account does not have enough LSK: 12492293221048511653L balance: 1
[ERR] 2017-01-08 17:27:34 | Failed to apply unconfirmed transaction: 989175740950637023 - Account does not have enough LSK: 17836753152065928804L balance: 493.2
[inf] 2017-01-08 17:27:44 | Starting sync

then stuck again

@Odsejen
Odsejen commented Jan 8, 2017

Here the tx "14467342019343228805" is expired (appr. at 17:09 ) on all 4 nodes,

@Gr33nDrag0n69

just retry a rebuild after the tx expired, no go

@viper-tkd

same here on all nodes

@Odsejen
Odsejen commented Jan 8, 2017 edited

ok no go, just tried a rebuild with one of the previous nodes which showed the tx "x" is expired, now i have similar messages as @simonmorgenthaler mentioned (and as before stuck at xxx836), but no expiration messages anymore, but instead these "failed to apply" messages and broadhash consensus is/was in all cases 100%.

  • Failed to apply unconfirmed transaction: 17649204730636712354 - Failed to remove vote, account has not voted for this delegate
  • Failed to apply unconfirmed transaction: 8224227198097948535 - Failed to add vote, account has already voted for this delegate
  • Failed to apply unconfirmed transaction: 1219768254938248202 - Failed to add vote, account has already voted for this delegate
@vrlc92
vrlc92 commented Jan 8, 2017 edited

Look:
#165

screen shot 2017-01-08 at 3 33 59 pm

@1towia
1towia commented Jan 8, 2017

@vrlc92 He didn't used the testnet, like last time. Not good @AxlSaddek.

@karmacoma karmacoma added a commit that closed this issue Jan 8, 2017
@karmacoma karmacoma Closes #381. Verifying each vote in trs.asset.votes.
Checking vote type, format and length.
b5cbd58
@karmacoma karmacoma closed this in b5cbd58 Jan 8, 2017
@karmacoma karmacoma added a commit that referenced this issue Jan 8, 2017
@karmacoma karmacoma Closes #381. Verifying each vote in trs.asset.votes.
Checking vote type, format and length.
d37a075
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment