New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mainnet stuck because of bad transaction #381

Closed
simonmorgenthaler opened this Issue Jan 8, 2017 · 27 comments

Comments

Projects
None yet
8 participants
@simonmorgenthaler

simonmorgenthaler commented Jan 8, 2017

This seems to be the bad transaction: https://explorer.lisk.io/tx/14467342019343228805

Error in pgsql.log:
2017-01-08 15:24:11 UTC ERROR: value too long for type character varying(64)
2017-01-08 15:24:11 UTC STATEMENT: insert into "mem_accounts2u_delegates" ("accountId", "dependentId") values ('202243225L', '8a6d629685b18e17e5f534065bad4984a8aa6b499c5783c3e65f61779e6da06czz');

The string '8a6d629685b18e17e5f534065bad4984a8aa6b499c5783c3e65f61779e6da06czz' is 66 chars long

My nodes are all in the status "Syncing"

@Gr33nDrag0n69

This comment has been minimized.

Show comment
Hide comment
@Gr33nDrag0n69

Gr33nDrag0n69 Jan 8, 2017

Member

cc001 infos is right (from what i posted in lisk chat) will try to add other things here

Member

Gr33nDrag0n69 commented Jan 8, 2017

cc001 infos is right (from what i posted in lisk chat) will try to add other things here

@Gr33nDrag0n69

This comment has been minimized.

Show comment
Hide comment
@Gr33nDrag0n69

Gr33nDrag0n69 Jan 8, 2017

Member

The transaction is linked to the same delegate that got the chain stuck last summer
axlsaddek
https://explorer.lisknode.io/address/202243225L

Member

Gr33nDrag0n69 commented Jan 8, 2017

The transaction is linked to the same delegate that got the chain stuck last summer
axlsaddek
https://explorer.lisknode.io/address/202243225L

@Gr33nDrag0n69

This comment has been minimized.

Show comment
Hide comment
@Gr33nDrag0n69

Gr33nDrag0n69 Jan 8, 2017

Member

The chain isnt forked and everyone is stuck at same height with 100% consensus

Member

Gr33nDrag0n69 commented Jan 8, 2017

The chain isnt forked and everyone is stuck at same height with 100% consensus

@Gr33nDrag0n69

This comment has been minimized.

Show comment
Hide comment
@Gr33nDrag0n69

Gr33nDrag0n69 Jan 8, 2017

Member

rebuild wont help and clearing memtable too because the tx isnt expired and get rebroadcasted

Member

Gr33nDrag0n69 commented Jan 8, 2017

rebuild wont help and clearing memtable too because the tx isnt expired and get rebroadcasted

@Gr33nDrag0n69

This comment has been minimized.

Show comment
Hide comment
@Gr33nDrag0n69

Gr33nDrag0n69 Jan 8, 2017

Member

a quick fix could be to extend the size of problematic field in the db so it get written in the chain but i fear we could have a biger problem after

Member

Gr33nDrag0n69 commented Jan 8, 2017

a quick fix could be to extend the size of problematic field in the db so it get written in the chain but i fear we could have a biger problem after

@simonmorgenthaler

This comment has been minimized.

Show comment
Hide comment
@simonmorgenthaler

simonmorgenthaler Jan 8, 2017

Question is: why and how is this string too long?

simonmorgenthaler commented Jan 8, 2017

Question is: why and how is this string too long?

@gregorst3

This comment has been minimized.

Show comment
Hide comment
@gregorst3

gregorst3 Jan 8, 2017

The problem starts from the lenght of the wallet... It's all related

gregorst3 commented Jan 8, 2017

The problem starts from the lenght of the wallet... It's all related

@Gr33nDrag0n69

This comment has been minimized.

Show comment
Hide comment
@Gr33nDrag0n69

Gr33nDrag0n69 Jan 8, 2017

Member

also, i dont know if related but i got a lot of this one (using debug level)

[dbg] 2017-01-08 16:42:07 | Broadcast relays exhausted - {"type":2,"amount":0,"senderPublicKey":"ac395d9a9144768284f29b1aa3f6a0237b03be4b37c4cf526630097540d43ffc","timestamp":19777491,"asset":{"delegate":{"username":"virtum","publicKey":"ac395d9a9144768284f29b1aa3f6a0237b03be4b37c4cf526630097540d43ffc"}},"signature":"9358d80a413a74f127562171c79ad42ec5b2e691014dc4bb9d2e2cfd4074b6f80bf6dd680fc9f29644595a9360c762eac0338f423290a494fb091155ac275800","id":"12145996864416172549","fee":2500000000,"senderId":"12492293221048511653L","relays":2}

Member

Gr33nDrag0n69 commented Jan 8, 2017

also, i dont know if related but i got a lot of this one (using debug level)

[dbg] 2017-01-08 16:42:07 | Broadcast relays exhausted - {"type":2,"amount":0,"senderPublicKey":"ac395d9a9144768284f29b1aa3f6a0237b03be4b37c4cf526630097540d43ffc","timestamp":19777491,"asset":{"delegate":{"username":"virtum","publicKey":"ac395d9a9144768284f29b1aa3f6a0237b03be4b37c4cf526630097540d43ffc"}},"signature":"9358d80a413a74f127562171c79ad42ec5b2e691014dc4bb9d2e2cfd4074b6f80bf6dd680fc9f29644595a9360c762eac0338f423290a494fb091155ac275800","id":"12145996864416172549","fee":2500000000,"senderId":"12492293221048511653L","relays":2}

@Gr33nDrag0n69

This comment has been minimized.

Show comment
Hide comment
@Gr33nDrag0n69

Gr33nDrag0n69 Jan 8, 2017

Member

but i think this is normal behavior not related

Member

Gr33nDrag0n69 commented Jan 8, 2017

but i think this is normal behavior not related

@Gr33nDrag0n69

This comment has been minimized.

Show comment
Hide comment
@Gr33nDrag0n69

Gr33nDrag0n69 Jan 8, 2017

Member

the account behind transaction is weird too, so it might be a weird fuck related to that

Member

Gr33nDrag0n69 commented Jan 8, 2017

the account behind transaction is weird too, so it might be a weird fuck related to that

@Gr33nDrag0n69

This comment has been minimized.

Show comment
Hide comment
@Gr33nDrag0n69

Gr33nDrag0n69 Jan 8, 2017

Member

i dont know cc001

Member

Gr33nDrag0n69 commented Jan 8, 2017

i dont know cc001

@viper-tkd

This comment has been minimized.

Show comment
Hide comment
@viper-tkd

viper-tkd Jan 8, 2017

Did anybody notice the word bad in the too long string: The string 8a6d629685b18e17e5f534065bad4984a8aa6b499c5783c3e65f61779e6da06czz

I don't know if it's a coincidence or if it was made like that to break the network.

viper-tkd commented Jan 8, 2017

Did anybody notice the word bad in the too long string: The string 8a6d629685b18e17e5f534065bad4984a8aa6b499c5783c3e65f61779e6da06czz

I don't know if it's a coincidence or if it was made like that to break the network.

@Odsejen

This comment has been minimized.

Show comment
Hide comment
@Odsejen

Odsejen Jan 8, 2017

just looked at the transactions from & to the the account "axlsaddek", one connected acount is the Lisk address "24213L" is this really a common address?

Odsejen commented Jan 8, 2017

just looked at the transactions from & to the the account "axlsaddek", one connected acount is the Lisk address "24213L" is this really a common address?

@Gr33nDrag0n69

This comment has been minimized.

Show comment
Hide comment
@Gr33nDrag0n69

Gr33nDrag0n69 Jan 8, 2017

Member

viper does the lenght is 3 chars too long ? thats odd

Member

Gr33nDrag0n69 commented Jan 8, 2017

viper does the lenght is 3 chars too long ? thats odd

@karmacoma karmacoma added the bug label Jan 8, 2017

@karmacoma karmacoma self-assigned this Jan 8, 2017

@viper-tkd

This comment has been minimized.

Show comment
Hide comment
@viper-tkd

viper-tkd Jan 8, 2017

@Gr33nDrag0n69 Dunno. I didn't check the length of the field in the database. But I suspect it is.

viper-tkd commented Jan 8, 2017

@Gr33nDrag0n69 Dunno. I didn't check the length of the field in the database. But I suspect it is.

@simonmorgenthaler

This comment has been minimized.

Show comment
Hide comment
@simonmorgenthaler

simonmorgenthaler Jan 8, 2017

@viper-tkd @Gr33nDrag0n69 the string is 66 chars long. 2 chars too long

simonmorgenthaler commented Jan 8, 2017

@viper-tkd @Gr33nDrag0n69 the string is 66 chars long. 2 chars too long

@vrlcrypt

This comment has been minimized.

Show comment
Hide comment
@vrlcrypt

vrlcrypt Jan 8, 2017

The bad transaction was made by @AxlSaddek.

Months ago he reported this issue (about Address length <16):
LiskHQ/lisk-explorer#9

screen shot 2017-01-08 at 1 55 20 pm

vrlcrypt commented Jan 8, 2017

The bad transaction was made by @AxlSaddek.

Months ago he reported this issue (about Address length <16):
LiskHQ/lisk-explorer#9

screen shot 2017-01-08 at 1 55 20 pm

@1towia

This comment has been minimized.

Show comment
Hide comment
@1towia

1towia Jan 8, 2017

There is some checks missing here I think :
https://github.com/LiskHQ/lisk/blob/master/logic/vote.js#L157

The code doesn't verify if the publicKeys in the transaction are valids.

1towia commented Jan 8, 2017

There is some checks missing here I think :
https://github.com/LiskHQ/lisk/blob/master/logic/vote.js#L157

The code doesn't verify if the publicKeys in the transaction are valids.

@simonmorgenthaler

This comment has been minimized.

Show comment
Hide comment
@simonmorgenthaler

simonmorgenthaler Jan 8, 2017

it just expired:
[inf] 2017-01-08 17:09:16 | Expired transaction: 14467342019343228805 received at: Sun, 08 Jan 2017 14:08:56 GMT

simonmorgenthaler commented Jan 8, 2017

it just expired:
[inf] 2017-01-08 17:09:16 | Expired transaction: 14467342019343228805 received at: Sun, 08 Jan 2017 14:08:56 GMT

@simonmorgenthaler

This comment has been minimized.

Show comment
Hide comment
@simonmorgenthaler

simonmorgenthaler Jan 8, 2017

after reload the nodes seems ok for a short time. I guess the transaction is still broadcasted
and make the nodes stuck again
The transaction is expired only on 2 of my 3 nodes

simonmorgenthaler commented Jan 8, 2017

after reload the nodes seems ok for a short time. I guess the transaction is still broadcasted
and make the nodes stuck again
The transaction is expired only on 2 of my 3 nodes

@simonmorgenthaler

This comment has been minimized.

Show comment
Hide comment
@simonmorgenthaler

simonmorgenthaler Jan 8, 2017

after reload:
OK, then these messages:
[ERR] 2017-01-08 17:27:33 | Failed to get height from peer: 213.136.81.40:8000
[ERR] 2017-01-08 17:27:33 | Failed to apply unconfirmed transaction: 8224227198097948535 - Failed to add vote, account has already voted for this delegate
[ERR] 2017-01-08 17:27:34 | Failed to apply unconfirmed transaction: 1219768254938248202 - Failed to add vote, account has already voted for this delegate
[ERR] 2017-01-08 17:27:34 | Failed to apply unconfirmed transaction: 12145996864416172549 - Account does not have enough LSK: 12492293221048511653L balance: 1
[ERR] 2017-01-08 17:27:34 | Failed to apply unconfirmed transaction: 989175740950637023 - Account does not have enough LSK: 17836753152065928804L balance: 493.2
[inf] 2017-01-08 17:27:44 | Starting sync

then stuck again

simonmorgenthaler commented Jan 8, 2017

after reload:
OK, then these messages:
[ERR] 2017-01-08 17:27:33 | Failed to get height from peer: 213.136.81.40:8000
[ERR] 2017-01-08 17:27:33 | Failed to apply unconfirmed transaction: 8224227198097948535 - Failed to add vote, account has already voted for this delegate
[ERR] 2017-01-08 17:27:34 | Failed to apply unconfirmed transaction: 1219768254938248202 - Failed to add vote, account has already voted for this delegate
[ERR] 2017-01-08 17:27:34 | Failed to apply unconfirmed transaction: 12145996864416172549 - Account does not have enough LSK: 12492293221048511653L balance: 1
[ERR] 2017-01-08 17:27:34 | Failed to apply unconfirmed transaction: 989175740950637023 - Account does not have enough LSK: 17836753152065928804L balance: 493.2
[inf] 2017-01-08 17:27:44 | Starting sync

then stuck again

@Odsejen

This comment has been minimized.

Show comment
Hide comment
@Odsejen

Odsejen Jan 8, 2017

Here the tx "14467342019343228805" is expired (appr. at 17:09 ) on all 4 nodes,

Odsejen commented Jan 8, 2017

Here the tx "14467342019343228805" is expired (appr. at 17:09 ) on all 4 nodes,

@Gr33nDrag0n69

This comment has been minimized.

Show comment
Hide comment
@Gr33nDrag0n69

Gr33nDrag0n69 Jan 8, 2017

Member

just retry a rebuild after the tx expired, no go

Member

Gr33nDrag0n69 commented Jan 8, 2017

just retry a rebuild after the tx expired, no go

@viper-tkd

This comment has been minimized.

Show comment
Hide comment
@viper-tkd

viper-tkd Jan 8, 2017

same here on all nodes

viper-tkd commented Jan 8, 2017

same here on all nodes

@Odsejen

This comment has been minimized.

Show comment
Hide comment
@Odsejen

Odsejen Jan 8, 2017

ok no go, just tried a rebuild with one of the previous nodes which showed the tx "x" is expired, now i have similar messages as @simonmorgenthaler mentioned (and as before stuck at xxx836), but no expiration messages anymore, but instead these "failed to apply" messages and broadhash consensus is/was in all cases 100%.

  • Failed to apply unconfirmed transaction: 17649204730636712354 - Failed to remove vote, account has not voted for this delegate
  • Failed to apply unconfirmed transaction: 8224227198097948535 - Failed to add vote, account has already voted for this delegate
  • Failed to apply unconfirmed transaction: 1219768254938248202 - Failed to add vote, account has already voted for this delegate

Odsejen commented Jan 8, 2017

ok no go, just tried a rebuild with one of the previous nodes which showed the tx "x" is expired, now i have similar messages as @simonmorgenthaler mentioned (and as before stuck at xxx836), but no expiration messages anymore, but instead these "failed to apply" messages and broadhash consensus is/was in all cases 100%.

  • Failed to apply unconfirmed transaction: 17649204730636712354 - Failed to remove vote, account has not voted for this delegate
  • Failed to apply unconfirmed transaction: 8224227198097948535 - Failed to add vote, account has already voted for this delegate
  • Failed to apply unconfirmed transaction: 1219768254938248202 - Failed to add vote, account has already voted for this delegate
@vrlcrypt

This comment has been minimized.

Show comment
Hide comment
@vrlcrypt

vrlcrypt commented Jan 8, 2017

Look:
#165

screen shot 2017-01-08 at 3 33 59 pm

@1towia

This comment has been minimized.

Show comment
Hide comment
@1towia

1towia Jan 8, 2017

@vrlc92 He didn't used the testnet, like last time. Not good @AxlSaddek.

1towia commented Jan 8, 2017

@vrlc92 He didn't used the testnet, like last time. Not good @AxlSaddek.

@karmacoma karmacoma closed this in b5cbd58 Jan 8, 2017

karmacoma added a commit that referenced this issue Jan 8, 2017

Closes #381. Verifying each vote in trs.asset.votes.
Checking vote type, format and length.

MaciejBaj added a commit to MaciejBaj/lisk that referenced this issue Jan 25, 2017

Closes #381. Verifying each vote in trs.asset.votes.
Checking vote type, format and length.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment