Differing asset balances on various nodes

[Cryptomaniac2]

Summary

After executing a bunch of transactions via qt wallet on windows to transfer CPS, the balance on my aliases differs across nodes. Most transactions were executed with a witness (See transaction list at the end).
Right now it is impossible for me to see which transactions succeeded and on which alias some of my CPS coins are right now. For one alias I am even unable to create further asset transactions via qt console.

Pictures with balances of some aliases (Taken at block 169930 +- 10)

shows balances as seen on qt console on my windows machine. This does not change after reindex or deleting whole chain with downloading from scratch.

shows balances seen on 7 of my masternodes.

shows balances seen on 2 of my masternodes.

shows balances seen on 1 of my masternodes.

Masternodes are on 3.1.0 and running for months with same chain data. Updated and restarted when required. Just the 1 node in the last picture got its blockchain wiped after all this happened to see a freshly downloaded state.

Balance for talavin should be 159440.64207604
Balance for evelon should be 0.00000348 as tx 13 failed according to coinpayments
Balance for kuruna should be 60307.21105612 when tx 13 failed

Transactions that caused this

Problem started for me with a failed transaction (Number 13 below) to coinpayments. This transaction never arrived at coinpayments.
The following transactions (14 to 23) went fine and arrived. Difference is that I always left a fraction of coins on each alias.
More problems started to show after transaction 23 when I tried to sent from talavin. This always resulted in this error regardless of amount to send:
SYSCOIN_ALIAS_CONSENSUS_ERROR: ERRCODE: 5003 - Witness signature not found (code 16) (code -1)

I tried sending some coins to talavin (tx 24) to see if it changes anything. Coins arrived but error stayed the same.
After tx 24 I contacted coinpayments support regarding failed tx 13 and according to them it failed on the blockchain.
As the balance for evelon shows 0.00000000 (Windows qt wallet) on my client, I tried to recover them by sending a transaction from a previous point of the chain.
Tx 25 was sent from a too early point (client at block 164400) and should obviously fail.
Tx was then sent again (client at block 164430) which is visible as successful on some nodes but not all.

Transactions

Most commands were executed with a witness because the first transactions for interest did not work without one.
Command usage differs from their help text because they did not work exactly as described (Already reported in issue #259). First alias acted as witness and second got interest credited.

Executed commands on qt 3.0.6.2:
assetallocationcollectinterest 777845ced7b6022b talavin ALIAS (for 10 aliases)
assetallocationcollectinterest 777845ced7b6022b evelon talavin (once)

Tx #	Block #	Tx Id
1	164376	77c31512ed9ddc00f73fce92b0311d0416dd318761395aa3d32b477e9cbe7ebc
2	164379	eb196f981787509d7074b1aa4a31a7a3f3f65195080cbb833e031527fa0233ff
3	164382	23607fb851fce9513f7e6989ae7299eebbc18f15f33982e8abeb14c91a4a17fe
4	164382	bb08ba1fec838c325da9210e39e008420dd4c4e036c33fef8d32972a293d3527
5	164384	52f08e5701383b050a02b666ed48058c1f518e71920d29f0509cb4f1fde38c90
6	164385	f17f3062373f942bb8311b133a3afa2b736752427d97d705516d6e6185dc79f8
7	164385	00783c266f2a20949cadcaf6064c88fb0e05e5716cbaab237bcd68107ae580e5
8	164385	95f5f42b6a3db0165517600943e4b6b25366dc70073495e7956fea016f661f4b
9	164386	1917bd446ffe9a97fc33a0bd9f5d56192a39a4f34d39e7f3ccd8a38a4a2f9fda
10	164388	e56e7b321def5e28108c71752939a2e3c486671a7fadf01f64a912b8d275cd0b
11	164392	c26780559c5424630c0038eed82dcf7ab152059c0662d2bf6f148babd35c9d15

How assetallocationsend worked was that talavin acted as the witness and coins were transferred from INPUT
Executed commands on qt 3.0.6.2:
assetallocationsend 777845ced7b6022b talavin '[{"aliasto":"coinpayments","amount":AMOUNT}]' MEMO INPUT

Tx #	Block #	Tx Id	INPUT	OUTPUT	Amount	Comment
12	164429	c099f6e13b3191e8e4035fb1519dc10f9a955d5e74c31aed6c523ea1357cdbb2	evelon	coinpayments	1 CPS
13	164438	2ff3212bd406a17eae65c3a35694cfea4b04ba751109ee5369d17a50e104c21f	evelon	coinpayments	60347.04000348 CPS	apparently failed
14	164458	d047538cbead1a39765bd73dc4d15bd29154bae0e3d6cfcbc1978e80d50551e7	aranea	coinpayments	60276 CPS
15	164461	1be6b7aa8502240661fa55f1f3e5f284273e0d81d16e1e3b9432429664bcb3ac	kuruna	coinpayments	60306 CPS
16	164464	89840fc54f35d1e542ed71e414c08a9bf46a8d07d1ee11fa6f083b609a4fcf59	zandorin	coinpayments	60216 CPS
17	164464	5cb10bd57b56f62794e90b929e0e07287b9ce0409fd756dea3bf5948398539d4	uldin	coinpayments	60351 CPS
18	164466	b33d5095df7e86dff242a5e3a5836a6aab0ccfb07cba9c3d1957483cdee3dad2	sakronis	coinpayments	60276 CPS
19	164467	a1f5eb56d9d725476a1e36400e6c4ef2e095a3bb44691a68556fe17083a8f30d	talavin	coinpayments	60348 CPS
20	164469	6688aa983a7391a17e8b6f1798df90e9add85bb30680f48de45c54f83699eef3	iridin	coinpayments	60333 CPS
21	164470	8047711dea86e9adc128d56539cea5d6326c92e11536f3b4f173f4b236e196ed	gwendolin	coinpayments	60367 CPS
22	164477	e0fffa5efff2edd687777089cfd847c35f635e2eca2c458156137979715d27bb	wesnari	coinpayments	60348 CPS
23	164477	90acc54ec6301445ab89fcb8abccde8c0fa9d400df56df22cc342d5825afeca2	bismur	coinpayments	60333 CPS
24	164526	2a9d832ef3c3ee52ffcd4e63a6692e486858efcd43446da29ddb0922103d260c	?unsure?	talavin	0.7 CPS

This time commands were executed without witness as I noticed that it was not required in case of assetallocationsend
Executed commands on qt 3.1.0:
assetallocationsend 777845ced7b6022b evelon '[{"ownerto":OUTPUT,"amount":AMOUNT}]' "" ""

Tx #	Block #	Tx Id	INPUT	OUTPUT	Amount	Comment
25	167108	2591074f2a43e2ca39a91872f8358c13676f844a663eafe76e79cd82e0be7e1c	evelon	coinpayments	60348 CPS	should fail
26	169448	6c90593f8dda28bae44101776b2db367dec4ff0a5162e6db1c4b40b747e673ef	evelon	kuruna	60347.04 CPS

[sidhujag]

This is what I see on my mac running syscoind:

Jags-MacBook-Pro:src jsidhu$ ./syscoin-cli assetallocationinfo 777845ced7b6022b talavin true
{
"_id": "777845ced7b6022b-talavin",
"asset": "777845ced7b6022b",
"symbol": "CPS",
"interest_rate": 0.25,
"txid": "2a9d832ef3c3ee52ffcd4e63a6692e486858efcd43446da29ddb0922103d260c",
"height": 164526,
"owner": "talavin",
"balance": "159440.64207604",
"interest_claim_height": 164386,
"memo": "D5c996af713de8197",
"inputs": [
],
"accumulated_interest": "0.00000000"
}
Jags-MacBook-Pro:src jsidhu$ ./syscoin-cli assetallocationinfo 777845ced7b6022b evelon true
{
"_id": "777845ced7b6022b-evelon",
"asset": "777845ced7b6022b",
"symbol": "CPS",
"interest_rate": 0.25,
"txid": "6c90593f8dda28bae44101776b2db367dec4ff0a5162e6db1c4b40b747e673ef",
"height": 169448,
"owner": "evelon",
"balance": "0.00000338",
"interest_claim_height": 164376,
"memo": "",
"inputs": [
],
"accumulated_interest": "0.00000000"
}
Jags-MacBook-Pro:src jsidhu$ ./syscoin-cli assetallocationinfo 777845ced7b6022b kuruna true
{
"_id": "777845ced7b6022b-kuruna",
"asset": "777845ced7b6022b",
"symbol": "CPS",
"interest_rate": 0.25,
"txid": "6c90593f8dda28bae44101776b2db367dec4ff0a5162e6db1c4b40b747e673ef",
"height": 169448,
"owner": "kuruna",
"balance": "60347.64552806",
"interest_claim_height": 164382,
"memo": "",
"inputs": [
],
"accumulated_interest": "0.00000000"
}

[sidhujag]

So if you had tried to collect interest and then send total amount from an alias to another then I think what happened was because during interest calculations a rounding error was introduced because of overflow (I've since fixed this in the 175k 3.1 fork) and then since nodes then have differing balances when you try to send the total some nodes think the balance is insufficient so it just does a NO-OP and doesn't update anything while others will go through and add to receiver and remove from sender balance. On my unit test I do extensive tests as can be seen here https://github.com/syscoin/syscoin/blob/dev-3.x-prep-3.2.0/src/test/syscoin_asset_tests.cpp#L531 and the 3 nodes all match up in balances exactly.

So what I'd like to do is step through each transaction with you and get what the balance updates to each step to see where they start to differ.

[Cryptomaniac2]

Explanation and your values look plausible.

Right now it is getting late for me. I am usually available from 16 - 22 GMT. We can step through tomorrow if that fits for you or any other day this week if that is better,

[Cryptomaniac2]

Interest collection is the place where the difference to your values happens. Alias evelon and kuruna show 10 sat more for me, talavin shows 39 more

interest collections:

transactions evelon:

transactions talavin:

[sidhujag]

Can you try https://github.com/syscoin/syscoin/tree/dev-3.x you should see consistent numbers after reindexing

I get this with assetinfo

Jags-MacBook-Pro:src jsidhu$ ./syscoin-cli assetinfo 777845ced7b6022b true
{
"_id": "777845ced7b6022b",
"symbol": "CPS",
"txid": "eb2e849a57a9306dd43bd8e741b3b65740c398d3f82d3ebe78778cd01cd40cdf",
"height": 83639,
"time": 1530398341,
"publicvalue": "CoinPayments Coin",
"category": "assets",
"owner": "cpscoin",
"balance": "0.00000000",
"total_supply": "2000177157.41598035",
"max_supply": "9999999999.00000000",
"interest_rate": 0.25,
"can_adjust_interest_rate": true,
"use_input_ranges": false,
"precision": 8,
"inputs": [
]
}
note "total_supply": "2000177157.41598035" that has everyones interest applied.. do you get this number after rebuild+reindex?

[Cryptomaniac2]

Have run the dev-3.x version on a Linux vm. I currently dont have the requirements for a windows build.
Result is matching yours:
assetinfo 777845ced7b6022b true
{
"_id": "777845ced7b6022b",
"symbol": "CPS",
"txid": "eb2e849a57a9306dd43bd8e741b3b65740c398d3f82d3ebe78778cd01cd40cdf",
"height": 83639,
"time": 1530398341,
"publicvalue": "CoinPayments Coin",
"category": "assets",
"owner": "cpscoin",
"balance": "0.00000000",
"total_supply": "2000177157.41598035",
"max_supply": "9999999999.00000000",
"interest_rate": 0.25,
"can_adjust_interest_rate": true,
"use_input_ranges": false,
"precision": 8,
"inputs": [
]
}

Creating a transaction from talavin also works again.
Shown balances of the 3 aliases have slightly increased compared to my first screenshot from windows console.

[sidhujag]

ok I would wait though to sent any of these anywhere... so mac/linux look like they match windows dont match still so i added a buffer.. i've released this 3.1.1 to fix transfer of assets when the balances are off by tiny amounts (1000 sats or less). Once cps exchange is updates you can send/recv them safely without worrying.

[sidhujag]

Pls update to 3.1.2 and resync, once CPS is on that and resyncs your balances and transactions with them will also be fine. Pls open issue again if something else comes up. Thanks for the investigation!