Unclaimed gas calculation incorrect

[brianlenz]

Current behavior

• Are you using the latest available version?
  Yes, running latest development branch.
• What is the problem?
  There is one transaction for which gas is unclaimable.

Expected behavior

• What should be happening?
  Gas claim works.

How to reproduce

• Please explain the steps to reproduce the problem.
  Trying to claim gas for the address AZXKdEc2M73qeQMmPj2zmKJFDWmzQf8Gyr. There is currently only one transaction with unclaimed gas: 5a6fd5a783fb1673bc8d7aebb981e6d96c64da1a0f0448e47c65f23e367253f3

neo-python thinks the total GAS to claim is 4.70274012. Neoscan and Neotracker, on the other hand, both report the total GAS to claim as 4.84828396:

https://neoscan.io/address/AZXKdEc2M73qeQMmPj2zmKJFDWmzQf8Gyr
https://neotracker.io/address/AZXKdEc2M73qeQMmPj2zmKJFDWmzQf8Gyr

My suspicion is that something in neo-python's calculation is wrong.

When you try to claim GAS for this transaction, here is the claim transaction that is created:

claim tx: {
    "txid": "0x7c60e83ee9f52e0d788e28675911f2e1e8e2e3554e3ee64007205b7b12e5b92f",
    "size": 292,
    "type": "ClaimTransaction",
    "version": 0,
    "attributes": [
        {
            "usage": 32,
            "data": "3f1d5432d67d24228574206849b1d7a7a96d632b"
        }
    ],
    "vout": [
        {
            "n": 0,
            "asset": "0x602c79718b16e442de58778e148d0b1084e3b2dffd5de6b7b16cee7969282de7",
            "value": "4.70274012",
            "address": "AZXKdEc2M73qeQMmPj2zmKJFDWmzQf8Gyr"
        }
    ],
    "vin": [],
    "sys_fee": "0",
    "net_fee": "0",
    "scripts": [
        {
            "invocation": "40caa705bf7f7d17ec6e70a7e66332f7cde7cd2b26d8f4024474d5263156cd752e084fdb8feadb656eb33b74601065d5009fb155a6eaf3a744ec73384218d7047b",
            "verification": "21024ab2c861a603070407c119a692adb697a4e99a66494a307534a0097549fa7d70ac"
        },
        {
            "invocation": "00407a517831ab0c50a697ae119b863feb867e7d04a6b945c475de85718c1d19067913eea1aa58d16317f03fccd92fa40a84d6d31f697e3423c69441a7741aba15ea",
            "verification": ""
        }
    ],
    "claims": [
        {
            "txid": "0x5a6fd5a783fb1673bc8d7aebb981e6d96c64da1a0f0448e47c65f23e367253f3",
            "vout": 0
        }
    ]
}

This transaction gets relayed, but it's never actually processed, presumably because it's not valid.

Note that there were 12 other transactions for which I was able to successfully claim gas (by modifying neo-python to process one claim at a time), so it appears there is something special about the gas calculation for this specific transaction that is causing the problem.

Note also that this is a smart contract address, so I don't know if that has anything to do with it.

I'm happy to help debug/troubleshoot if it's helpful.

Your environment

Let us know in what environment you're running into the issue:

• OS: macOS 10.15.2
• neo-python version: v0.9.2-dev
• Python version: 3.7.4

[ixje]

I think failure would make sense if the total gas to claim suggested by neo-python is bigger than what the others think, but smaller shouldn't fail. A wrong total value is still an issue of course.

I have to let this one sink in for a moment. I've been working on neo3 for a while now that I have to refresh my mind on the 2.x code base and UTXO model.

[ixje]

I think it will be hard for me to debug anything given that it is a claim on your account. My first idea is to compare the unclaimed returned here

neo-python/neo/Core/Blockchain.py

Line 441 in f343df6

def CalculateBonusIgnoreClaimed(inputs, ignore_claimed=True):

with what the C# client would say here (make sure to not accidentally relay the transaction on the C# side)
https://github.com/neo-project/neo/blob/54d0f29c82fc38b4af7d7d2df05e4e564e7966dd/neo/Persistence/Snapshot.cs#L35

Comparing start height, end height and output value. If there is no difference there, then it must be somewhere here

neo-python/neo/Core/Blockchain.py

Line 487 in f343df6

def CalculateBonusInternal(unclaimed):

[brianlenz]

OK, I have tracked the issue down to GetSysFeeAmountByHeight. The end block height for this transaction is 5028923. I checked the expected fee via JSON-RPC, and it's supposed to be 307291. The value being returned by neo-python is 27399.

When I manually run the rest of the GAS calculation with the proper/expected value, it does result in the proper GAS amount, so this is definitely the issue.

I'm going to see if I can get a new chain DB to see if it's just my local DB being out-of-whack that's the issue.

[brianlenz]

I checked a different fully sync'd neo-python chain, and it has the same issue.

@ixje, do you have a neo-python 2.0 chain you can test against? I'm just using the RPC server to quickly test:

np-api-server --mainnet --port-rpc 8099

With the JSON-RPC call:

curl -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","method":"getblocksysfee","params":[5028923],"id":1234}' http://localhost:8099

Expecting a value of 307291 but getting a value of 27399.

[ixje]

@ixje, do you have a neo-python 2.0 chain you can test against?

I unfortunately have to say no. I used to have a VPS running the latest version for audit reasons but that seems to have died and I've not been able to recover it (yet). The fastest option I can think of is taking a neo-python bootstrap 4117000 and the latest available chain.acc from NGD and using np-import to sync back up (something like np-import -m -a -i chain.acc --datadir <root folder containing "Chains/Main/">).

iirc the neo-python bootstrap was made with 0.9.1. That should be the fastest route to checking if it's a local chain issue or not.

[brianlenz]

It worked! 🎉 I did a full re-sync, starting from @ixje's bootstrap, syncing with chain.acc, and then syncing the last 200k blocks in np-prompt (which took over a week in total 😳). The gas calculations that resulted from the full rebuild were correct and solved the issue.

Thanks, @ixje! Glad to hear neo-python had it right; once I saw your confidence that it should work, I was very hopeful that's all it would be 👍