Index Tokens balances per Address

[igorffs]

Description

Stores a snapshot of an Addresses Tokens balances whenever it receives a TokenTransfer.

The Token balance info is present in the blockchain, so we'll need to interact with it throught the Token Smart Contract using the balanceOf function - which is present in ERC-20 and ERC-721 specifications.

Acceptance Criteria

• Token balances should be useful as logs. Every record are scoped by block so it's possible to check what's the balance for a address in a specific block.
• It must be possible to query all tokens balances for an Address
• It must be possible to query all holders and their current balance for an Token.
• The balance should be stored as Wei.

Questions

• What balance should we store when the balanceOf function is not implemented in the Token's contract?
  • For now, we're not going to index this particular balance
• Is it possible call a function for a contract taking in consideration a block number?
  • this might affect the requirement of keeping a log since we're probably going to consume the data from the blockchain assync
  • https://github.com/ethereum/wiki/wiki/JSON-RPC#eth_call QUANTITY argument
• Does it make sense to store the values already converted using the contract decimals?

Observations

Both #50 and #410 will need to show the balance per Token in the Address page and interacting with the blockchain on demand may not perform properly.

For example:

An Address could have transferred 140 different tokens, which mean that we'd need to use the balanceOf for all of them at the same time in the #50 feature.

Task Breakdown

• Support to block number in eth_call
• Fetch Token Balances from Token Transfers in the blockchain
• Develop Token Balances in Realtime
• Improve tests
  • Indexer.TokenBalances
  • Indexer.BlockFetcher
  • Indexer.Realtime
• Develop Token Balance Fetcher async

[gfreh]

@igorffs We've just had a call to discuss some aspects of #51 and we've come up with some points we might have on our radar, chiefly while investigating the best approach to retrieve users' token balances.

In the issue #51, a summary with the count of addresses that have balance > 0 for that token should have been developed. In order to have that, we'll need a way to find those addresses.

Also, we're going to start working on the Holders tab (see #516) soon, which basically lists all addresses with balance > 0 alongside their respective balances.

Example: https://kovan.etherscan.io/token/0xdab1c67232f92b7707f49c08047b96a4db7a9fc6#balances

[alexgaribay]

I think I'd be similar to how address balances are fetched now with a separate process. The difference would be that we'd have to execute a token's contract with the balanceOf function and also passing along a block number for the eth_call.

During token transfer indexing, we'd have to extract out the target address, token contract address, and block number and pass that information along to the process that can fetch token balance asynchronously.

[igorffs]

Thanks for the feedback @alexgaribay. ❤️

My first impression was that we should use Chain.Balance to store both POA balance and token balances due to its relation to Chain.Address and make use of the BalanceFetcher process for the same sake. Although @acravenho mentioned that the balances table is the largest table we have in our database and including token balances there could actually increase that situation.

During token transfer indexing, we'd have to extract out the target address, token contract address, and block number and pass that information along to the process that can fetch token balance asynchronously.

Your comment also made notice that we probably wouldn't need, at least, to use the same BalanceFetcher process because the events that requires creating a Chain.Balances and a AddressTokenBalance (I'll just call it this way, ok?):

• Chain.Balances dispatches BalanceFetcher after importing addresses in general (in catchup and internal_transaction fetcher).
• AddressTokenBalance indexing would only need to dispatch this after an TokenTransfer was created.

Conceptually using the same Balance entity to represent this token balance still makes sense to me, but this storage thing could be a issue. Do you folks have an opinion about that?

Also, considering we're storing the AddressTokensBalances in a different table, would say the same log behavior is a important requirement? cc @acravenho @alexgaribay

[acravenho]

Balances table:
core ~ 6.5m rows
sokol ~ 8.9m rows

[alexgaribay]

@igorffs It wouldn't happen in the same process. I was think of having a TokenBalanceFetcher process. The queue would only get populated when there's token transfers found during indexing.

As for schema, we can't leverage the balance table since we would need another column for the token's contract address. We have to do an index like:

create(
  unique_index(
    :token_balances,
    [:address_hash, :token_contract_address_hash, :block_number],
    name: :unfetched_balances,
    where: "value_fetched_at IS NULL"
  )
)

[igorffs]

@acravenho I've just updated the description, can you take a look at it? There are a few questions there that I have to confirm yet before sending to ready to dev. \o

[gfreh]

Tested it here and everything seems to be working fine! 👍