Token balances not loading (getBalance HRC20 calls), leading to DEXs like Viper, Mochi unusable

[gupadhyaya]

Currently there is an issue going on related to explorer nodes lagging due to increase in the number of transactions per block (more info here), and it is understandable that transaction confirmations could get impacted due to this, however, currently even the normal EVM calls like getBalance of a HRC20 token is slow and most times does not work, which is rendering DEXs like viper, mochi, almost unusable. Feels like this is more to do with rate limit of api calls (this commit added the rate limit). The rate limit of 300 per second seems low, especially during the high load times. Note that the current spamming is adding 50-100 transactions every block, which also have several evm calls. Further, any DEX operation have several getBalance call for every token that any user has.

DEX like viper shows no balance or liquidity when it is unable to fetch them from the contracts (as shown below):

[LeoHChen]

let me check if the rate limiter had reached the upper bound in the explorer nodes.

[LeoHChen]

unfortunately, the PR to add rate limit is not merged yet, #3743

so I will have to do the following to mitigate the current issue.

1. add a customized build to check the rate limiter
2. increase the rate limit to 1000 by now

[gupadhyaya]

Right now I see the explorer is insync. There are small windows where the explorer is insync and also see the DEX able to load balances and not have any issue. Any rationale for this @LeoHChen ?

Viper mining loads fine:

[LeoHChen]

I have to restart the explorer nodes, they were out of sync mostly due to #3740

[gupadhyaya]

I have to restart the explorer nodes, they were out of sync mostly due to #3740

is regularly restarting explorer nodes a temporary solution until #3740 is resolved?

[LeoHChen]

nodes are out of sync very fast, need to work on some automation solution to detect and restart harmony process as a mitigation right now

[LeoHChen]

The problem is still the nodes out of sync issue. I have added a solution to detect the node out-of-sync, and to restart the nodes. The cronjob will be checking/run every 15 minutes as temporary mitigation. Closing this ticket, for now, #3740 is still the root cause.