New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Poor performance of GET /api/delegates/[address]/forging_statistics #2352

Closed
nazarhussain opened this Issue Aug 29, 2018 · 1 comment

Comments

Projects
3 participants
@nazarhussain
Contributor

nazarhussain commented Aug 29, 2018

Expected behavior

Performance of API GET /api/delegates/[address]/forging_statistics should be satisfactory (under 1 second).

Actual behavior

We run some benchmarks with following spec:

CPU: 1vCPU (Digital Ocean) 
RAM: 1 GB Memory
Disk: 25 GB Disk
Connections: 10 
Time: 30s

And got the following results;

Fastest: 8.23
Slowest: 8.91
Average: 9.74

Steps to reproduce

Query endpoints mentioned above.

Which version(s) does this affect? (Environment, OS, etc...)

1.0.1 (after indexes)

@4miners

This comment has been minimized.

Show comment
Hide comment
@4miners

4miners Aug 30, 2018

Member

The main reason for introducing this endpoint (together with new rounds_rewards table) was to allow getting forging statistics for the period of time, for example, last month, last week, etc. This endpoint is not a reliable one for getting total forging statistics (from all time), as the underlying query is performing aggregate function SUM on two columns over large data set (~6M records and growing). Please also see my comment here: LiskHQ/lisk-private#73 (comment)

I would recommend taking the following approach for now:

When a user (not providing fromTimestamp or fromTimestamp provided is equal to Lisk epoch) and (toTimestamp is also not provided or is greater or equal to last block timestamp) - this means that user requested statistics for all time

var filters = {
address: params.address.value,
start: params.fromTimestamp.value || constants.epochTime.getTime(),
end: params.toTimestamp.value || Date.now(),

For this we can just query mem_accounts and get columns rewards and fees, the results for a particular delegate are the same:
mem_accounts:

    rewards     |     fees     
----------------+--------------
 24128500000000 | 284136733031
(1 row)

rounds_rewards (aggregate_blocks_reward):

 delegate | count |     fees     |    rewards     
----------+-------+--------------+----------------
        1 | 53107 | 284136733031 | 24128500000000
(1 row)

Please review the approach, @nazarhussain.

Member

4miners commented Aug 30, 2018

The main reason for introducing this endpoint (together with new rounds_rewards table) was to allow getting forging statistics for the period of time, for example, last month, last week, etc. This endpoint is not a reliable one for getting total forging statistics (from all time), as the underlying query is performing aggregate function SUM on two columns over large data set (~6M records and growing). Please also see my comment here: LiskHQ/lisk-private#73 (comment)

I would recommend taking the following approach for now:

When a user (not providing fromTimestamp or fromTimestamp provided is equal to Lisk epoch) and (toTimestamp is also not provided or is greater or equal to last block timestamp) - this means that user requested statistics for all time

var filters = {
address: params.address.value,
start: params.fromTimestamp.value || constants.epochTime.getTime(),
end: params.toTimestamp.value || Date.now(),

For this we can just query mem_accounts and get columns rewards and fees, the results for a particular delegate are the same:
mem_accounts:

    rewards     |     fees     
----------------+--------------
 24128500000000 | 284136733031
(1 row)

rounds_rewards (aggregate_blocks_reward):

 delegate | count |     fees     |    rewards     
----------+-------+--------------+----------------
        1 | 53107 | 284136733031 | 24128500000000
(1 row)

Please review the approach, @nazarhussain.

@nazarhussain nazarhussain self-assigned this Aug 30, 2018

@MaciejBaj MaciejBaj added this to Open Issues in Version 1.0.0 via automation Aug 30, 2018

@MaciejBaj MaciejBaj added this to the Version 1.0.2 milestone Aug 30, 2018

@MaciejBaj MaciejBaj removed this from the Version 1.0.2 milestone Aug 30, 2018

@MaciejBaj MaciejBaj added this to New Issues in Version 1.1.0 via automation Aug 30, 2018

@MaciejBaj MaciejBaj removed this from Open Issues in Version 1.0.0 Aug 30, 2018

@MaciejBaj MaciejBaj added this to the Version 1.1.0 milestone Aug 30, 2018

nazarhussain added a commit that referenced this issue Aug 31, 2018

Merge pull request #2360 from LiskHQ/2352-froging-stats
Poor performance of GET /api/delegates/[address]/forging_statistics - Closes #2352

Version 1.1.0 automation moved this from New Issues to Closed Issues Aug 31, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment