Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get account history starting at a block and ending in head #1820

Closed
guilhermelawless opened this issue Mar 12, 2019 · 15 comments

Comments

4 participants
@guilhermelawless
Copy link
Contributor

commented Mar 12, 2019

Reported by Natrium.
Currently it is cumbersome for integrators to build an incremental account history internally. Scenario:

  • Initially, request full account_history to cache it internally.
  • When updating the history, to prevent getting too much data unecessarily (some accounts have hundreds of thousands of blocks), one would expect to use account_history starting at the internally cached head block, and up to the new head block.

Currently I see no easy way of doing this, and it seems like a pretty important thing to have. successors doesn't let you start at a block, only end. If that option existed, one could use successors followed by blocks_info as an alternative.

Ideally, account_history would have an optional reverse and could also start at a block. However, the semantics of the returned values stop making sense in that RPC, namely, the returned field "previous" would have to be "next" instead.

What would be the best way to approach this? I don't mind coding it once it's agreed upon.


Also, maybe we could add (block) "height" to the blocks in account_history?

@renesq

This comment has been minimized.

Copy link

commented Mar 12, 2019

You're right, such a feature could come in handy for the account_history command. Altering the behavior of the existing "head" option with an additional "reverse" option might be more align with other RPC commands, but in real life scenarios, a "end" option (="until block with hash"="recent blocks but no farther than...") would make more sense I guess, as a reversed block sequence is not needed in normal UI.

In your earlier edit, you misunderstood the successors command. It will actually list blocks that happened in the more recent time, unless the "reverse" option is set to true. But that command will only list block hashes, so you'd have to do some extra steps to get those block contents (either by using block_info one-by-one which is slow and tedious, or much better by querying account_history accordingly - including the "head" option to avoid race conditions, and the "count" parameter that matches the response of the successors command, and the "raw" parameter because account_history potentially leaves out blocks otherwise).

But, there is another simple, straightforward workaround to gather only new blocks, yet it involves more round trips than being able to set an "end", and has a slight chance of race conditions you'd need to check for:

  • After your first sync, count the number of blocks you gathered.
  • Before the second sync, query the block_count via account_info (optionally, make sure that the frontier hash returned is still in local storage, in order to detect forks)
  • In the second sync, set count to "current block-count minus first sync block count"
    Keep in mind that this will only work with the "raw" option activated on account_history, because the default query will leave out change blocks and epoch blocks, so the numbers wouldn't add up in some cases. Unfortunately, the raw option will take up to 4x longer than the default query according to my research.
    Also keep in mind that this workaround will only work if no new block occurred between requesting the block_count and doing the account_history query - which should normally be the case because you're in full control of your own account and won't insert new blocks in the meantime. If you want to incorporate a solution for this potential flaw: either reduce the "count" limiter on "account_history" by 1 so you can cross-check if you already got the last block (if not, try to sync again), or re-query the block_count after querying account_history. But this race condition should be rather rare.

I actually prefer the simple way to "sync": Don't cache anything locally, simply request the 200 most recent blocks of the Tx history and cut off after that. This way, it's quick, saves round-trips, and you'll never have to deal with forks and stuff. If a user wants to scroll farther back, they can either use a fully fledged block explorer, or one could implement pagination via the "head" option of the "account_history" command.
IIRC, canoe cached the Tx history locally, and manually inserted newly generated blocks in there before they were confirmed by reps, but this apparently caused some issues and race conditions, so users often had to use the "repair" feature.

Appendix:
A "raw" account_history block for reference:

            "type": "state",
            "representative": "xrb_3jwrszth46rk1mu7rmb4rhm54us8yg1gw3ipodftqtikf5yqdyr7471nsg1k",
            "link": "FD439DDD7940E90A2CA4B7AFE6743932008C9638EA321F0E5669E08F5DF54E84",
            "balance": "4924131438934728148868649242841566583",
            "previous": "A9775AC157DD91D72A27F6925940CC7278DEF3727D7905BB9CF6D309BD93CBB9",
            "subtype": "send",
            "account": "xrb_3zc5mqgqki9b3apcbfxhwst5kei1jkd5jtjk5w97eth1jxgzcmn6r95bqmg9",
            "amount": "349092750000000000000000000000000",
            "local_timestamp": "1552403461",
            "hash": "32E73F47005A00D2DF5672A67994EE0903BC9DCA2B262D4B7651B7BF0173D085",
            "work": "616a9c1a997fdea6",
            "signature": "E6436E90DDAEFFC468D8EA04AD26F52C9AD984E8BA33F7DCCD023AA081A412B7EE456F945474A6ABAF371C4F4165ED07EECEC79C7D080655A113C0CDFC2CFF0B"
        }

vs. the standard:

            "type": "receive",
            "account": "xrb_3btyufq6jnr4mx3eemfh965rf5381y8wmjyrytxcoyk5ehnu9t5n3qinr1dd",
            "amount": "393919542000000000000000000000000",
            "local_timestamp": "1552405206",
            "hash": "203A9E0F9D02D3647E6C697780AE6A2727BEA07CFCB4F5705CA6A6B33260EE18"
        },
        {
            "type": "receive",
            "account": "xrb_3o1texi7gy48cq9895swdkoanecati674n4drmae1s5gpbgw7jis7id1spoa",
            "amount": "15429540139990000000000000000000000",
            "local_timestamp": "1552405205",
            "hash": "9AFC408D37DD6CA2BCA61A158C9CCCC0E0C21F7A1CBE717D7061578767EF5BD0"
        },
        {
            "type": "send",
            "account": "xrb_3zc5mqgqki9b3apcbfxhwst5kei1jkd5jtjk5w97eth1jxgzcmn6r95bqmg9",
            "amount": "349092750000000000000000000000000",
            "local_timestamp": "1552403461",
            "hash": "32E73F47005A00D2DF5672A67994EE0903BC9DCA2B262D4B7651B7BF0173D085"
        },

Also the successors query for the same three blocks, but from the bottom up:

>   "block": "32E73F47005A00D2DF5672A67994EE0903BC9DCA2B262D4B7651B7BF0173D085",
>   "count": "3"
> }' '[::1]:7076'
{
    "blocks": [
        "32E73F47005A00D2DF5672A67994EE0903BC9DCA2B262D4B7651B7BF0173D085",
        "9AFC408D37DD6CA2BCA61A158C9CCCC0E0C21F7A1CBE717D7061578767EF5BD0",
        "203A9E0F9D02D3647E6C697780AE6A2727BEA07CFCB4F5705CA6A6B33260EE18"
    ]
@guilhermelawless

This comment has been minimized.

Copy link
Contributor Author

commented Mar 12, 2019

Thank you for digging into this. I was indeed wrong about successors so it does offer an alternative solution.

Some takeaways then:

  • account_history documentation should reflect that only send and receive blocks are returned if raw is false. It is not clear at all.
  • In my opinion, successors documentation should say "Returns a list of block hashes in the account chain endingstarting at block up to count"
  • account_history needs some enhancements, namely "reverse" and perhaps "height" of each block
@SergiySW

This comment has been minimized.

Copy link
Collaborator

commented Mar 12, 2019

@guilhermelawless Expand description

Change blocks are skipped, open blocks will appear as receive (unless raw is set to true - see optional parameters below).

successors definition depends on other descriptions especially on chain & account_history. Default order for chain there is from newest blocks to older. "Returns a list of block hashes in the account chain starting at block up to count" is exactly "chain" description while it should show difference. More clear description may be required, right

Both height & reverse should be easy to impement. Why not

@guilhermelawless

This comment has been minimized.

Copy link
Contributor Author

commented Mar 12, 2019

@SergiySW Sorry, missed that part of the description.

You can leave height & reverse to me.

@renesq

This comment has been minimized.

Copy link

commented Mar 12, 2019

Guilherme is right with his proposal for the successors description. chain works differently than successors, the latter essentially is 'going forward' from the given hash's standpoint and puts the given hash on top, so it should be rephrased from "ending" to "starting". Ultimately, real life examples will help understanding the mechanism.

When I wrote this

Change blocks are skipped, open blocks will appear as receive (unless raw is set to true - see optional parameters below).

state blocks and epoch blocks didn't exist yet. Their role may be worth mentioning.

You can also add the undocumented "offset" parameter for account_history to the wiki page

@SergiySW

This comment has been minimized.

Copy link
Collaborator

commented Mar 12, 2019

@guilhermelawless if you want to make it :)

@guilhermelawless

This comment has been minimized.

Copy link
Contributor Author

commented Mar 12, 2019

I looked through the documentation and the only mention of "starting" and "ending" related to blocks in an account is in chain and successors, so you could just switch the meanings there:

  • Successors: starts at account
  • Chain: ends at account

I know in the code/protocol it works the other way but for whoever is reading the RPC documentation it makes more sense as I wrote, I think (and we all seem to agree).

@zhyatt zhyatt added this to the V19.0 milestone Mar 13, 2019

@guilhermelawless

This comment has been minimized.

Copy link
Contributor Author

commented Mar 13, 2019

I wonder if #1355 will be merged soon so I can use the core_test from there.

@SergiySW SergiySW added this to CP2 (2019-03-27) in V19 Mar 14, 2019

@SergiySW SergiySW self-assigned this Mar 14, 2019

@SergiySW

This comment has been minimized.

Copy link
Collaborator

commented Mar 18, 2019

@guilhermelawless @renesq chain, successors & account_history definitions in wiki slightly updated for clarification

@SergiySW

This comment has been minimized.

Copy link
Collaborator

commented Mar 18, 2019

Closed automatically, reopening

@SergiySW SergiySW reopened this Mar 18, 2019

@guilhermelawless

This comment has been minimized.

Copy link
Contributor Author

commented Mar 18, 2019

What is missing? @SergiySW

@SergiySW

This comment has been minimized.

Copy link
Collaborator

commented Mar 18, 2019

Wiki check with fresh eyes

@guilhermelawless

This comment has been minimized.

Copy link
Contributor Author

commented Mar 19, 2019

I think it might be worth repeating Will list all blocks back to the open block of this chain when count is set to "-1". The requested block hash is included in the answer. in successors.

For account_history, offset was available before, just not documented. Since v11 it's there: https://github.com/nanocurrency/nano-node/blob/release_v11/rai/node/rpc.cpp#L1747

@SergiySW

This comment has been minimized.

Copy link
Collaborator

commented Mar 19, 2019

@zhyatt

This comment has been minimized.

Copy link
Collaborator

commented Mar 19, 2019

Reviewed wiki updates and they appear accurate.

@zhyatt zhyatt closed this Mar 19, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.