Our current implementation is great in terms of space consumption and it's pretty fast to get needed deltas, but for some accounts it might be needed to get enormous number of deltas in order to sync to the latest state. Obvious solution is to limit number of deltas processed by one single call to the endpoint. For cases, when deltas number is too hight it might be better to just request account's full state.
I see two solutions for it:
- Limit maximum number of deltas processed in current solution. This will require adding block number to the response in order for the client to be aware that not all requested deltas were returned and make next requests starting from the last returned block number until it get all needed deltas. Another solution is to just return an error, if number of deltas is too high and client will request account's full latest state.
- Limit maximum number of account changes requested (by nonce). Return an error, if number of nonces (
to_nonce - from_nonce) requested is higher than the limit. In this case client should request account's full latest state.
From @bobbinth:
I'm leaning toward the first approach. In general, I think we could introduce "epoch-based pagination" I described in #404 (comment) to a couple of endpoints and this would be one of them. We should also "stress-test" at some point to see what happens if the response is too big. The default Tonic message size is 4MB. Would be good to find out what happens if we exceed this limit (i.e., what will break and what kind of an error message we'll get).
Our current implementation is great in terms of space consumption and it's pretty fast to get needed deltas, but for some accounts it might be needed to get enormous number of deltas in order to sync to the latest state. Obvious solution is to limit number of deltas processed by one single call to the endpoint. For cases, when deltas number is too hight it might be better to just request account's full state.
I see two solutions for it:
to_nonce - from_nonce) requested is higher than the limit. In this case client should request account's full latest state.From @bobbinth:
I'm leaning toward the first approach. In general, I think we could introduce "epoch-based pagination" I described in #404 (comment) to a couple of endpoints and this would be one of them. We should also "stress-test" at some point to see what happens if the response is too big. The default Tonic message size is 4MB. Would be good to find out what happens if we exceed this limit (i.e., what will break and what kind of an error message we'll get).