Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cli_wallet get_relative_account_history result incorrect when limit > 100 and start is too big #298

Closed
abitmore opened this issue Jun 2, 2017 · 6 comments

Comments

@abitmore
Copy link
Member

abitmore commented Jun 2, 2017

get_relative_account_history command / API in CLI returns incorrect result when limit is more than 100 and start is too big.

For example, the last record of get_relative_account_history openledger 0 101 12345678 command is same to the first record, although the first 100 records are correct.

>>> get_relative_account_history openledger 0 101 12345678
2017-06-02T17:53:18 Create Account 'tximiss0'   (Fee: 1.22648 BTS)
...
2017-06-02T11:43:24 Create Account 'm1980'   (Fee: 1.22624 BTS)
2017-06-02T17:53:18 Create Account 'tximiss0'   (Fee: 1.22648 BTS)
oxarbitrage added a commit to oxarbitrage/bitshares-core that referenced this issue Jun 8, 2017
@oxarbitrage
Copy link
Member

we should not allow a limit greater than 100, pull: #305 fix it.

@oxarbitrage
Copy link
Member

sample with patch:

new >>> get_relative_account_history openledger 0 101 12345678
get_relative_account_history openledger 0 101 12345678
10 assert_exception: Assert Exception
start > 0 && limit <= 100: 
    {}
    th_a  wallet.cpp:2841 get_relative_account_history
new >>> get_relative_account_history openledger 0 100 12345678
get_relative_account_history openledger 0 100 12345678
2017-06-08T12:59:12 Create Account 'rcr1986'   (Fee: 1.22640 BTS) 
2017-06-08T12:55:18 Create Account 'm-marz'   (Fee: 1.22632 BTS) 
2017-06-08T12:54:51 Create Account 'hsngs14'   (Fee: 1.22640 BTS) 
2017-06-08T12:54:48 Create Account 'couba-123'   (Fee: 1.22657 BTS) 
2017-06-08T12:52:51 Create Account 'hannes46'   (Fee: 1.22648 BTS) 
2017-06-08T12:51:48 Create Account 'solsek-1'   (Fee: 1.22648 BTS) 
2017-06-08T12:51:24 Create Account 'mk-frrn'   (Fee: 1.22640 BTS) 
2017-06-08T12:48:06 Create Account 'icevenom-shine'   (Fee: 1.22698 BTS) 
2017-06-08T12:45:24 Create Account 'bluesky-angel'   (Fee: 1.22690 BTS) 
2017-06-08T12:44:57 Create Account 'dufe1993'   (Fee: 1.22648 BTS) 
2017-06-08T12:42:27 Create Account 'damir-o'   (Fee: 1.22640 BTS) 
2017-06-08T12:37:51 Create Account 'dan4354'   (Fee: 1.22640 BTS) 
2017-06-08T12:32:33 Create Account 'mmxmm'   (Fee: 1.22624 BTS) 
2017-06-08T12:27:39 Create Account 'chbtt'   (Fee: 1.22624 BTS) 
2017-06-08T12:25:42 Create Account 'lrcost3'   (Fee: 1.22640 BTS) 
2017-06-08T12:21:57 Create Account 'bobwood56'   (Fee: 1.22657 BTS) 
2017-06-08T12:21:09 Create Account 'chance-8'   (Fee: 1.22648 BTS) 
2017-06-08T12:20:51 Create Account 'kgb666'   (Fee: 1.22632 BTS) 
2017-06-08T12:20:09 Create Account 'jsv9015'   (Fee: 1.22640 BTS) 
2017-06-08T12:19:30 Create Account 'test-crazyrooster'   (Fee: 1.22723 BTS) 
2017-06-08T12:18:18 Create Account 'memet-kare'   (Fee: 1.22665 BTS) 
2017-06-08T12:14:06 Create Account 'leontius02'   (Fee: 1.22665 BTS) 
2017-06-08T12:10:21 Create Account 'btc-eu'   (Fee: 1.22632 BTS) 
2017-06-08T12:08:33 Create Account 'liokant22'   (Fee: 1.22657 BTS) 
2017-06-08T12:05:15 Create Account 'chhbts'   (Fee: 1.22632 BTS) 
2017-06-08T12:04:15 Create Account 'kkkkkkkkkk'   (Fee: 1.22665 BTS) 
2017-06-08T12:01:03 Create Account 'dong-a'   (Fee: 1.22632 BTS) 
2017-06-08T11:59:06 Create Account 'pcfranco-bmv'   (Fee: 1.22682 BTS) 
2017-06-08T11:58:39 Create Account 'dbramedya91'   (Fee: 1.22673 BTS) 
2017-06-08T11:56:57 Create Account 'chbtts'   (Fee: 1.22632 BTS) 
2017-06-08T11:56:24 Create Account 'sd3913'   (Fee: 1.22632 BTS) 
2017-06-08T11:51:51 Create Account 'hzgy69'   (Fee: 1.22632 BTS) 
2017-06-08T11:51:27 Create Account 'maha5ch'   (Fee: 1.22640 BTS) 
2017-06-08T11:51:24 Create Account 'ulrich-j'   (Fee: 1.22648 BTS) 
2017-06-08T11:50:57 Create Account 'v34936023'   (Fee: 1.22657 BTS) 
2017-06-08T11:50:54 Create Account 'serenus2017'   (Fee: 1.22673 BTS) 
2017-06-08T11:46:00 Create Account 'martiege-10'   (Fee: 1.22673 BTS) 
2017-06-08T11:45:51 Create Account 'fauli-fauli'   (Fee: 1.22673 BTS) 
2017-06-08T11:44:15 Create Account 'shaymu0829'   (Fee: 1.22665 BTS) 
2017-06-08T11:42:39 Create Account 'schnbsl'   (Fee: 1.22640 BTS) 
2017-06-08T11:41:30 Create Account 'cath123'   (Fee: 1.22640 BTS) 
2017-06-08T11:40:18 Create Account 'oaxafll74r'   (Fee: 1.22665 BTS) 
2017-06-08T11:36:12 Create Account 'fokiu-paymi'   (Fee: 1.22673 BTS) 
2017-06-08T11:33:48 Create Account 'saskia101'   (Fee: 1.22657 BTS) 
2017-06-08T11:31:33 Create Account 'ad-omo'   (Fee: 1.22632 BTS) 
2017-06-08T11:28:45 Create Account 'floriang91'   (Fee: 1.22665 BTS) 
2017-06-08T11:28:27 Create Account 'benjamin1'   (Fee: 1.22657 BTS) 
2017-06-08T11:25:39 Create Account 'vdeniz3'   (Fee: 1.22640 BTS) 
2017-06-08T11:18:33 Create Account 'elmor1961'   (Fee: 1.22657 BTS) 
2017-06-08T11:16:06 Create Account 'mhvb'   (Fee: 1.22615 BTS) 
2017-06-08T11:13:21 Create Account 'borivoje1'   (Fee: 1.22657 BTS) 
2017-06-08T11:12:12 Create Account 'nikhil-86'   (Fee: 1.22657 BTS) 
2017-06-08T11:10:36 Create Account 'jeff-circeo'   (Fee: 1.22673 BTS) 
2017-06-08T11:10:03 Create Account 'amaya230'   (Fee: 1.22648 BTS) 
2017-06-08T11:07:57 Create Account 'nhanhua90'   (Fee: 1.22657 BTS) 
2017-06-08T11:02:24 Create Account 'b1tsh4r3'   (Fee: 1.22648 BTS) 
2017-06-08T10:54:21 Create Account 'tc163mail163.com'   (Fee: 1.22715 BTS) 
2017-06-08T10:52:15 Create Account 'vine96'   (Fee: 1.22632 BTS) 
2017-06-08T10:45:36 Create Account 'kimmi78'   (Fee: 1.22640 BTS) 
2017-06-08T10:44:42 Create Account 'booger420'   (Fee: 1.22657 BTS) 
2017-06-08T10:41:39 Create Account 'c555666'   (Fee: 1.22640 BTS) 
2017-06-08T10:34:15 Create Account 'sam-whittemore'   (Fee: 1.22698 BTS) 
2017-06-08T10:33:15 Create Account 'w888888'   (Fee: 1.22640 BTS) 
2017-06-08T10:29:51 Create Account 'darkcoin-0'   (Fee: 1.22665 BTS) 
2017-06-08T10:28:54 Create Account 'aaffaad123'   (Fee: 1.22665 BTS) 
2017-06-08T10:28:45 Create Account 'kugl0f-x'   (Fee: 1.22648 BTS) 
2017-06-08T10:19:39 Create Account 'cjf-bts'   (Fee: 1.22640 BTS) 
2017-06-08T10:12:39 Create Account 'szzc-com1.eeee'   (Fee: 1.22698 BTS) 
2017-06-08T10:10:54 Create Account 'jubi-bts'   (Fee: 1.22648 BTS) 
2017-06-08T10:10:06 Create Account 'manko-chinko'   (Fee: 1.22682 BTS) 
2017-06-08T10:09:00 Create Account 'mancini87'   (Fee: 1.22657 BTS) 
2017-06-08T10:05:39 Create Account 'g-m-collingridge'   (Fee: 1.22715 BTS) 
2017-06-08T10:01:06 Create Account 'resupo01'   (Fee: 1.22648 BTS) 
2017-06-08T10:01:00 Create Account 'ramahandra111'   (Fee: 1.22690 BTS) 
2017-06-08T09:59:51 Create Account 'magin1'   (Fee: 1.22632 BTS) 
2017-06-08T09:55:36 Create Account 'dudule352'   (Fee: 1.22657 BTS) 
2017-06-08T09:54:39 Create Account 'happy-daddy'   (Fee: 1.22673 BTS) 
2017-06-08T09:53:00 Create Account 'sxche1'   (Fee: 1.22632 BTS) 
2017-06-08T09:51:48 Create Account 'tamosom-mikami'   (Fee: 1.22698 BTS) 
2017-06-08T09:45:45 Create Account 'kaiserwoo79'   (Fee: 1.22673 BTS) 
2017-06-08T09:45:33 Create Account 'davide72'   (Fee: 1.22648 BTS) 
2017-06-08T09:45:15 Create Account 'lastwarrior-arabela'   (Fee: 1.22740 BTS) 
2017-06-08T09:44:48 Create Account 'lang-ya'   (Fee: 1.22640 BTS) 
2017-06-08T09:42:51 Create Account 'yzsol2016'   (Fee: 1.22657 BTS) 
2017-06-08T09:41:48 Create Account 'qiankun-zhu'   (Fee: 1.22673 BTS) 
2017-06-08T09:41:18 Create Account 'wdfy1314'   (Fee: 1.22648 BTS) 
2017-06-08T09:23:51 Create Account 'globalshare888'   (Fee: 1.22698 BTS) 
2017-06-08T09:23:39 Create Account 'cht123'   (Fee: 1.22632 BTS) 
2017-06-08T09:23:03 Create Account 'rafa1239'   (Fee: 1.22648 BTS) 
2017-06-08T09:20:12 Create Account 'amyzyy-bts'   (Fee: 1.22665 BTS) 
2017-06-08T09:19:15 Create Account 'ccqqhzxgh'   (Fee: 1.22657 BTS) 
2017-06-08T09:12:51 Create Account 'lacrim-1'   (Fee: 1.22648 BTS) 
2017-06-08T09:12:00 Create Account 'dimat76'   (Fee: 1.22640 BTS) 
2017-06-08T09:09:33 Create Account 'brktt'   (Fee: 1.22624 BTS) 
2017-06-08T09:04:06 Create Account 'fe397381333'   (Fee: 1.22673 BTS) 
2017-06-08T09:01:57 Create Account 'iagaev1'   (Fee: 1.22640 BTS) 
2017-06-08T08:57:36 Create Account 'snova001'   (Fee: 1.22648 BTS) 
2017-06-08T08:50:21 Create Account 'austinedung5'   (Fee: 1.22682 BTS) 
2017-06-08T08:48:57 Create Account 'fonds25'   (Fee: 1.22640 BTS) 
2017-06-08T08:46:09 Create Account 'li-shu-de'   (Fee: 1.22657 BTS) 

new >>> 

@destenson
Copy link

I think someone should fix this properly instead of placing arbitrary limits that may cause additional distasteful side effects for those of us that use this daily.

@oxarbitrage
Copy link
Member

@destenson there is always some limit for pagination in bitshares core. 100 is most of the time the number selected. the issue and fix have nothing to do with if 100 is a good number as a limit for pagination; is actually a bug in the intended action.

@destenson
Copy link

Ok, well this may not be the same issue that I've been dealing with for the past year, but my wallet now has 385 keys in it, and overall performance has always been horrendous, lately it's been especially slow and rarely able to recover. Among the things I've noticed is console.log() continuously outputting the following error message repeated dozenes of times per second:

 20:08:32.138 app.js?ab254acffe8bd5939945d7da4fa26147:formatted:113822 !!! GrapheneApi error:  get_full_accounts Array(2) Object {"code":1,"message":"10 assert_exception: Assert Exception\nstd::distance(_subscribed_accounts.begin(), _subscribed_accounts.end()) < 100: \n    {}\n    th_a  database_api.cpp:621 get_full_accounts\n\n    {\"call.method\":\"call\",\"call.params\":[2,\"get_full_accounts\",[[\"1.2.NNNNNN\"],true]]}\n    th_a  websocket_api.cpp:121 on_message","data":{"code":10,"name":"assert_exception","message":"Assert Exception","stack":[{"context":{"level":"error","file":"database_api.cpp","line":621,"method":"get_full_accounts","hostname":"","thread_name":"th_a","timestamp":"2017-06-10T03:08:29"},"format":"std::distance(_subscribed_accounts.begin(), _subscribed_accounts.end()) < 100: ","data":{}},{"context":{"level":"warn","file":"websocket_api.cpp","line":121,"method":"on_message","hostname":"","thread_name":"th_a","timestamp":"2017-06-10T03:08:29"},"format":"","data":{"call.method":"call","call.params":[2,"get_full_accounts",[["1.2.NNNNN"],true]]}}]}}
 20:08:32.138 app.js?ab254acffe8bd5939945d7da4fa26147:formatted:114450 Error:  Object

It apparently tries to 'get_full_accounts' for every account in the wallet every second, and it simply can't keep up. In a quick & dirty analysis of a log taken over 7 minutes, I found it prints this message 69 times for each unique account, once every 6 seconds on average. It's very bursty though, printing out few hundred errors over the course of a couple seconds then nothing for several seconds & repeat....

My point is, I think my issues are related these pagination issues within bitshares-core. If there is a hard-limit of 100 throughout, then it should handle these situations more gracefully, (without errors, incorrect output, or bad UX). Wallets should probably be prevented from adding more than 100 accounts during creation, or have the ability to be split into smaller wallets. And the code that opens processes these accounts should catch that the list to process is too long far before an assertion in native code does. It should check that the length is within reason first, and if not, not try to keep loading them anyway every 1-5 seconds causing degradation of performance quickly approaching unusable.

@oxarbitrage
Copy link
Member

@destenson your point is totally valid, it is just that should go here:

#295

the confusion is because in this issue(#298) the 100 is for pagination. this pagination is used for pulling big amounts of data in blocks, for example you ask the code for the last 100 results with 0,100. next 100 results with 100, 200, next with 200,300 and so on.

In the case of #295 the number is 100 but it is a different thing as the 100 is not acting as a pagination there but as a limitation on how many accounts a user can be getting updates at the same time.

lets continue the discussion about it at #295

@oxarbitrage oxarbitrage mentioned this issue Jul 23, 2017
oxarbitrage added a commit that referenced this issue Aug 14, 2017
* new pull for issue 298

* Revert "finish sentence in readme"

This reverts commit bf76c59.

* Revert "Add api-access console log to node startup when present"

This reverts commit 80d81ac.

* add support for start=0

* changes to rvalues

* Update application.cpp

* Update application.cpp

* Update application.cpp

* Update README.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants