Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Account.history is missing operations due to wrongly estimated op numbers #36

Closed
crokkon opened this issue Jun 29, 2018 · 0 comments
Closed

Comments

@crokkon
Copy link
Contributor

crokkon commented Jun 29, 2018

Expected behavior

Account.history(start, stop) with start and stop being datetime instances should return all blockchain operations from an account for the given start/stop timestamps.

Actual behavior

Account.history(start, stop) may be missing operations from the given time range

How to reproduce

I know I've created a post on 2018-02-05 14:06:54, so the account history should contain at least one comment operation on that day:

from beem.account import Account
from datetime import datetime, timedelta
from beem.utils import parse_time, addTzInfo

a = Account("stmdev")
start = addTzInfo(datetime(2018, 2, 5))
stop = addTzInfo(datetime(2018, 2, 6))
for op in a.history(start=start, stop=stop, only_ops=['comment']):
    print(op['timestamp'], op['author'], op['permlink'])

The result is empty even though there were blockchain operations from this account in the given time range.

Deeper analysis

The virtual op number estimation gives wrong results in this case. I know that this comment operation has index=2:

for op in a.get_account_history(2, 0):
    print(op['timestamp'], op['author'], op['permlink'])
2018-02-05T14:06:54 stmdev test

So a.estimate_virtual_op_num(datetime(2018, 2, 5, 14, 6, 54)) should return 2, but it actually returns 282:

from beem.account import Account
from datetime import datetime, timedelta
from beem.utils import parse_time, addTzInfo

a = Account("stmdev")
print(a.estimate_virtual_op_num(datetime(2018, 2, 5, 14, 6, 54)))

estimate_virtual_op_num has a max_count parameter that defines how many attempts beem should make before giving up. The default is 100. The limit can be disabled via max_count=-1, so beem should do as many attempts as necessary to find the according virtual op number:

a.estimate_virtual_op_num(datetime(2018, 2, 5, 14, 6, 54), max_count=-1)

This runs in an endless loop and never returns. The virtual op number estimation does not converge. The result 282 from above is just by coincidence the result after the 100th iteration for the default max_count until beem gives up. Printing out the current op num from the while loop shows that this number is oscillating and not converging towards the target value.

This behavior is the same across beem-0.19.37 up to 0.19.41, so it's not introduced from the latest changes to this function.

Environment

beempy, version 0.19.41
# python --version
Python 3.6.5
crokkon added a commit to crokkon/beem that referenced this issue Jul 4, 2018
Fixes holgern#36. More reliable convergence and therefore less RPC calls.
crokkon added a commit to crokkon/beem that referenced this issue Jul 4, 2018
Fixes holgern#36. More reliable convergence and therefore less RPC calls.
crokkon added a commit to crokkon/beem that referenced this issue Jul 4, 2018
Fixes holgern#36. More reliable convergence and therefore less RPC calls.
KINGdotNET added a commit to KINGdotNET/beem that referenced this issue Oct 11, 2018
* Missing Account import

* Update pytz from 2018.4 to 2018.5

* Fix wrong version number in known_chains

* Add vit to known_chains

* Prepare next release and add chain for testnet.steemitdev.com

* reworking account.estimate_virtual_op_num

Fixes holgern#36. More reliable convergence and therefore less RPC calls.

* Add lazy  as option to all discussion classes

* Fix start and stop for history_reverse

* Prepare next release

* Add option use_del to ObjectCache, for making the cache thread safe

* Try to fix clear_expired_items

* Add bool for clear_expired_items to make it thread safe

* Add RLock to Cache for make it thread safe

* Remove use_del again from ObjectCache and optimize OpjectCache

Example
* A cache performance evalution script was added to examples
Blockchain
* set_cache_auto_clean disabled

* Fix unit test for Object Cache

* Add support for nodes with version below 0.19.5 and fix node version comparision

Relase nodes for 0.19.45 added

* Replace recursive call by while loop

* Update nodes in Nodelist
* Disable seed.bitcoiner.me node

* Add vit to known_formats

* Release of 0.19.46

Snapshot added
* This class can be used to access old account states. At the moment, only changes in vests are tracked over time.
* example added for plotting SP over time

Steem
* last_node added and refresh of json_config data is forced on node switch

* fix SteemConnect TransactionBuilder example

Fixes holgern#42

* snapshot improved and unit tests fixed

Snapshot
* impoved and STEEM and SBD history added
* Documentation added
* get_data, reset, build functions added

* Fix unit test and improve documentation

* Snapshot improved

* get_data is now fast
* update_rewards added
* enable_rewards added to build
* build_curation_arrays added

* Add blocknumber check to wait_for_and_get_block

* Update pyyaml from 3.12 to 3.13

* Prepare next release and fix Amount for condenser broadcast ops (used for appbase right now)

* fixes issue holgern#45

* Fixes issue holgern#45 by checking the timestamp of last_payout instead of net_rshares. With these changes, voting of unvoted posts/comments is possible.
* Refactoring of Transactionbuilder and adding of _use_condenser_api for defining the use of condenser api on broadcasting when using appbase nodes..

* Add get_all_replies to comment for fetching all replies

* Fix unit tests

* Fix formatToTimestamp for python 2.7

* Improve unit test for vote

* Fix unit test_vote for python 2.7

*  Refactor Amount detection in account, claimreward improved

* Several improvements and fixes

CLI
* upvote and downvote fixed
Snapshot
* refactoring
* update_vote and build_vp_arrays added for showing vote power history
Examples
* account_vp_over_time added

* Prepare next release and add next_witness_block_countdown example

* add get_parent to comment and fix for beempy reward

* beem/steem.py: calculate with absolute used_power values

Fixes holgern#46

* Update pycryptodomex from 3.6.3 to 3.6.4

* discussions: set steem inst. as keyword argument

Fixes holgern#49

* fix discussions.Replies_by_last_update()

`start_parent_author` parameter was missing in the RPC call and the
Query class.
Fixes holgern#52

* Fix issue holgern#51

* Fixes replies for Discussions

* Fixes Discussions, when only one reply is returned

* discussions: allow empty replies

setting `set_next_node_on_empty_reply` for Discussions_by_feed and
Discussions_by_comments
Fixes holgern#54

* snapshot improved and unit test fixed

Snapshot
* update_in_vote added 
* enable_votes changed to enable_in_votes and enable_out_votes
* build_rep_arrays added
Example
* account_vp_over_time added
* account_reputation_by_SP added
Discussion
* check if parameter exists in discussion_query added

* Fix unit test

* adding account.blog_history()

* adding account.comment_history()

* adding account.reply_history()

* adding account.feed_history()

* fix spelling errors reported by misspell

* Fix holgern#57 and try to improve error "Client returned invalid format. Expected JSON!"  handling

* Fix unit tests and improve json export for account

Comment
* dpeth property added

* Fix flake8 and improve steemconnect fix for holgern#57

* Prepare next release

* Switch to condeser call for list_witnesses with by_vote_name
* changed steemconnect url
* add check in transactionbuilder

* ecdsasig: allow to skip hashing on signing

* Removed asset_precision and used precision from beemgraphenebase.chains  instead

* Add option to load custom_chain into steem

* adding intial Conveyor class

Methods:
* `get_user_data()`
* `get_feature_flags()`

* Conveyor: adding set_user_data

* Conveyor: adding get_feature_flag

* Conveyor: adding draft handling

* save_draft()
* list_drafts()
* remove_draft()

* Conveyor: adding healthcheck

* adding basic Conveyor unittest

* Revert "ecdsasig: allow to skip hashing on signing"

This reverts commit c08c815.

* Conveyor: fix hashing

* Fix transactionbuilder for python 2.7

* Update cryptography from 2.2.2 to 2.3

* Add custom chains function to signed transactions

* Add docu for conveyor and readme updated

* Fix conveyor documentation and prepare next release

* Fix more documentation

* Fix docu

* Add d.tube format to resolve_authorperm, 

disable_chain_detection added to graphenerpc (for testing hivemind e.g.)

* Add missing trx_num to streamed block operation

* set_next_node_on_empty_reply added to some appbase rpc calls

* Fix unit test

* Prepare next release and fix unit test  for py 2.7

* Add new appbase node: appbase.buildteam.io

* Fix pep8

* Make account history appbase ready

* Prepare next version and fix account refresh and ops_statistics

* fix Account.get_tags_used_by_author on appbase

Fixes holgern#62

* Fix Blockchain.get_transaction_hex() on appbase

Refs holgern#64

* Prepare next version

* steemd.pevo.science and steemd.steemgigs.org removed from Nodelist
* bug fixed in allow and disallow for CLI
* fix holgern#64
* Show results for get_user_data and get_feature_flags in Conveyor

* Fix vests_to_rshares (issue holgern#66)

* Fix unit tests and adapt mvest/vest relation for vests_to_rshares

* Fix unnecessary repeated code

* Add new_vote flag to vote value calculations

* Fix issue holgern#69 and fix a bug in batched streaming + cli

remove auto_clean from streaming

* Fix unit test and improve get_blog, get_replies and get_comments

* Add last_current_block_num parameter to wait_for_and_get_block for reducing the number of api calls

* Update nodelist

* new_vote renamed to not_broadcasted_vote

* Added an example to the custom_json section for documentation.

It was slightly unclear that one of the two required auths were needed since both also say optional. Added a note explaining the need for one of the two as well as a block of pseudocode to display the message further.

* Add function for transforming sbd into rshares

* Add sbd_to_vote_pct function

Adds the possibility of calculating the needed voting weight for an account with a specified amount of SP/vests

* Add get_vote_pct_for_SBD function to account module

This calculates the vote percentage needed for having a given vote value.

* Add information to AssertionError in sbd_to_rshares

* Add documentation to sbd_to_vote_pct

* Add documentation to sbd_to_rshares

* Enhance get_vote_pct_for_SBD input possibilities

* Add documentation to get_vote_pct_for_SBD

* Fix documentation and typos thanks to @crokkon

* Add not_broadcasted_vote parameter to several new functions.

* Update sbd_to_rshares documentation

* Enhance sbd_to_rshares precision and add error

* Add unit tests for new functions

* Fix bug in get_vote_pct_for_SBD

* Fix error in check_asset in amount, custom_chain example added

* fix flake8

* fix flake8

* beembase/objects: fix serialization of appbase trx

Refs holgern#72

* Fix many documentation errors (based on error messages when building)

* Add missing parameter docu to sbd_to_rshares

* Add missing parts to sbd_to_vote_pct docu

* Add limit to get_following, get_followers, get_muters and get_mutings

Some more unit tests

* Fix appbase detection and unit tests

* Readme changelog added

* Fix unit test

* Optimize sbd_to_rshares to not use an iterative approach

We can get rid of the iterative approach by utilizing a few formulars. This makes the function signature smaller as well as improving performance drastically while increasing the precision. 
I also added a few comments to make it more obvious what I'm doing here.

* Make the pr sider conform

Just a bunch of deleted whitespaces.

* adding weku chain properties

* adding weku format in base58 list

* Add delegate to beempy and some HF20 fixes

* Fix flake8 and revert changes to sbd_to_rshares

* Fix warnings

* Add hardfork property to steem returning hardfork version as integer

* get_dust_threshold for getting the dust_threshold in a function

* Add the dust threshold calculations of HF20

* Make HF20 changes downvote compatible

* Additional perks to vests_to_rshares in preparation for HF20

* adding methods to claim and create discounted accouts

* Remove non appbase nodes and related tests from all unit tests Improve blockchain.stream() for appbase

* Fix more unit tests for appbase and upcoming HF20

* Prevent unnecessary RPC calls

* Fix remaining unit tests

* Improved several functions and nodelist updated

CONSTANTS
* STEEM_REVERSE_AUCTION_WINDOW_SECONDS_HF20 added

Nodelist
* nodes information updated

Steem
* get_dust_threshold() improved and get_config used
* hardfork checks improved
* hardfork property improved by using get_hardfork_properties function

Changelog.rst added and Issue holgern#80 fixed

* fix circleci

* Fix circleci config

* revert changes

* Fix unit tests

* Fix constants and unittests

* STEEM_VOTING_MANA_REGENERATION_SECONDS is used for regenerated_vp calculation

Fix unit test

* Fix get_voting_power

* Add Witness_set_properties to operations and add Offline constructTx

Example for using offline signing is provided
Next release prepared

* Add missing changes to transactionbuilder

Add parameter for enabling usage of condenser_api  for broadcasting
Add ref_block_num and ref_block_prefix to constructTx for offline signing

* Implementation of Witness_set_properties

All parameter are packed now in a correct way
Correct operationid for Witness_set_properties

objects.Amount
* naming of symbol and asset improved

* Add unit test for witness_set_properties

RawString added to types
Some improvements and fixes

* Update nodelist and add not_working parameter to get_nodes()

* Fix bytes representation of hex strings and add key sorting to Witness_set_properties

* add new witness_set_properties function 

* witnessproperties is added to beempy for setting new witness parameter 
* witnessfeed uses the new witness_set_properties function when a wif is provided

* Preparing first release for HF20, adapt account create for HF20

* Release of 0.20.0

* Fully supporting hf20
* add get_resource_params(), get_resource_pool(), claim_account(), create_claimed_account() to Steem
* fix 30x fee for create_account
* add find_rc_accounts() to Blockchain
* get_rc(), get_rc_manabar(), get_manabar() added to Account
* get_voting_power() fixed

* Release of 0.20.1

* Improved get_rc_manabar(), get_manabar() output
* get_voting_power() fixed again
* print_info for account improved
* get_manabar_recharge_time_str(), get_manabar_recharge_timedelta() and get_manabar_recharge_time() added
* https://steemd-appbase.steemit.com added to nodelist

* release of 0.20.2

* estimated_mana is now capped by estimated_max
* print_info fixed()
* get_api_methods() and get_apis() added to Steem

* Prepare next release and add changes to changelog

* several bug fixes

* Implementation of a RC costs calculation class 

* get_rc_cost was added to steem to calculation RC costs from resource count

RC
* get_tx_size(), get_resource_count(), comment(), vote(), transfer() and custom_json() added

* Fix witness update for beempy disablewitness and beempy enablewitness

* Release of 0.20.3

Account
* add comment, vote, transfer RC costs in account.print_info() and beempy power
* Shows number of possible comments, votes, tranfers with available RCs in account.print_info() and beempy power

RC
* add possibility to specify tx_size instead of constructing a dict for comment, transfer, vote and custom_json

* Release 0.20.4

* get_effective_vesting_shares() added to calculated max_mana correctly
* dict key words adapted to steemd for get_manabar() and get_rc_manabar()
* Voting mana fixed for 0 SP accounts
* comment_benefactor_reward adapted for snapshot
* Custom_json RC costs added to print_info

* Release 0.20.5

0.20.5
------
* fix get_effective_vesting_shares()

* corrected typo

invironment -> environment

* Update websocket-client from 0.48.0 to 0.53.0

* Update pycryptodomex from 3.6.4 to 3.6.6

* Update cryptography from 2.3 to 2.3.1

* Update click from 6.7 to 7.0

* beembase: adding chain prefix handling

* steem.py: adding generic asset symbol names

* account.py: using generic asset symbol names

* comment.py: using generic asset symbol names

* witness.py: using generic asset symbol names

* snapshot.py: using generic asset symbol names

* prepare next release

* add claim_account RC calculation
* fix Bytes type
* Improve testnet example
* test_types improved

* Fix unit tests for new amount dict format in test_transactions

* Fix unit test for python2.7

* Add support for EFTG

* Add support for EFTG

* Several improvements and fixes

Account
* some code improvements
Graphenerpc
* small improvement in chain detection
Steem
* replace_steemit_by_steem option removed from get_config
* witness_update fixed
Comment
* Fix docu
chain
* Waleshares added
Unit test
* unit tests fixed

* Fix unit tests

* Fix get_effective_vesting_shares for python 2.7

* Fix timestamp

* Add before_date, author to Query

* Fix unit tests and skip not working tests on testnet

* Try to fix unit test

* Fix input params for Discussions_by_author_before_date in get_discussions

* Update docs Discussions_by_author_before_date

* Prepare changelog for release

* WLS prefix fixed and next version prepared

* Fix typo in documentation

This is a common mistake, its vs it's
Read about it at http://www.elearnenglishlanguage.com/blog/english-mistakes/its/

* Fix get_discussions infinite cycle when discussions are empty

* Fix typo and improved handling of symbols in comment

* account.py: fix potential ZeroDivisionError

closes holgern#99

* account.py: further division-by-0 preventions

refs holgern#99

* Add claimaccount to beempy and some improvements for steem.sbd_symbol

* remove debug print

* add create_claimed_account to beempy and add correct operationids for WLS

* Small improvements

* Fix prefix parameter for signedtransaction init function and improve steem.refresh_data()

* Fix test_block unittest
* Adapt Changelog for next release

* Fix unit tests

* All constant STEEM and SBD strings inside beempy are replaced by steem.steem_steem steem.sbd_symbol properties

* Prepare next release and fix account_create for chains comp. to HF19

* Fix hardfork property

* Fix resource_market_bytes calculation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant