Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance AH rocksdb-prototype to cover all exiting functionality of account-history plugin #2066

Closed
vogel76 opened this issue Jan 31, 2018 · 2 comments
Closed

Enhance AH rocksdb-prototype to cover all exiting functionality of account-history plugin #2066

vogel76 opened this issue Jan 31, 2018 · 2 comments
Assignees
@vogel76

Comments

@vogel76
Copy link
Contributor

vogel76 commented Jan 31, 2018

Add support for:

  • account white-listing, black-listing
  • operation filtering
  • data pruning to cover all exiting functionality of account-history plugin
@vogel76 vogel76 mentioned this issue Jan 31, 2018
Open
7 tasks
@vogel76 vogel76 self-assigned this Jan 31, 2018
vogel76 added a commit that referenced this issue Feb 1, 2018
@vogel76
Issue #2066 Extending AH-rocksdb plugin functionality to match old one.
5ef416a 
- Implemented same options as in old AH plugin. Options in old AH plugin renamed to avoid conflict.
- Implemented filtering data collection (for account filtering).
Not finished yet. Can fail CI.
vogel76 added a commit that referenced this issue Feb 1, 2018
@vogel76
Issue #2066 Implemented account and operation filtering options. Need…
0349004 
…s testing.
vogel76 added a commit that referenced this issue Feb 4, 2018
@vogel76
Issue #2066 Enhanced debug logging to simplify potential bug hunting.
9dad4d2
vogel76 added a commit that referenced this issue Feb 4, 2018
@vogel76
Issue #2066 Prerequisite changes for prune option support. Slightly c…
de1c13e 
…hanged data structures to allow counting AH records for given account (similary to original implementation).
vogel76 added a commit that referenced this issue Feb 7, 2018
@vogel76
Issue #2066 Redesigned data structures to make possible implementatio…
626c777 
…n of AH data prunning. Reimplemented get_account_history API method accordingly.
vogel76 added a commit that referenced this issue Feb 8, 2018
@vogel76
Issue #2066 Compile fixes to errors appeared after merging develop.
c26c899
vogel76 added a commit that referenced this issue Feb 8, 2018
@vogel76
Issue #2066 Implemented handling of reindexStart, reindexStop signals.
5c542e0
@vogel76
Copy link
Contributor Author

vogel76 commented Feb 9, 2018

Added previously missing functionality:

  • account filtering
  • operation filtering
  • AH data pruning
    Needed changes did force redesign of data structures and it lead to small slowdown according to golden results initially taken (up to 19M blocks):
  • reindex process is a little longer (reindex slowdown is near to 10%) and takes now 6678s (6065s originally, and 13328 in develop) what is caused by directly reading data while writing them into db.
  • reindex process is significantly longer when prune is enabled and takes 10411s. Comparison will be added when reindex + prune will complete on develop version (it is longer than no-prune).
  • memory allocation is stable, depends on cache configuration.
    To make PR, docker build configuration must be fixed, what will be next step to be done.

@vogel76
Copy link
Contributor Author

vogel76 commented Feb 9, 2018

As a side note, prune option by default shall be disabled now. It damages historical data, makes data collection longer. So it makes no sense to have it enabled by default

@mvandeberg mvandeberg mentioned this issue Feb 9, 2018
Closed
vogel76 added a commit that referenced this issue Feb 15, 2018
@vogel76
Issue #2099 Issue #2066 Implemented enum_virtual_ops method.
9e1561f
@vogel76 vogel76 mentioned this issue Feb 16, 2018
Merged
This was referenced Mar 2, 2018
Closed
Open
vogel76 added a commit that referenced this issue Mar 7, 2018
@vogel76
Issue #2066 Configured RocksDB max_open_file limit to avoid write err…
984bb06 
…ors caused by Ubuntu ulimit exceeded.
vogel76 added a commit that referenced this issue Mar 7, 2018
@vogel76
Issue #2066 History pruning is disabled by default. Warning is printe…
4ceef77 
…d about prune option deprecation.
mvandeberg pushed a commit that referenced this issue Mar 20, 2018
@vogel76 @mvandeberg
Issue #2066 Extending AH-rocksdb plugin functionality to match old one.
e7619a5 
- Implemented same options as in old AH plugin. Options in old AH plugin renamed to avoid conflict.
- Implemented filtering data collection (for account filtering).
Not finished yet. Can fail CI.
mvandeberg pushed a commit that referenced this issue Mar 20, 2018
@vogel76 @mvandeberg
Issue #2066 Implemented account and operation filtering options. Need…
4af0073 
…s testing.
mvandeberg pushed a commit that referenced this issue Mar 20, 2018
@vogel76 @mvandeberg
Issue #2066 Enhanced debug logging to simplify potential bug hunting.
235a0ac
mvandeberg pushed a commit that referenced this issue Mar 20, 2018
@vogel76 @mvandeberg
Issue #2066 Prerequisite changes for prune option support. Slightly c…
bad3aff 
…hanged data structures to allow counting AH records for given account (similary to original implementation).
mvandeberg pushed a commit that referenced this issue Mar 20, 2018
@vogel76 @mvandeberg
Issue #2066 Redesigned data structures to make possible implementatio…
7e752a6 
…n of AH data prunning. Reimplemented get_account_history API method accordingly.
mvandeberg pushed a commit that referenced this issue Mar 20, 2018
@vogel76 @mvandeberg
Issue #2066 Compile fixes to errors appeared after merging develop.
c0eb8c4
mvandeberg pushed a commit that referenced this issue Mar 20, 2018
@vogel76 @mvandeberg
Issue #2066 Implemented handling of reindexStart, reindexStop signals.
de873db
mvandeberg pushed a commit that referenced this issue Mar 20, 2018
@vogel76 @mvandeberg
Issue #2099 Issue #2066 Implemented enum_virtual_ops method.
b4a58ae
mvandeberg pushed a commit that referenced this issue Mar 20, 2018
@vogel76 @mvandeberg
Issue #2066 Configured RocksDB max_open_file limit to avoid write err…
044dc54 
…ors caused by Ubuntu ulimit exceeded.
mvandeberg pushed a commit that referenced this issue Mar 20, 2018
@vogel76 @mvandeberg
Issue #2066 History pruning is disabled by default. Warning is printe…
49bd77a 
…d about prune option deprecation.
mvandeberg added a commit that referenced this issue Mar 20, 2018
@mvandeberg
Vendor rocksdb #2066
10cb164
mvandeberg pushed a commit that referenced this issue Mar 21, 2018
@vogel76 @mvandeberg
Issue #2066 Extending AH-rocksdb plugin functionality to match old one.
4459c17 
- Implemented same options as in old AH plugin. Options in old AH plugin renamed to avoid conflict.
- Implemented filtering data collection (for account filtering).
Not finished yet. Can fail CI.
mvandeberg pushed a commit that referenced this issue Mar 21, 2018
@vogel76 @mvandeberg
Issue #2066 Implemented account and operation filtering options. Need…
c54384b 
…s testing.
mvandeberg pushed a commit that referenced this issue Mar 21, 2018
@vogel76 @mvandeberg
Issue #2066 Enhanced debug logging to simplify potential bug hunting.
2cf97e8
mvandeberg pushed a commit that referenced this issue Mar 21, 2018
@vogel76 @mvandeberg
Issue #2066 Prerequisite changes for prune option support. Slightly c…
7305d5b 
…hanged data structures to allow counting AH records for given account (similary to original implementation).
mvandeberg pushed a commit that referenced this issue Mar 21, 2018
@vogel76 @mvandeberg
Issue #2066 Redesigned data structures to make possible implementatio…
61ab993 
…n of AH data prunning. Reimplemented get_account_history API method accordingly.
mvandeberg pushed a commit that referenced this issue Mar 21, 2018
@vogel76 @mvandeberg
Issue #2066 Compile fixes to errors appeared after merging develop.
f3093c8
mvandeberg pushed a commit that referenced this issue Mar 21, 2018
@vogel76 @mvandeberg
Issue #2066 Implemented handling of reindexStart, reindexStop signals.
03e1303
mvandeberg pushed a commit that referenced this issue Mar 21, 2018
@vogel76 @mvandeberg
Issue #2099 Issue #2066 Implemented enum_virtual_ops method.
61a5e40
mvandeberg pushed a commit that referenced this issue Mar 21, 2018
@vogel76 @mvandeberg
Issue #2066 Configured RocksDB max_open_file limit to avoid write err…
2c6b109 
…ors caused by Ubuntu ulimit exceeded.
mvandeberg pushed a commit that referenced this issue Mar 21, 2018
@vogel76 @mvandeberg
Issue #2066 History pruning is disabled by default. Warning is printe…
cafd8ff 
…d about prune option deprecation.
mvandeberg added a commit that referenced this issue Mar 21, 2018
@mvandeberg
Vendor rocksdb #2066
d269629
mvandeberg added a commit that referenced this issue Mar 21, 2018
@mvandeberg
Fix error from rebase #2066
4273361
@mvandeberg mvandeberg mentioned this issue Mar 23, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vogel76 vogel76

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vogel76 @mvandeberg