Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store some data in ES as objects instead of just as strings #2565

Merged
merged 21 commits into from
Dec 26, 2021

Conversation

abitmore
Copy link
Member

@abitmore abitmore commented Dec 20, 2021

PR for #2380 and #1400.

The strings are kept for backward compatibility. The objects are stored redundantly in new fields with name (old_field_name)_object. The affected fields are listed below.

  • maps are stored as [ { "key_(type)": k, "data_(type)": d } , ... ], where (type) can be int, bool, string or object. The fields are:
    • account_auths in account_object and related operations
    • address_auths in account_object and related operations
    • key_auths in account_object and related operations
    • feeds in asset_bitasset_data_object
    • acceptable_collateral in credit_offer_object and related operations
    • acceptable_borrowers in credit_offer_object and related operations
  • static_variants are stored as { "which": w, "data_(type)" : d }, where (type) can be int, bool, string or object. The fields are:
    • op in proposal_create_operation.proposed_ops
    • initializer in worker_create_operation
    • policy in vesting_balance_create_operation and etc
    • active_special_authority in account_object and related operations
    • owner_special_authority in account_object and related operations
    • htlc_preimage_hash in htlc_create_operation
    • argument in restriction class, used in custom authority operations (added in Adapt restriction::argument as an object for ES #2681)
    • operation_result
  • object arrays are stored normally as object arrays, especially, among them, static_variant arrays are stored as object arrays, including:
    • parameters in committee_update_chain_parameters_operation.**.current_fees
    • proposed_ops in proposal_create_operation
    • operations in proposal_object
    • predicates in assert_operation

Note: more disk space is required for ES to store the redundant data. Here is some data for comparison.

index old size new size
history-2015-10 81.9M 89.1M
history-2015-11 120.1M 146.5M
history-2015-12 133.4M 147.7M
history-2016-01 150.2M 166.1M

BTW #2568 is the next work to be done.

@abitmore abitmore added this to the 6.1.0 - Feature Release milestone Dec 20, 2021
@abitmore abitmore added this to In progress in Feature Release (6.1.0) via automation Dec 20, 2021
@abitmore abitmore linked an issue Dec 20, 2021 that may be closed by this pull request
17 tasks
@abitmore abitmore changed the title Store some data in ES as objects rather than strings Store some data in ES as objects instead of just as strings Dec 24, 2021
@abitmore abitmore marked this pull request as ready for review December 25, 2021 15:29
@abitmore abitmore linked an issue Dec 25, 2021 that may be closed by this pull request
2 tasks
@sonarcloud
Copy link

sonarcloud bot commented Dec 25, 2021

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

63.8% 63.8% Coverage
0.0% 0.0% Duplication

@abitmore abitmore merged commit 78beec6 into develop Dec 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

Save operation_result and proposed_ops as objects in ElasticSearch plugin elasticsearch plugin fixes.
1 participant