elasticsearch history api #1682 #1725

oxarbitrage · 2019-04-20T00:33:03Z

#1682

Added get_operation_by_id as requested at ES Plugin: Get operation by ID #1682 (comment) to elasticsearch plugin.
Added get_account_history to elasticsearch and use it when plugin is available to serve get_account_history api calls.
Test cases added are the same as history_api_tests/get_account_history_additional to make sure results are consistent.

Considerations:

Option elasticsearch-operation-string was added and is needed to serve the history api calls. The op_object field is modified to fit elasticsearch when indexing, we need the original object in a string to transmit it back on request. This will increase the size of the resources needed for ES but as an optional option users can use the string if they only want the node to serve history api calls or save the object for kibana/others. Can also store the 2 things if he haves the space.

Questions/TODO:

Implement get_account_history_operations - should be easy, it is almost the same query as get_account_history.
Implement get_relative_account_history - this is a bit trickier to do in ES - do we need it ?
Add get_operation_by_id as api call?

libraries/plugins/elasticsearch/elasticsearch_plugin.cpp

libraries/app/api.cpp

pmconrad · 2019-07-26T15:02:40Z

libraries/app/api.cpp

+       if(_app.is_plugin_enabled("elasticsearch")) {
+          auto es = _app.get_plugin<elasticsearch::elasticsearch_plugin>("elasticsearch");
+          auto _thread = std::make_shared<fc::thread>("elasticsearch");
+          return _thread->async([&](){ return es->get_account_history(account, stop, limit, start); },


Please capture only what's necessary.

Also, don't use this if mode is only_save.

Does this actually work as intended? That would be cool.
We should load-test this before putting it into production.

libraries/app/application.cpp

libraries/plugins/elasticsearch/elasticsearch_plugin.cpp

pmconrad · 2019-07-26T15:08:28Z

libraries/plugins/elasticsearch/elasticsearch_plugin.cpp

+            {
+               "range": {
+                  "block_data.block_time": {
+                     "gte": "now-20y",


Put a TODO in your calendar for the year 2034: bump this. ;-)

Can't you leave out this part of the query?

I agree that the range could be omitted here.

Also I am wondering why you do query_string and not

"query": { "match" : { "account_history.operation_id" : "operation_id_string" } }

my apologies, this was not a production ready query but just a proof of concept probably pasted from kibana. i had changed this at 1f6ac14

for the second query i removed the range but query_string needs to stay as we are making a lucene query there and not just matching a field. 63f7aff

libraries/plugins/elasticsearch/elasticsearch_plugin.cpp

pmconrad · 2019-07-26T15:14:46Z

libraries/plugins/elasticsearch/elasticsearch_plugin.cpp

@@ -59,6 +58,8 @@ class elasticsearch_plugin_impl
      std::string _elasticsearch_index_prefix = "bitshares-";
      bool _elasticsearch_operation_object = false;
      uint32_t _elasticsearch_start_es_after_block = 0;
+      bool _elasticsearch_operation_string = true;
+      std::string _elasticsearch_mode = "only_save";


For more efficient comparison, make this an enum type.

It is a bit tricky to do this because the boost::program_options will not accept the enum but basically just a string, number or boolean.
I picked the number option to do a static cast to the enum by checking the upper boundary to avoid inserting invalid options.

Implemented at de76301

changed the exception introduced here to graphene::chain::plugin_exception in the context of 5838a38

pmconrad · 2019-07-26T15:16:39Z

libraries/plugins/elasticsearch/elasticsearch_plugin.cpp

+
+   if(my->_elasticsearch_mode != "only_query") {
+      if (my->_elasticsearch_mode == "all")
+         my->_elasticsearch_operation_string = true;


Silently overriding this will cause confusion.
IMO add log output if user has configured this to false. Or perhaps fail to make this very explicit.

Agree, i am now not allowing the mode to be all if operation_string is false so no silent changes are made.

5838a38

pmconrad · 2019-08-01T15:27:56Z

libraries/plugins/elasticsearch/elasticsearch_plugin.cpp

-         ("elasticsearch-mode", boost::program_options::value<std::string>(),
-               "Mode of operation: only_save, only_query, all(only_save)")
+         ("elasticsearch-mode", boost::program_options::value<uint16_t>(),
+               "Mode of operation: only_save(0), only_query(1), all(2) - Default: 0")


I'd use values 1,2,3 so the option behaves like a bitset. But that's just me.

libraries/app/api.cpp

…core into issue1682

abitmore · 2021-12-29T23:25:43Z

libraries/plugins/elasticsearch/elasticsearch_plugin.cpp

+      "query": {
+         "match":
+         {
+            "account_history.operation_id": )" + operation_id_string + R"("


A " is missing before )" + operation_id_string.

It seems this code is not used anywhere though.

@sschiessl-bcp I found that you were in the conversation, do you remember anything about this?

Update: fixed in #2573 via 8cf4144.

serve get_account_history with elasticsearch plugin

10a382f

pmconrad suggested changes Apr 20, 2019

View reviewed changes

libraries/plugins/elasticsearch/elasticsearch_plugin.cpp Outdated Show resolved Hide resolved

libraries/app/api.cpp Outdated Show resolved Hide resolved

libraries/app/api.cpp Outdated Show resolved Hide resolved

libraries/app/api.cpp Outdated Show resolved Hide resolved

replace IdToString

00496a9

abitmore added this to In development in Feature Release (3.1.0) via automation Apr 23, 2019

abitmore added this to the 3.1.0 - Feature Release milestone Apr 23, 2019

ryanRfox added this to In development in Feature Release (3.2.0) via automation Apr 23, 2019

ryanRfox removed this from In development in Feature Release (3.1.0) Apr 23, 2019

ryanRfox modified the milestones: 3.1.0 - Feature Release, 3.2.0 - Feature Release Apr 23, 2019

pmconrad modified the milestones: 3.2.0 - Feature Release, 3.3.0 - Feature Release Jun 19, 2019

pmconrad added this to In development in Feature Release (3.3.0) via automation Jun 19, 2019

pmconrad removed this from In development in Feature Release (3.2.0) Jun 19, 2019

oxarbitrage added 3 commits July 15, 2019 13:45

add is_plugin_enabled to application

2a6d7b2

add elasticsearch-mode command for save, query or all

1737d45

execute elasticsearch account_history in a new thread

6dda9fe

pmconrad reviewed Jul 26, 2019

View reviewed changes

oxarbitrage added 9 commits July 31, 2019 10:16

simplify conditional

26e9f16

capture only what is necessary in lambda

4fac171

wrap command line option long lines

29bf975

make mode an enum for comparison performance

de76301

throw when mode::all if elasticsearch-operation-string is false

5838a38

change ES query

1f6ac14

remove range from second ES query

63f7aff

add get_running_mode() to plugin and use it in api, fix capture

a7fc6b0

Merge branch 'develop' into issue1682

13d8034

pmconrad reviewed Aug 1, 2019

View reviewed changes

libraries/app/api.cpp Outdated Show resolved Hide resolved

oxarbitrage added 2 commits August 1, 2019 14:49

create only 1 thread to serve ES queries

07de45e

Merge branch 'issue1682' of https://github.com/oxarbitrage/bitshares-…

f1bedc7

…core into issue1682

jmjatlanta moved this from In development to In testing in Feature Release (3.3.0) Aug 7, 2019

MichelSantos mentioned this pull request Aug 9, 2019

Release Notes: BitShares Core 3.3.0 #1892

Closed

pmconrad approved these changes Aug 14, 2019

View reviewed changes

oxarbitrage merged commit cea7b74 into bitshares:develop Aug 14, 2019

Feature Release (3.3.0) automation moved this from In testing to Done Aug 14, 2019

oxarbitrage mentioned this pull request Aug 14, 2019

ES Plugin: Get operation by ID #1682

Closed

17 tasks

gladcow mentioned this pull request Dec 6, 2019

ElasticSearch Plugin to support for Peerplays peerplays-network/peerplays#161

Closed

16 tasks

abitmore reviewed Dec 29, 2021

View reviewed changes

abitmore mentioned this pull request Jan 8, 2023

Use ElasticSearch 8.5.3 in Github Actions, remove Ubuntu 18 #2707

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

elasticsearch history api #1682 #1725

elasticsearch history api #1682 #1725

oxarbitrage commented Apr 20, 2019

pmconrad Jul 26, 2019

pmconrad Jul 26, 2019

pmconrad Jul 26, 2019

pmconrad Jul 26, 2019

sschiessl-bcp Jul 29, 2019

oxarbitrage Jul 31, 2019

oxarbitrage Jul 31, 2019

pmconrad Jul 26, 2019

oxarbitrage Jul 31, 2019 •

edited

Loading

oxarbitrage Jul 31, 2019

pmconrad Jul 26, 2019

oxarbitrage Jul 31, 2019

pmconrad Aug 1, 2019

abitmore Dec 29, 2021 •

edited

Loading

elasticsearch history api #1682 #1725

elasticsearch history api #1682 #1725

Conversation

oxarbitrage commented Apr 20, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oxarbitrage Jul 31, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abitmore Dec 29, 2021 • edited Loading

Choose a reason for hiding this comment

oxarbitrage Jul 31, 2019 •

edited

Loading

abitmore Dec 29, 2021 •

edited

Loading