batch insertion events in archive database #14779

ghost-not-in-the-shell · 2024-01-03T16:55:51Z

Explain your changes:
This PR does batch insertion for zkapp_events in archive db.

Since Account_update.Body.Events'.t is defined as field array list, which is ismorphic to a list of list of fields.

We could batch the insertion of field and field_array to optimize the speed of archiving max-cost zkapps.

we flatten the list of list of fields to get all the field elements
insert all the field elements in one query
construct a map M from field_id to field by querying against the zkapp_field table
use M and the list of list of fields to compute the list of list of field_ids
insert all list of list of field_ids in one query
construct a map M' from field_array_id to field_id array by querying against the zkapp_field_array table
use M' and the list of list of field_ids to compute the list of field_array_ids
insert the list of field_arrays

To patch the existing database use add_unique_constraints.sql

Explain how you tested your changes:
Tested locally and also on testworld-2-0

Checklist:

Dependency versions are unchanged
- Notify Velocity team if dependencies must change in CI
Modified the current draft of release notes with details on what is completed or incomplete within this project
Document code purpose, how to use it
- Mention expected invariants, implicit constraints
Tests were added for the new behavior
- Document test purpose, significance of failures
- Test names should reflect their purpose
All tests pass (CI will check this if you didn't)
Serialized types are in stable-versioned modules
Does this close issues? List them

Closes #0000

…MinaProtocol/mina into fix/archive-db-slowness

ghost-not-in-the-shell · 2024-01-04T22:37:29Z

!ci-build-me

…e-db-slowness

georgeee · 2024-01-05T16:43:26Z

src/app/archive/lib/processor.ml

    in
    let increment_nonce = body.increment_nonce in
    let%bind events_id =
-      Zkapp_events.add_if_doesn't_exist (module Conn) body.events
+      Metrics.time ~label:"Zkapp_events.add"


Do we need all of these Metrics.time?

georgeee · 2024-01-05T16:44:47Z

src/lib/mina_caqti/mina_caqti.ml

@@ -287,15 +287,19 @@ let select_insert_into_cols ~(select : string * 'select Caqti_type.t)
    @@ select_cols ~select:(fst select) ~table_name ?tannot ~cols:(fst cols) ()
    )
    value
-  >>= function


Previous code with >>= function looked cleaner

Ah. I changed it to add some more logs. Let me revert those changes.

georgeee · 2024-01-05T16:46:16Z

src/lib/mina_caqti/mina_caqti.ml

-        @@ insert_into_cols ~returning:(fst select) ~table_name ?tannot
-             ~cols:(fst cols) () )
-        value
+      let%map res =


Redundant let%map introduced

Again this change was for logging. Let me undo that.

georgeee · 2024-01-05T16:52:23Z

src/app/archive/lib/processor.ml

+                ()
+              >>| String.Map.of_alist_exn
+            in
+            let field_id_list_list =


Suggested change

let field_id_list_list =

field_id_list_list is not needed at all

I added a let binding here to make the code more readable. Just to give a name to the result of that computation.

georgeee · 2024-01-05T16:58:16Z

src/app/archive/lib/processor.ml

+              WHERE element_ids in (%s) |sql}
+          @@ sep_by_comma field_array_list
+        in
+        let module Field_array_map = Map.Make (struct


Move this module to the module-level context (so that it's not defined in the middle of function definition)?

georgeee · 2024-01-05T17:00:20Z

src/app/archive/lib/processor.ml

+        let%bind field_id_list_list =
+          if not @@ List.is_empty fields then
+            let field_insert =
+              sprintf


Do you think we can generalize the pattern and extract it as a separate function to reduce code duplication?

I'm taking about insert on conflict do nothing paired with select of some_field, id with conversion of it to map preserving the initial order.

It should be possible to do that. In fact, Mina_caqti.select_insert_into_cols did a similar abstraction over 2 queries. But for this one, it's a bit harder, because I was hand-written some of the query and didn't get to use caqti's typ stuff. Let me leave a comment around this part of code saying we could abstract this pattern out once I figure out how to use typ to convert between values and sql representations. I think we want to merge this soon so let create a ticket for do that.

Nevermind, I got most part of the refactor done. I left out the conversion from value to string.

ghost-not-in-the-shell · 2024-01-05T20:19:39Z

!ci-build-me

ghost-not-in-the-shell · 2024-01-05T20:53:37Z

!ci-build-me

deepthiskumar

Waiting for rampup8 release before landing this

ghost-not-in-the-shell · 2024-01-08T13:52:58Z

!ci-build-me

Include in rampup8

deepthiskumar · 2024-01-09T21:48:54Z

!ci-nightly-me

deepthiskumar · 2024-01-09T21:49:01Z

!ci-build-me

ghost-not-in-the-shell · 2024-01-10T12:07:09Z

!ci-build-me

ghost-not-in-the-shell · 2024-01-10T13:41:37Z

!ci-nightly-me

ghost-not-in-the-shell · 2024-01-10T15:42:16Z

!ci-nightly-me

ghost-not-in-the-shell · 2024-01-10T15:51:57Z

!ci-nightly-me

ghost-not-in-the-shell · 2024-01-10T20:49:17Z

!ci-build-me

deepthiskumar · 2024-01-10T21:15:32Z

!approved-for-mainnet

Tang Jiawei added 6 commits December 22, 2023 07:49

add timing logs to various functions

9ecf29e

add more logs to various transactions

79d9cda

add more logs for zkapp commands

a9347f5

add more logs to update body

4be459c

add more logs

d9f961c

add idx for element_ids

55433d3

ghost-not-in-the-shell requested a review from a team as a code owner January 3, 2024 16:55

georgeee approved these changes Jan 3, 2024

View reviewed changes

Tang Jiawei added 3 commits January 4, 2024 07:08

add an index for zkapp_events

153bfe4

batch insertion of events and actions

c64cba9

Merge branch 'log/add-timing-log-to-archive-processor' of github.com:…

42cb2f6

…MinaProtocol/mina into fix/archive-db-slowness

ghost-not-in-the-shell requested a review from a team as a code owner January 4, 2024 22:34

ghost-not-in-the-shell changed the title ~~add idx for element_ids~~ various changes to improve the performance of archive process Jan 4, 2024

Tang Jiawei added 2 commits January 5, 2024 22:48

add comments

2fb21f6

Merge branch 'rampup' of github.com:MinaProtocol/mina into fix/archiv…

6eeefb1

…e-db-slowness

georgeee requested changes Jan 5, 2024

View reviewed changes

Tang Jiawei added 3 commits January 6, 2024 03:01

remove unnecessary logs

49874ea

refactor sql code into mina_caqti

6e5f053

clean up

8734f96

ghost-not-in-the-shell changed the title ~~various changes to improve the performance of archive process~~ batch insertion events in archive database Jan 5, 2024

add sql file to patch the database

9fb47bb

georgeee approved these changes Jan 5, 2024

View reviewed changes

deepthiskumar previously requested changes Jan 6, 2024

View reviewed changes

Tang Jiawei and others added 2 commits January 8, 2024 21:51

add tracing of processor.add_block

db92754

Merge branch 'rampup' into fix/archive-db-slowness

22583bf

Merge branch 'rampup' into fix/archive-db-slowness

5213421

Merge branch 'rampup' into fix/archive-db-slowness

7e72362

ghost-not-in-the-shell changed the base branch from rampup to berkeley January 10, 2024 15:41

ghost-not-in-the-shell requested review from a team as code owners January 10, 2024 15:41

ghost-not-in-the-shell changed the base branch from berkeley to rampup January 10, 2024 15:42

ghost-not-in-the-shell changed the base branch from rampup to berkeley January 10, 2024 15:49

ghost-not-in-the-shell changed the base branch from berkeley to rampup January 10, 2024 15:53

joaosreis approved these changes Jan 10, 2024

View reviewed changes

Merge branch 'rampup' into fix/archive-db-slowness

49e407d

mitschabaude approved these changes Jan 10, 2024

View reviewed changes

ghost-not-in-the-shell merged commit ceaae53 into rampup Jan 10, 2024
36 checks passed

ghost-not-in-the-shell deleted the fix/archive-db-slowness branch January 10, 2024 22:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

batch insertion events in archive database #14779

batch insertion events in archive database #14779

ghost-not-in-the-shell commented Jan 3, 2024 •

edited

Loading

ghost-not-in-the-shell commented Jan 4, 2024

georgeee Jan 5, 2024

georgeee Jan 5, 2024

ghost-not-in-the-shell Jan 5, 2024

georgeee Jan 5, 2024

ghost-not-in-the-shell Jan 5, 2024

georgeee Jan 5, 2024

ghost-not-in-the-shell Jan 5, 2024

georgeee Jan 5, 2024

georgeee Jan 5, 2024

ghost-not-in-the-shell Jan 5, 2024 •

edited

Loading

ghost-not-in-the-shell Jan 5, 2024

ghost-not-in-the-shell commented Jan 5, 2024

ghost-not-in-the-shell commented Jan 5, 2024

deepthiskumar left a comment

ghost-not-in-the-shell commented Jan 8, 2024

deepthiskumar commented Jan 9, 2024

deepthiskumar commented Jan 9, 2024

ghost-not-in-the-shell commented Jan 10, 2024

ghost-not-in-the-shell commented Jan 10, 2024

ghost-not-in-the-shell commented Jan 10, 2024

ghost-not-in-the-shell commented Jan 10, 2024

ghost-not-in-the-shell commented Jan 10, 2024

deepthiskumar commented Jan 10, 2024

batch insertion events in archive database #14779

batch insertion events in archive database #14779

Conversation

ghost-not-in-the-shell commented Jan 3, 2024 • edited Loading

ghost-not-in-the-shell commented Jan 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost-not-in-the-shell Jan 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost-not-in-the-shell commented Jan 5, 2024

ghost-not-in-the-shell commented Jan 5, 2024

deepthiskumar left a comment

Choose a reason for hiding this comment

ghost-not-in-the-shell commented Jan 8, 2024

deepthiskumar commented Jan 9, 2024

deepthiskumar commented Jan 9, 2024

ghost-not-in-the-shell commented Jan 10, 2024

ghost-not-in-the-shell commented Jan 10, 2024

ghost-not-in-the-shell commented Jan 10, 2024

ghost-not-in-the-shell commented Jan 10, 2024

ghost-not-in-the-shell commented Jan 10, 2024

deepthiskumar commented Jan 10, 2024

ghost-not-in-the-shell commented Jan 3, 2024 •

edited

Loading

ghost-not-in-the-shell Jan 5, 2024 •

edited

Loading