Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: synchronize async tasks write #818

Merged
merged 2 commits into from
Aug 5, 2022
Merged

Conversation

jyeshe
Copy link
Member

@jyeshe jyeshe commented Aug 2, 2022

What

  • Runs aex9 tasks asynchronously
  • Purges aex9 in-memory and persisted records belonging to invalidated blocks
  • Synchronizes the writing of async task result with the main sync database transaction so that if the task finishes later it writes directly to the database.
  • Avoid rerunning the tasks for on-db sync (use previous in-memory task result to write to database)

Why

Refs #749

Validation steps

  • test/ae_mdw/sync/async_tasks/update_aex9_state_test.exs
    • validates writing to async (in-memory) store
  • test/ae_mdw/sync/async_tasks/store_test.exs
    • validates not to insert a task when it had been added before (if added to on-memory doesn't do for on-db sync)
  • test/ae_mdw/db/state_test.exs
    • validates the synchronization to write to memory or to disk
    • validates that if task was enqueued by on-memory sync it does not run again during on-db sync
    • validates that commit to database uses previously run task results
    • validates that pending tasks are only persisted during commit to database
  • test/ae_mdw/db/sync/async_tasks/producer_consumer_test.exs
    • validates enqueuing and dequeuing tasks
  • test/ae_mdw/db/sync/async_tasks/producer_test.exs
    • validates that pending tasks are persisted after saving in case of server restart

Notes

Next tasks:

  1. Keep the sorting while enqueuing with State
  2. Allow enqueuing tasks to rerun if it's in processing state
  3. Avoid rerunning the tasks during in-memory sync by detecting invalidations outside of the task
  4. After 3, cache next hashes in a way to avoid walking through the blocks repeatedly for every task call.

@jyeshe jyeshe force-pushed the synchronized-async-tasks branch 2 times, most recently from bf34a85 to 80c3926 Compare August 3, 2022 08:14
@jyeshe jyeshe self-assigned this Aug 3, 2022
@jyeshe jyeshe force-pushed the synchronized-async-tasks branch 6 times, most recently from e6a9eee to 13e7ab6 Compare August 4, 2022 10:40

state
|> State.put(Model.Aex9AccountPresence, m_acc_presence)
|> State.put(Model.IdxAex9AccountPresence, m_idx_presence)
Copy link
Member Author

@jyeshe jyeshe Aug 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IdxAex9AccountPresence to be removed from database later to allow reusing previous snapshots without faking the table locally.

@jyeshe jyeshe force-pushed the synchronized-async-tasks branch 2 times, most recently from a8028fa to fc02742 Compare August 4, 2022 16:39
@jyeshe jyeshe marked this pull request as ready for review August 5, 2022 11:48
@@ -1,6 +1,7 @@
alias AeMdw.Db.Model
alias AeMdw.Db.Name
alias AeMdw.Db.Util
alias AeMdw.Db.State
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙏

%{last_db_kbi: last_db_kbi} = state
) do
# IO.inspect {kbi, Database.exists?(Model.Block, block_index)}, label: :write_mutations
# IO.inspect {kbi, last_db_kbi}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug comments

@jyeshe jyeshe merged commit 407576e into master Aug 5, 2022
@jyeshe jyeshe deleted the synchronized-async-tasks branch August 5, 2022 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants