Batch apply dml events #352

Merged
merged 8 commits into from Jan 9, 2017

Projects

None yet

1 participant

@shlomi-noach
Collaborator

This PR applies DML events (events read from the binary logs) in batches, i.e. it aggregates events from the binary log and writes multiple DMLs (insert, delete, update) in a single transaction.

This has the potential to increase gh-ost's write capacity. This would potentially help #348, though we have not yet diagnosed #348 enough.

The PR introduces a --dml-batch-size flag, default 10.

Noteworthy about the implementation: gh-ost does not wait for events to arrive in order to batch. The logic is as follows:

  • If there's nothing in the event queue (coming from the binary log), copy some rows
  • If there's an event to handle in the events queue, read it. Further determine how many more available events are in the queue. Of these, read at most dml-batch-size. Of these, only handle the prefix that is true DML events (distinguished from gh-ost's internal events).

So the logic doesn't block on waiting for "more events to arrive", but is greedy, and takes as many available events as there are, up to some limit, and only in suh case that there are events to apply.

@shlomi-noach shlomi-noach deployed to production/ghost-mysql001 Jan 3, 2017 Active
@shlomi-noach shlomi-noach deployed to production/ghost-mysql001 Jan 4, 2017 Active
@shlomi-noach shlomi-noach deployed to production/ghost-mysql001 Jan 4, 2017 Active
@shlomi-noach shlomi-noach deployed to production/ghost-mysql001 Jan 5, 2017 Active
@shlomi-noach
Collaborator

This was subjected to prod testing for the past days and behaves as expected; merging.

@shlomi-noach shlomi-noach merged commit 8d8ef34 into master Jan 9, 2017

1 check passed

gh-ost-build-deploy-tarball Build #5123521 succeeded in 5s
Details
@shlomi-noach shlomi-noach deleted the batch-apply-dml-events branch Jan 9, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment