internal error when inserting/deleting many jobs with binlog #38

Closed
kr opened this Issue May 1, 2010 · 2 comments

Comments

Projects
None yet
1 participant
@kr

This comment has been minimized.

Show comment
Hide comment
@kr

kr May 10, 2010

Owner

Add a new invariant to reserved space.

A DELETE record can be written at any time, including times when we are
unable to shuffle reserved bytes around. So, to make sure that we don't
lose any reserved bytes, the current binlog must always have reserved
space for some number of complete DELETE records. This lets us fill up
the current binlog exactly with no lost reserved bytes. The only time we
are able to shuffle things around to maintain this property is when
reserving bytes.

We used to try to accomplish this by checking the reserved size of the
current_binlog after each reservation, but that isn't sufficient. The
next binlog can become current and run out of space without any
reservations taking place. This would force us to lose bytes.

Now we check that the current_binlog and all future binlog reservation
sizes are suitable to become the current binlog and fill up exactly.

Closed by 4d56a66.

Owner

kr commented May 10, 2010

Add a new invariant to reserved space.

A DELETE record can be written at any time, including times when we are
unable to shuffle reserved bytes around. So, to make sure that we don't
lose any reserved bytes, the current binlog must always have reserved
space for some number of complete DELETE records. This lets us fill up
the current binlog exactly with no lost reserved bytes. The only time we
are able to shuffle things around to maintain this property is when
reserving bytes.

We used to try to accomplish this by checking the reserved size of the
current_binlog after each reservation, but that isn't sufficient. The
next binlog can become current and run out of space without any
reservations taking place. This would force us to lose bytes.

Now we check that the current_binlog and all future binlog reservation
sizes are suitable to become the current binlog and fill up exactly.

Closed by 4d56a66.

@kr

This comment has been minimized.

Show comment
Hide comment
@kr

kr May 22, 2010

Owner

Just to clarify, this bug would only get triggered in a rare situation: the client must issue enough contiguous delete commands to fill up an entire binlog segment, from beginning to end. If any commands that allocate binlog space are interspersed, the bug is not triggered.

Owner

kr commented May 22, 2010

Just to clarify, this bug would only get triggered in a rare situation: the client must issue enough contiguous delete commands to fill up an entire binlog segment, from beginning to end. If any commands that allocate binlog space are interspersed, the bug is not triggered.

dustin pushed a commit that referenced this issue Apr 19, 2011

Add a new invariant to reserved space.
A DELETE record can be written at any time, including times when we are
unable to shuffle reserved bytes around. So, to make sure that we don't
lose any reserved bytes, the current binlog must always have reserved
space for some number of complete DELETE records. This lets us fill up
the current binlog exactly with no lost reserved bytes. The only time we
are able to shuffle things around to maintain this property is when
reserving bytes.

We used to try to accomplish this by checking the reserved size of the
current_binlog after each reservation, but that isn't sufficient. The
next binlog can become current and run out of space without any
reservations taking place. This would force us to lose bytes.

Now we check that the current_binlog and *all future binlog* reservation
sizes are suitable to become the current binlog and fill up exactly.

Closes gh-38.

This issue was closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment