Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

memtx: replication JOIN should work off a checkpoint, not a snapshot file #1271

Closed
kostja opened this issue Jan 13, 2016 · 4 comments
Closed

memtx: replication JOIN should work off a checkpoint, not a snapshot file #1271

kostja opened this issue Jan 13, 2016 · 4 comments
Labels
feature A new functionality
Milestone
2.3.1

Comments

@kostja
Copy link
Contributor

kostja commented Jan 13, 2016

Replication JOIN in MemTX engine should create a checkpoint on the fly, and send data from the in-memory state, rather than a disk file. This is a pre-requisite task for #1270 and continuous snapshotting in memtx.
This was referenced Jan 13, 2016
Open
Closed
@kostja kostja added this to the 1.7.0 milestone Feb 18, 2016
@kostja kostja added the feature A new functionality label Feb 18, 2016
@kostja kostja modified the milestones: 1.7.1, 1.7.2 Jul 11, 2016
@rtsisyk rtsisyk modified the milestones: 1.7.3, 1.7.2 Aug 2, 2016
@rtsisyk
Copy link
Contributor

rtsisyk commented Aug 2, 2016

rolling feature, moving to 1.7.3

@kostja kostja modified the milestones: 1.8.0, 1.7.3 Sep 27, 2016
@rtsisyk rtsisyk mentioned this issue Feb 14, 2017
Closed
@kostja kostja modified the milestones: 1.8.0, 1.8.1 Mar 31, 2017
@kyukhin
Copy link
Contributor

kyukhin commented Aug 17, 2017

This issue'll help to avoid creation of initial snapshots.
And thus will allow not to create any snapshot in wal_mode=none mode.

I'll set it as prio1.

@kyukhin kyukhin added the prio1 label Aug 17, 2017
@kostja kostja modified the milestones: 1.8.3, 1.8.4, wishlist Oct 27, 2017
@kostja kostja assigned rtokarev and rtsisyk and unassigned alyapunov Oct 27, 2017
@Gerold103 Gerold103 assigned Gerold103 and unassigned rtsisyk Feb 9, 2018
@locker locker self-assigned this Aug 10, 2019
@kyukhin kyukhin modified the milestones: wishlist, 2.3.1 Aug 12, 2019
locker added a commit that referenced this issue Aug 14, 2019
@locker
test: disable replication/on_schema_init
ff28d6f 
The test uses box.on_schema_init to install space.before_replace trigger
that changes the engine/locality of a space received by a replica. This
works, only because we don't make a snapshot after creating those spaces
on the master so that they are relayed from an xlog. If we added
box.snapshot(), the test would fail, because space.before_replace
trigger isn't run for changes received on initial join (see #4417).
Once we make the initial join stage work off the current read view
rather than the last snapshot (see #1271), the test will fail as well.
Let's disable the test until the issue is resolved.

Needed for #1271
See #4417
locker added a commit that referenced this issue Aug 14, 2019
@locker
relay: join new replicas off read view
febc90c 
Historically, we join a new replica off the last checkpoint. As a
result, we must always keep the last memtx snapshot and all vinyl data
files corresponding to it. Actually, there's no need to use the last
checkpoint for joining a replica. Instead we can use the current read
view as both memtx and vinyl support it. This should speed up the
process of joining a new replica, because we don't need to replay all
xlogs written after the last checkpoint, only those that are accumulated
while we are relaying the current read view. This should also allow us
to avoid creating a snapshot file on bootstrap, because the only reason
why we need it is allowing joining replicas. Besides, this is a step
towards decoupling the vinyl metadata log from checkpointing in
particular and from xlogs in general.

Closes #1271
locker added a commit that referenced this issue Aug 19, 2019
@locker
relay: join new replicas off read view
960c54b 
Historically, we join a new replica off the last checkpoint. As a
result, we must always keep the last memtx snapshot and all vinyl data
files corresponding to it. Actually, there's no need to use the last
checkpoint for joining a replica. Instead we can use the current read
view as both memtx and vinyl support it. This should speed up the
process of joining a new replica, because we don't need to replay all
xlogs written after the last checkpoint, only those that are accumulated
while we are relaying the current read view. This should also allow us
to avoid creating a snapshot file on bootstrap, because the only reason
why we need it is allowing joining replicas. Besides, this is a step
towards decoupling the vinyl metadata log from checkpointing in
particular and from xlogs in general.

Closes #1271
locker added a commit that referenced this issue Aug 19, 2019
@locker
relay: join new replicas off read view
9ed5cd2 
Historically, we join a new replica off the last checkpoint. As a
result, we must always keep the last memtx snapshot and all vinyl data
files corresponding to it. Actually, there's no need to use the last
checkpoint for joining a replica. Instead we can use the current read
view as both memtx and vinyl support it. This should speed up the
process of joining a new replica, because we don't need to replay all
xlogs written after the last checkpoint, only those that are accumulated
while we are relaying the current read view. This should also allow us
to avoid creating a snapshot file on bootstrap, because the only reason
why we need it is allowing joining replicas. Besides, this is a step
towards decoupling the vinyl metadata log from checkpointing in
particular and from xlogs in general.

Closes #1271
locker added a commit that referenced this issue Aug 20, 2019
@locker
relay: join new replicas off read view
220859a 
Historically, we join a new replica off the last checkpoint. As a
result, we must always keep the last memtx snapshot and all vinyl data
files corresponding to it. Actually, there's no need to use the last
checkpoint for joining a replica. Instead we can use the current read
view as both memtx and vinyl support it. This should speed up the
process of joining a new replica, because we don't need to replay all
xlogs written after the last checkpoint, only those that are accumulated
while we are relaying the current read view. This should also allow us
to avoid creating a snapshot file on bootstrap, because the only reason
why we need it is allowing joining replicas. Besides, this is a step
towards decoupling the vinyl metadata log from checkpointing in
particular and from xlogs in general.

Closes #1271
locker added a commit that referenced this issue Aug 20, 2019
@locker
relay: join new replicas off read view
88009e3 
Historically, we join a new replica off the last checkpoint. As a
result, we must always keep the last memtx snapshot and all vinyl data
files corresponding to it. Actually, there's no need to use the last
checkpoint for joining a replica. Instead we can use the current read
view as both memtx and vinyl support it. This should speed up the
process of joining a new replica, because we don't need to replay all
xlogs written after the last checkpoint, only those that are accumulated
while we are relaying the current read view. This should also allow us
to avoid creating a snapshot file on bootstrap, because the only reason
why we need it is allowing joining replicas. Besides, this is a step
towards decoupling the vinyl metadata log from checkpointing in
particular and from xlogs in general.

Closes #1271
locker added a commit that referenced this issue Aug 27, 2019
@locker
relay: join new replicas off read view
fd1cc7c 
Historically, we join a new replica off the last checkpoint. As a
result, we must always keep the last memtx snapshot and all vinyl data
files corresponding to it. Actually, there's no need to use the last
checkpoint for joining a replica. Instead we can use the current read
view as both memtx and vinyl support it. This should speed up the
process of joining a new replica, because we don't need to replay all
xlogs written after the last checkpoint, only those that are accumulated
while we are relaying the current read view. This should also allow us
to avoid creating a snapshot file on bootstrap, because the only reason
why we need it is allowing joining replicas. Besides, this is a step
towards decoupling the vinyl metadata log from checkpointing in
particular and from xlogs in general.

Closes #1271
kostja pushed a commit to kostja/tarantool that referenced this issue Sep 6, 2019
@locker @kostja
relay: join new replicas off read view
64a9103 
Historically, we join a new replica off the last checkpoint. As a
result, we must always keep the last memtx snapshot and all vinyl data
files corresponding to it. Actually, there's no need to use the last
checkpoint for joining a replica. Instead we can use the current read
view as both memtx and vinyl support it. This should speed up the
process of joining a new replica, because we don't need to replay all
xlogs written after the last checkpoint, only those that are accumulated
while we are relaying the current read view. This should also allow us
to avoid creating a snapshot file on bootstrap, because the only reason
why we need it is allowing joining replicas. Besides, this is a step
towards decoupling the vinyl metadata log from checkpointing in
particular and from xlogs in general.

Closes tarantool#1271
kostja pushed a commit to kostja/tarantool that referenced this issue Sep 9, 2019
@locker @kostja
relay: join new replicas off read view
bc5ba91 
Historically, we join a new replica off the last checkpoint. As a
result, we must always keep the last memtx snapshot and all vinyl data
files corresponding to it. Actually, there's no need to use the last
checkpoint for joining a replica. Instead we can use the current read
view as both memtx and vinyl support it. This should speed up the
process of joining a new replica, because we don't need to replay all
xlogs written after the last checkpoint, only those that are accumulated
while we are relaying the current read view. This should also allow us
to avoid creating a snapshot file on bootstrap, because the only reason
why we need it is allowing joining replicas. Besides, this is a step
towards decoupling the vinyl metadata log from checkpointing in
particular and from xlogs in general.

Closes tarantool#1271
kostja added a commit to kostja/tarantool that referenced this issue Sep 11, 2019
@kostja
Merge branch 'dv/tarantoolgh-1271'
6fadc8c
kostja pushed a commit to kostja/tarantool that referenced this issue Sep 11, 2019
@locker @kostja
relay: join new replicas off read view
7de7306 
Historically, we join a new replica off the last checkpoint. As a
result, we must always keep the last memtx snapshot and all vinyl data
files corresponding to it. Actually, there's no need to use the last
checkpoint for joining a replica. Instead we can use the current read
view as both memtx and vinyl support it. This should speed up the
process of joining a new replica, because we don't need to replay all
xlogs written after the last checkpoint, only those that are accumulated
while we are relaying the current read view. This should also allow us
to avoid creating a snapshot file on bootstrap, because the only reason
why we need it is allowing joining replicas. Besides, this is a step
towards decoupling the vinyl metadata log from checkpointing in
particular and from xlogs in general.

Closes tarantool#1271
sergepetrenko added a commit that referenced this issue Nov 1, 2019
@sergepetrenko
box: update comment describing join protocol
95c99ef 
The comment states that relay sends the latest snapshot to replica
during initial join, however, this was changed in commit
6332aca (relay: join new replicas off
read view).
Now relay sends rows from the read view created at the moment of join.
Update the comment to match.

Follow-up #1271
@viciious
Copy link

viciious commented Nov 15, 2019 •
edited
Loading

So how does this feature work? Does the newly introduced RO view require additional RAM or CPU to function?

@kostja
Copy link
Contributor Author

kostja commented Nov 15, 2019

It uses the same technique as when taking a snapshot - so it does require a little extra ram, yes, but this is the same amount as is usually required to take a snapshot.

sergepetrenko added a commit that referenced this issue Dec 13, 2019
@sergepetrenko
box: update comment describing join protocol
f1bf74f 
The comment states that relay sends the latest snapshot to replica
during initial join, however, this was changed in commit
6332aca (relay: join new replicas off
read view).
Now relay sends rows from the read view created at the moment of join.
Update the comment to match.

Follow-up #1271
sergepetrenko added a commit that referenced this issue Dec 20, 2019
@sergepetrenko
box: update comment describing join protocol
145f814 
The comment states that relay sends the latest snapshot to replica
during initial join, however, this was changed in commit
6332aca (relay: join new replicas off
read view).
Now relay sends rows from the read view created at the moment of join.
Update the comment to match.

Follow-up #1271
sergepetrenko added a commit that referenced this issue Dec 25, 2019
@sergepetrenko
box: update comment describing join protocol
60a6cf5 
The comment states that relay sends the latest snapshot to replica
during initial join, however, this was changed in commit
6332aca (relay: join new replicas off
read view).
Now relay sends rows from the read view created at the moment of join.
Update the comment to match.

Follow-up #1271
sergepetrenko added a commit that referenced this issue Dec 25, 2019
@sergepetrenko
box: update comment describing join protocol
8c6525b 
The comment states that relay sends the latest snapshot to replica
during initial join, however, this was changed in commit
6332aca (relay: join new replicas off
read view).
Now relay sends rows from the read view created at the moment of join.
Update the comment to match.

Follow-up #1271
sergepetrenko added a commit that referenced this issue Dec 27, 2019
@sergepetrenko
box: update comment describing join protocol
b815070 
The comment states that relay sends the latest snapshot to replica
during initial join, however, this was changed in commit
6332aca (relay: join new replicas off
read view).
Now relay sends rows from the read view created at the moment of join.
Update the comment to match.

Follow-up #1271
sergepetrenko added a commit that referenced this issue Dec 28, 2019
@sergepetrenko
box: update comment describing join protocol
f3ed60f 
The comment states that relay sends the latest snapshot to replica
during initial join, however, this was changed in commit
6332aca (relay: join new replicas off
read view).
Now relay sends rows from the read view created at the moment of join.
Update the comment to match.

Follow-up #1271
kyukhin pushed a commit that referenced this issue Dec 30, 2019
@sergepetrenko @kyukhin
box: update comment describing join protocol
d3031e4 
The comment states that relay sends the latest snapshot to replica
during initial join, however, this was changed in commit
6332aca (relay: join new replicas off
read view).
Now relay sends rows from the read view created at the moment of join.
Update the comment to match.

Follow-up #1271
avtikhon pushed a commit to avtikhon/tarantool that referenced this issue Feb 3, 2020
@sergepetrenko @avtikhon
box: update comment describing join protocol
aa55910 
The comment states that relay sends the latest snapshot to replica
during initial join, however, this was changed in commit
6332aca (relay: join new replicas off
read view).
Now relay sends rows from the read view created at the moment of join.
Update the comment to match.

Follow-up tarantool#1271
@Gerold103 Gerold103 mentioned this issue Apr 5, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature A new functionality
Projects
None yet
Milestone
2.3.1
Development

No branches or pull requests
9 participants
@Totktonada @kostja @rtokarev @locker @viciious @rtsisyk @Gerold103 @alyapunov @kyukhin