/foundationdb

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Share log mutations between backups and DRs which have the same backu… #3

Merged
merged 3 commits into from Mar 19, 2018

Conversation

Reviewers

@etschannen etschannen

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@yichic @alecgrieser @etschannen
@yichic
Contributor

yichic commented Feb 20, 2018

…p range
@etschannen

etschannen reviewed Feb 20, 2018
View changes

fdbclient/DatabaseBackupAgent.actor.cpp Outdated
@@ -1141,7 +1273,7 @@ namespace dbBackup {
return Void();
}
ACTOR static Future<Key> addTask(Reference<ReadYourWritesTransaction> tr, Reference<TaskBucket> taskBucket, Key logUid, Key backupUid, Key keyAddPrefix, Key keyRemovePrefix, Key keyConfigBackupRanges, Key tagName, TaskCompletionKey completionKey, Reference<TaskFuture> waitFor = Reference<TaskFuture>(), bool databasesInSync=false)
ACTOR static Future<Key> addTask(Reference<ReadYourWritesTransaction> tr, Reference<TaskBucket> taskBucket, Key logUid, Key backupUid, /*Key destUid,*/ Key keyAddPrefix, Key keyRemovePrefix, Key keyConfigBackupRanges, Key tagName, TaskCompletionKey completionKey, Reference<TaskFuture> waitFor = Reference<TaskFuture>(), bool databasesInSync=false)

This comment has been minimized.

Sign in to view
Copy link
@etschannen

etschannen Feb 20, 2018

Contributor

remove comment
@etschannen

etschannen reviewed Feb 20, 2018
View changes

fdbclient/FileBackupAgent.actor.cpp Outdated
@@ -1496,7 +1497,7 @@ namespace fileBackup {
.detail("ScheduledVersion", scheduledVersion)
.detail("BeginKey", range.begin.printable())
.detail("EndKey", range.end.printable())
.suppressFor(2);
.suppressFor(2, true);

This comment has been minimized.

Sign in to view
Copy link
@etschannen

etschannen Feb 20, 2018

Contributor

unneeded
@etschannen

etschannen reviewed Feb 20, 2018
View changes

fdbserver/SimulatedCluster.actor.cpp
Reference<Cluster> cluster = Cluster::createCluster(connFile, -1);
Database cx = cluster->createDatabase(LiteralStringRef("DB")).get();
Reference<ClusterConnectionFile> extraFile(new ClusterConnectionFile(*g_simulator.extraDB));
Reference<Cluster> extraCluster = Cluster::createCluster(extraFile, -1);
state Database extraDB = extraCluster->createDatabase(LiteralStringRef("DB")).get();
TraceEvent("StartingBackupAgents").detail("connFile", connFile->getConnectionString().toString()).detail("extraString", extraFile->getConnectionString().toString());
TraceEvent("StartingDrAgents").detail("connFile", connFile->getConnectionString().toString()).detail("extraString", extraFile->getConnectionString().toString());

This comment has been minimized.

Sign in to view
Copy link
@etschannen

etschannen Feb 20, 2018

Contributor

add a similar trace event for backup agents
@etschannen

etschannen reviewed Feb 20, 2018
View changes

fdbserver/SimulatedCluster.actor.cpp
Void _ = wait(delay(1.0));
}
TraceEvent("StoppingBackupAgents");
TraceEvent("StoppingDrAgents");

This comment has been minimized.

Sign in to view
Copy link
@etschannen

etschannen Feb 20, 2018

Contributor

add a similar trace event for backup
@etschannen

etschannen reviewed Feb 20, 2018
View changes

tests/slow/SharedBackupCorrectness.txt Outdated
shareLogRange=true
performRestore=false
testName=BackupToDBCorrectness

This comment has been minimized.

Sign in to view
Copy link
@etschannen

etschannen Feb 20, 2018

Contributor

something needs to restore
@etschannen

etschannen reviewed Feb 21, 2018
View changes

fdbclient/BackupAgentBase.actor.cpp Outdated
}
Future<Void> clearLogRanges(Reference<ReadYourWritesTransaction> tr, bool clearVersionHistory, Key logUidValue, Key destUidValue, Version beginVersion, Version endVersion) {
return _clearLogRanges(tr, clearVersionHistory, logUidValue, destUidValue, beginVersion, endVersion);

This comment has been minimized.

Sign in to view
Copy link
@etschannen

etschannen Feb 21, 2018

Contributor

add a comment about the fact that endVersion - beginVersion should not be too large
@etschannen

etschannen reviewed Feb 21, 2018
View changes

fdbclient/BackupAgentBase.actor.cpp Outdated
if (!v.present()) {
return Void();
}

This comment has been minimized.

Sign in to view
Copy link
@etschannen

etschannen Feb 21, 2018

Contributor

use logUid instead of beginVersion for finding ourself in the list of versions
add an ASSERT_WE_THINK that v matches beginVersion, and matches ourself in the backupLatestVersions list

This comment has been minimized.

Sign in to view
Copy link
@etschannen

etschannen Feb 21, 2018

Contributor

do not get backupLatestVersionsKey separately; loop through backupVersions and check for ourself in those results to accomplish the same thing
@etschannen

etschannen reviewed Feb 21, 2018
View changes

fdbclient/BackupAgentBase.actor.cpp Outdated
if (currVersion > beginVersion) {
if (currVersion < nextSmallestVersion) {
nextSmallestVersion = currVersion;
}

This comment has been minimized.

Sign in to view
Copy link
@etschannen

etschannen Feb 21, 2018

Contributor

use std::min for code clarity
@etschannen

etschannen reviewed Feb 21, 2018
View changes

fdbclient/DatabaseBackupAgent.actor.cpp
@@ -609,7 +625,7 @@ namespace dbBackup {
tr.addReadConflictRange(singleKeyRange(kv.key));
first = false;
}
tr.set(kv.key.removePrefix(backupLogKeys.begin).withPrefix(applyLogKeys.begin), kv.value);
tr.set(kv.key.removePrefix(backupLogKeys.begin).removePrefix(task->params[BackupAgentBase::destUid]).withPrefix(task->params[BackupAgentBase::keyConfigLogUid]).withPrefix(applyLogKeys.begin), kv.value);
bytesSet += kv.expectedSize() - backupLogKeys.begin.expectedSize() + applyLogKeys.begin.expectedSize();

This comment has been minimized.

Sign in to view
Copy link
@etschannen

etschannen Feb 21, 2018

Contributor

calculate the two withPrefixs into a single string
use mutateString to directly change the bytes in the key instead of using withPrefix to copy the bytes into a new string
@etschannen

etschannen reviewed Feb 21, 2018
View changes

fdbclient/DatabaseBackupAgent.actor.cpp Outdated
beginVersion = BinaryReader::fromStringRef<Version>(bVersion.get(), Unversioned());
Void _ = wait(tr->commit());
endVersion = tr->getCommittedVersion();
break;

This comment has been minimized.

Sign in to view
Copy link
@etschannen

etschannen Feb 21, 2018

Contributor

for safety use committedVersion + 1
@etschannen

etschannen reviewed Feb 21, 2018
View changes

fdbclient/DatabaseBackupAgent.actor.cpp Outdated
Void _ = wait(tr->commit());
break;
} catch (Error &e) {
Void _ = wait(srcTr->onError(e));

This comment has been minimized.

Sign in to view
Copy link
@etschannen

etschannen Feb 21, 2018

Contributor

tr not srcTr
@etschannen

etschannen reviewed Feb 21, 2018
View changes

fdbclient/DatabaseBackupAgent.actor.cpp Outdated
Void _ = wait(srcTr->commit());
endVersion = srcTr->getCommittedVersion();

This comment has been minimized.

Sign in to view
Copy link
@etschannen

etschannen Feb 21, 2018

Contributor

+1 committed version for safety
@etschannen

etschannen reviewed Feb 21, 2018
View changes

fdbclient/FileBackupAgent.actor.cpp Outdated
}
}
}

This comment has been minimized.

Sign in to view
Copy link
@etschannen

etschannen Feb 21, 2018

Contributor

factor this transaction into a separate function since it is used so many times

@yichic yichic force-pushed the share-log-mutations branch 2 times, most recently from 79d6cb8 to 65a7a75 Mar 9, 2018

@alecgrieser

This comment has been minimized.

Sign in to view
Contributor

alecgrieser commented Mar 14, 2018

Should this PR be against the release-5.2 branch instead of the master branch?

@yichic yichic force-pushed the share-log-mutations branch from 9130327 to ca15e12 Mar 16, 2018

@yichic yichic force-pushed the share-log-mutations branch from ca15e12 to 582b875 Mar 17, 2018

@yichic yichic changed the base branch from master to release-5.2 Mar 17, 2018

@yichic yichic changed the base branch from release-5.2 to master Mar 17, 2018

@alecgrieser alecgrieser changed the base branch from master to release-5.2 Mar 19, 2018

@alecgrieser alecgrieser changed the base branch from release-5.2 to master Mar 19, 2018

@yichic yichic referenced this pull request Mar 19, 2018

Merged

Share log mutations 5.2 #89

@yichic yichic merged commit 59bd6c3 into apple:master Mar 19, 2018

mpilman pushed a commit to snowflakedb/foundationdb that referenced this pull request Dec 3, 2018

Markus Pilman Markus Pilman
Merge pull request apple#3 in OPFDB/foundationdb from ~MPILMAN/founda… 
…tiondb:features/flatbuffers to features/flatbuffers

* commit '5863ea5be3a621cdbb6b5141eea662483571fc69': (23 commits)
  make usage of old serializer more explicit for keys
  Ported SerializedStatusObject
  fixed stupid bug to take references to deserialize
  handle OptionalInterface differently in flat_buffers
  generate error definitions from xml file
  fixed silly typo
  include FDBTypes to make cquery errors go away
  fix silly bug where we compare with wrong version
  Change client version string to both versions
  address code-review issues
  fix for gcc
  make the client interface flat-buffers compatible
  compilation fix
  fdbserver is now compiling
  most of fdbserver is compiling
  modernize some serialization stuff
  Continued refactoring
  second attempt in serializing ReplicationPolicy
  Next chunk of refactoring
  in the middle of refactoring (doesn't compile)
  ...
fcdf681
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment