New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Share log mutations between backups and DRs which have the same backu… #3

Merged
merged 3 commits into from Mar 19, 2018

Conversation

Projects
None yet
3 participants
@yichic
Contributor

yichic commented Feb 20, 2018

…p range

@@ -1141,7 +1273,7 @@ namespace dbBackup {
return Void();
}
ACTOR static Future<Key> addTask(Reference<ReadYourWritesTransaction> tr, Reference<TaskBucket> taskBucket, Key logUid, Key backupUid, Key keyAddPrefix, Key keyRemovePrefix, Key keyConfigBackupRanges, Key tagName, TaskCompletionKey completionKey, Reference<TaskFuture> waitFor = Reference<TaskFuture>(), bool databasesInSync=false)
ACTOR static Future<Key> addTask(Reference<ReadYourWritesTransaction> tr, Reference<TaskBucket> taskBucket, Key logUid, Key backupUid, /*Key destUid,*/ Key keyAddPrefix, Key keyRemovePrefix, Key keyConfigBackupRanges, Key tagName, TaskCompletionKey completionKey, Reference<TaskFuture> waitFor = Reference<TaskFuture>(), bool databasesInSync=false)

This comment has been minimized.

@etschannen

etschannen Feb 20, 2018

Contributor

remove comment

@@ -1496,7 +1497,7 @@ namespace fileBackup {
.detail("ScheduledVersion", scheduledVersion)
.detail("BeginKey", range.begin.printable())
.detail("EndKey", range.end.printable())
.suppressFor(2);
.suppressFor(2, true);

This comment has been minimized.

@etschannen

etschannen Feb 20, 2018

Contributor

unneeded

Reference<Cluster> cluster = Cluster::createCluster(connFile, -1);
Database cx = cluster->createDatabase(LiteralStringRef("DB")).get();
Reference<ClusterConnectionFile> extraFile(new ClusterConnectionFile(*g_simulator.extraDB));
Reference<Cluster> extraCluster = Cluster::createCluster(extraFile, -1);
state Database extraDB = extraCluster->createDatabase(LiteralStringRef("DB")).get();
TraceEvent("StartingBackupAgents").detail("connFile", connFile->getConnectionString().toString()).detail("extraString", extraFile->getConnectionString().toString());
TraceEvent("StartingDrAgents").detail("connFile", connFile->getConnectionString().toString()).detail("extraString", extraFile->getConnectionString().toString());

This comment has been minimized.

@etschannen

etschannen Feb 20, 2018

Contributor

add a similar trace event for backup agents

Void _ = wait(delay(1.0));
}
TraceEvent("StoppingBackupAgents");
TraceEvent("StoppingDrAgents");

This comment has been minimized.

@etschannen

etschannen Feb 20, 2018

Contributor

add a similar trace event for backup

shareLogRange=true
performRestore=false
testName=BackupToDBCorrectness

This comment has been minimized.

@etschannen

etschannen Feb 20, 2018

Contributor

something needs to restore

}
Future<Void> clearLogRanges(Reference<ReadYourWritesTransaction> tr, bool clearVersionHistory, Key logUidValue, Key destUidValue, Version beginVersion, Version endVersion) {
return _clearLogRanges(tr, clearVersionHistory, logUidValue, destUidValue, beginVersion, endVersion);

This comment has been minimized.

@etschannen

etschannen Feb 21, 2018

Contributor

add a comment about the fact that endVersion - beginVersion should not be too large

if (!v.present()) {
return Void();
}

This comment has been minimized.

@etschannen

etschannen Feb 21, 2018

Contributor

use logUid instead of beginVersion for finding ourself in the list of versions
add an ASSERT_WE_THINK that v matches beginVersion, and matches ourself in the backupLatestVersions list

This comment has been minimized.

@etschannen

etschannen Feb 21, 2018

Contributor

do not get backupLatestVersionsKey separately; loop through backupVersions and check for ourself in those results to accomplish the same thing

if (currVersion > beginVersion) {
if (currVersion < nextSmallestVersion) {
nextSmallestVersion = currVersion;
}

This comment has been minimized.

@etschannen

etschannen Feb 21, 2018

Contributor

use std::min for code clarity

@@ -609,7 +625,7 @@ namespace dbBackup {
tr.addReadConflictRange(singleKeyRange(kv.key));
first = false;
}
tr.set(kv.key.removePrefix(backupLogKeys.begin).withPrefix(applyLogKeys.begin), kv.value);
tr.set(kv.key.removePrefix(backupLogKeys.begin).removePrefix(task->params[BackupAgentBase::destUid]).withPrefix(task->params[BackupAgentBase::keyConfigLogUid]).withPrefix(applyLogKeys.begin), kv.value);
bytesSet += kv.expectedSize() - backupLogKeys.begin.expectedSize() + applyLogKeys.begin.expectedSize();

This comment has been minimized.

@etschannen

etschannen Feb 21, 2018

Contributor

calculate the two withPrefixs into a single string
use mutateString to directly change the bytes in the key instead of using withPrefix to copy the bytes into a new string

beginVersion = BinaryReader::fromStringRef<Version>(bVersion.get(), Unversioned());
Void _ = wait(tr->commit());
endVersion = tr->getCommittedVersion();
break;

This comment has been minimized.

@etschannen

etschannen Feb 21, 2018

Contributor

for safety use committedVersion + 1

Void _ = wait(tr->commit());
break;
} catch (Error &e) {
Void _ = wait(srcTr->onError(e));

This comment has been minimized.

@etschannen

etschannen Feb 21, 2018

Contributor

tr not srcTr

Void _ = wait(srcTr->commit());
endVersion = srcTr->getCommittedVersion();

This comment has been minimized.

@etschannen

etschannen Feb 21, 2018

Contributor

+1 committed version for safety

}
}
}

This comment has been minimized.

@etschannen

etschannen Feb 21, 2018

Contributor

factor this transaction into a separate function since it is used so many times

@yichic yichic force-pushed the share-log-mutations branch 2 times, most recently from 79d6cb8 to 65a7a75 Mar 9, 2018

@alecgrieser

This comment has been minimized.

Contributor

alecgrieser commented Mar 14, 2018

Should this PR be against the release-5.2 branch instead of the master branch?

@yichic yichic force-pushed the share-log-mutations branch from 9130327 to ca15e12 Mar 16, 2018

@yichic yichic force-pushed the share-log-mutations branch from ca15e12 to 582b875 Mar 17, 2018

@yichic yichic changed the base branch from master to release-5.2 Mar 17, 2018

@yichic yichic changed the base branch from release-5.2 to master Mar 17, 2018

@alecgrieser alecgrieser changed the base branch from master to release-5.2 Mar 19, 2018

@alecgrieser alecgrieser changed the base branch from release-5.2 to master Mar 19, 2018

@yichic yichic referenced this pull request Mar 19, 2018

Merged

Share log mutations 5.2 #89

@yichic yichic merged commit 59bd6c3 into apple:master Mar 19, 2018

mpilman pushed a commit to snowflakedb/foundationdb that referenced this pull request Dec 3, 2018

Markus Pilman Markus Pilman
Merge pull request apple#3 in OPFDB/foundationdb from ~MPILMAN/founda…
…tiondb:features/flatbuffers to features/flatbuffers

* commit '5863ea5be3a621cdbb6b5141eea662483571fc69': (23 commits)
  make usage of old serializer more explicit for keys
  Ported SerializedStatusObject
  fixed stupid bug to take references to deserialize
  handle OptionalInterface differently in flat_buffers
  generate error definitions from xml file
  fixed silly typo
  include FDBTypes to make cquery errors go away
  fix silly bug where we compare with wrong version
  Change client version string to both versions
  address code-review issues
  fix for gcc
  make the client interface flat-buffers compatible
  compilation fix
  fdbserver is now compiling
  most of fdbserver is compiling
  modernize some serialization stuff
  Continued refactoring
  second attempt in serializing ReplicationPolicy
  Next chunk of refactoring
  in the middle of refactoring (doesn't compile)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment