Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Backup server" mode; init when data already exists #53

Closed
joachimtingvold opened this issue Jun 25, 2017 · 17 comments
Closed

"Backup server" mode; init when data already exists #53

joachimtingvold opened this issue Jun 25, 2017 · 17 comments

Comments

@joachimtingvold
Copy link

joachimtingvold commented Jun 25, 2017 •
edited
Loading

Hi,

I'm trying to use the "backup server" mode of zrep, but I cannot get it to work.

Setup as follows;

### backup server
hostname = backup1.foo.bar
zfs_dataset = backup/client1/rpool

### client
hostname == client1.foo.bar
zfs_dataset = rpool

Goal; to use "backup server" mode to take recursive backup of existing, with existing data, pool rpool on client1 to existing, but empty, dataset/destination backup/client1/rpool on backup server backup1.

Tried the following on backup1;
$ZREP_PATH init backup/client1/rpool client1.foo.bar rpool

This results in zrep trying to create the remote dataset rpool, which fails (since the remote dataset/pool already is present, and have lots of data);

cannot create 'rpool': missing dataset name
Error: Cannot create client1.foo.bar:rpool

Trying to do different variations of $ZREP_PATH changeconfig doesn't seem to do the trick. One of the errors was as follows;

Error: backup/client1/rpool not master. Cannot fail over

export ZREP_R=-R is set (on both sides, if relevant).

I'm probably brainfarting hard here, but...
@joachimtingvold
Copy link
Author

Top level should work now according to #48 ... I'm using latest zrep directly from git.

@ppbrown
Copy link
Member

ppbrown commented Jun 25, 2017 via email

@joachimtingvold
Copy link
Author

Okay, so if I manually send the entire rpool tree from client1.foo.bar into the backup/client1/rpool tree on backup1.foo.bar, then what? It's not really clear to me what zrep commands needs to be issued on client1 and backup1, and in what order, for this to work.

I guess the inverted behavior of the "backup server" mode sets me a bit off here (but then again the instructions in the documentation isn't that straight forward regarding this topic (-: ).

@ppbrown
Copy link
Member

ppbrown commented Jun 26, 2017 via email

@joachimtingvold
Copy link
Author

joachimtingvold commented Jun 26, 2017 •
edited
Loading

But that section needs to be reversed due to how "backup server" mode works, yes?

Then it would be something like this;

  1. Copy the ZFS tree manually from client1 to backup1
  2. Make first snapshot (matching the zrep naming scheme) manually on client1?
  3. Then we need to set the configuration, in "reverse". On client1 (aka "srchost"?): zrep changeconfig -f rpool backup1.foo.bar backup/client1/rpool. On backup1 (aka "dsthost"?): zrep changeconfig -f -d backup/client1/rpool client1.foo.bar rpool. Yes? No?
  4. Then we do zrep sentsync rpool@zrep_snapnamehere on client1?
  5. Then we do zrep failover backup/client1/rpool on backup1 to "failover" to client1?
  6. Then we do zrep refresh backup/client1/rpool on backup1 every time we want to take a new backup?

I'm sorry if I'm asking dumb questions here. Even if I have a somewhat OK understanding on how ZFS works, I feel that it wasn't really "straight forward" how to do this by just reading the documentation for zrep (the combination of "srchost", "dsthost", and the "reverse behaviour", and with no real examples when having pre-existing file systems when in "backup server" mode). I'd be happy to make a PR explaining the steps in detail once I hammer this down, but until then; please bear with me (-:

@ppbrown
Copy link
Member

ppbrown commented Jun 26, 2017 via email

@joachimtingvold
Copy link
Author

joachimtingvold commented Jun 27, 2017 •
edited
Loading

So, there's still something funky with the order of dsthost/srchost.

This is what I've done so far;

  1. On client1: zfs snapshot -r rpool@zrep_000000 (had to search a bit to find the "standard naming scheme" -- the documentation is referring to it, but never explains the default).
  2. On backup1: ssh root@client1.foo.bar "zfs send -R rpool@zrep_000000" | pv | zfs recv -e backup/client1/rpool. This completes successfully, and the entire tree (including snapshots) is copied.
  3. On client1: $ZREP_PATH changeconfig -f rpool backup1.foo.bar backup/client1/rpool.
  4. On backup1: $ZREP_PATH changeconfig -f -d backup/client1/rpool client1.foo.bar rpool.
  5. At this point, trying to do $ZREP_PATH sentsync rpool@zrep_000000 on client1, it would try to SSH into backup1. Either step 3) and/or 4) has the wrong "direction", or step 6) is correct.
  6. Setting $ZREP_PATH sentsync backup/client1/rpool@zrep_000000 on backup1 works.

At this point, if I try to failover, the following happens;

root@backup1:~# $ZREP_PATH failover backup/client1/rpool
Setting readonly on local backup/client1/rpool, then syncing
ERROR: we are not master host for backup/client1/rpool
print master is client1.foo.bar, we are backup1
Error: zrep_sync could not create new snapshot for backup/client1/rpool

It already thinks it's in read-only mode, and that client1 is master. However, trying to refresh/sync;

root@backup1:~# $ZREP_PATH refresh backup/client1/rpool
Error: Sorry, you cant run refresh on a master mode fs backup/client1/rpool

Now it suddenly says it's master mode, while trying to failover it says it's not master? Sigh.

edit: fixed typo (extra -d on client1).

@joachimtingvold
Copy link
Author

joachimtingvold commented Jun 27, 2017 •
edited
Loading

This is the status after doing the above commands... looks correct to me, except that it's saying both readonly on and zrep:master yes on backup1 (so it should probably be zrep:master no on backup1, and zrep:master yes on client1, but not sure how to change that "properly" -- either manually, or by figuring out what went wrong in my commands above).

root@client1:~# $ZREP_PATH list -v
rpool:
zrep:savecount	5
zrep:dest-host	backup1.foo.bar
zrep:dest-fs	backup/client1/rpool
zrep:src-fs	rpoool
zrep:src-host	client1
last snapshot synced: 

root@backup1:~# $ZREP_PATH list -v
backup/client1/rpool:
readonly	on
zrep:master	yes
zrep:src-fs	rpoool
zrep:dest-fs	backup/client1/rpool
zrep:src-host	client1.foo.bar
zrep:savecount	5
zrep:dest-host	backup1
zrep:lock-pid	19482
zrep:lock-time	20170627211932
last snapshot synced: backup/client1/rpool@zrep_000000

@ppbrown
Copy link
Member

ppbrown commented Jun 27, 2017 via email

@joachimtingvold
Copy link
Author

But the backup1 is already master? Or does the takover command remove the readonly on flag? And on that topic, what does the readonly flag "mean"? Is readonly on == "we should never do zfs recv on this host"?

@ppbrown
Copy link
Member

ppbrown commented Jun 28, 2017 via email

@joachimtingvold
Copy link
Author

joachimtingvold commented Jun 28, 2017 •
edited
Loading

But there is still something wrong here? You say that client1 should be the source here..

Your code clearly states the following;

#  setfsconfigs srcfs desthost destfs
#  setfsconfigs -d destfs srchost srcfs

... and as such, my previous steps should be correct (skipping the "copy the ZFS tree manually beforehand", as that is now solved);

  1. On client1: $ZREP_PATH changeconfig -f rpool backup1.foo.bar backup/client1/rpool.
  2. On backup1: $ZREP_PATH changeconfig -f -d backup/client1/rpool client1.foo.bar rpool.

This results in the following;

###
### as seen on client1
###
root@client1:~# $ZREP_PATH changeconfig -f rpool backup1.foo.bar backup/client1/rpool
root@client1:~# $ZREP_PATH list -v
rpool:
zrep:savecount	5
zrep:dest-host	backup1.foo.bar
zrep:dest-fs	backup/client1/rpool
zrep:src-fs	rpool
zrep:src-host	client1
last snapshot synced: 


###
### as seen on backup1
###
root@backup1:~# $ZREP_PATH changeconfig -f -d backup/client1/rpool client1.foo.bar rpool
root@backup1:~# $ZREP_PATH list -v
backup/client1/rpool:
readonly	on
zrep:src-fs	rpool
zrep:dest-fs	backup/client1/rpool
zrep:src-host	client1.foo.bar
zrep:savecount	5
zrep:dest-host	backup1
last snapshot synced:

Which seems correct to me? Source is client1 on both sides, and destination is backup1 on both sides.

At this point, according to the documentation you linked to, we need to do a zrep sentsync (before doing the zrep takeover). However, sentsync can only be issued on backup1 (if I try to issue it on client1, then it tries to SSH into backup1, which is not what we want). If I issue it on backup1, the backup1 gets the zrep:master yes flag...

At this point;

  • If I issue zrep sentsync on client1, it tries to SSH into backup1, which defeats the point of "backup server" mode.
  • If I issue zrep sentsync on backup1, it works, but zrep:master yes is set on backup1, which, according to you is wrong?
  • If I issue zrep takeover on backup1 before setting zrep sentsync, it fails with Starting failover from remote side client1.foo.bar. Error: rpool not master. Cannot fail over. This is because none of the sides have zrep:master yes.
  • In order for zrep takeover to work, then rpool on client1 needs zrep:master yes set, but that only seems to happen if zrep sentsync is set on client1, which does not work.
  • If I set zrep sentsync on backup1, then both zrep failover and zrep takeover fails;
root@backup1:~# $ZREP_PATH takeover backup/client1/rpool
Error: backup/client1/rpool is already master. Cannot takeover
root@backup1:~# $ZREP_PATH failover backup/client1/rpool
Setting readonly on local backup/client1/rpool, then syncing
ERROR: we are not master host for backup/client1/rpool
print master is client1.foo.bar, we are backup1
Error: zrep_sync could not create new snapshot for backup/client1/rpool

I feel I have tested all possible combinations now, but it still doesn't seem clear to me how this is supposed to work.

In stead of all this back-and-forth, can't you just give the exact commands you mean is needed to set this up from scratch given the test values I've provided? (see below). Should be 3 or 4 commands, based on what you've said so far (two changeconfig, one sentsync, one takeover/failover).

### backup server
hostname = backup1.foo.bar
zfs_dataset = backup/client1/rpool

### client
hostname == client1.foo.bar
zfs_dataset = rpool

@joachimtingvold
Copy link
Author

Due to lack of answer, and not getting this to work (I tried other things not mentioned in this issue), I gave up. Tried to figure out where the logic might be broken in the code, but never got that far. Ended up using znapzend.

@ppbrown
Copy link
Member

ppbrown commented Sep 12, 2017 via email

@joachimtingvold
Copy link
Author

joachimtingvold commented Sep 13, 2017 •
edited
Loading

Hi,

You clearly didn't read my latest post (Jun 28th), which basically covers every single combination (as far as I can see). Specifically this part;

  • If I set zrep sentsync on backup1, then both zrep failover and zrep takeover fails;
root@backup1:~# $ZREP_PATH takeover backup/client1/rpool
Error: backup/client1/rpool is already master. Cannot takeover
root@backup1:~# $ZREP_PATH failover backup/client1/rpool
Setting readonly on local backup/client1/rpool, then syncing
ERROR: we are not master host for backup/client1/rpool
print master is client1.foo.bar, we are backup1
Error: zrep_sync could not create new snapshot for backup/client1/rpool

Again, as I asked in my latest update, if you believe this should work, can you please provide the commands needed, given the following;

### backup server
hostname = backup1.foo.bar
zfs_dataset = backup/client1/rpool

### client
hostname == client1.foo.bar
zfs_dataset = rpool

@ppbrown
Copy link
Member

ppbrown commented Sep 14, 2017 via email

@joachimtingvold
Copy link
Author

After updating the code with a new flag, yes. I won't be surprised if it works now, tbh (-:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@joachimtingvold @ppbrown