-
Notifications
You must be signed in to change notification settings - Fork 57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
zrep_expire Internal Err caller did not hold fs lock #44
Comments
Hmm.
okay, so the two immediate questions that come to mind are:
1. does this still happen if it is to an actual separate remote system
2. does it still happen if you are not using recursive, with ZREP_R ?
|
To 2.: Without ZPREP_R there is no error message. Of course the functionality is then also not what I would like to have for hierarchical filesystems. To 1.: is it needed to have a really separate remote system, or would it be enough to use a different destination pool or an SSH to the own machine? |
Hmm.
So this is a bug in expire for recursive.
Ick.
As far as remote:
zrep takes some shortcuts when destination target is localhost.
Also, there may be issues with global lock contention in a single machine
that does not happen with a remote machine.
Would be a good experiment if you have the time.
|
to update the original case (with recursion): when doing the |
right, the "remote" expires are failing. which is why I was wondering if
they would work if it was "really" remote.
|
Now I found the time to set things up on 2 different machines. There is works without problems. So it looks like the problem only appears when running against localhost. If you need more logs or tests or so, tell me. |
Hmm.
Thanks for doing the comparison.
|
Odd. I cant see a problem in the logic flow.
Would you please grab the latest git version and try the sync again,
with debug on?
|
Will do.
|
Interesting that the debug var doesnt transfer to remote side.
|
Did a quickie git patch that forces DEBUG value to be passed
|
And here is the output of a retest on the original single machine, incl. remote DEBUG:
|
zrep_sync -> _expire -> $ZREP_PATH expire zrep_expire DOES call zrep_lock_fs I'm thinking that you are perhaps calling first invocation of zrep, with However, that one then just calls "zrep", which is getting picked up from somewhere ELSE in your path, and it doesnt have the new DEBUG output in it? |
aha. dont call it with env DEBUG=1 zrep ... JUST call it with DEBUG=1 zrep ... because "env" nukes your PATH variable. So if your path=/usr/local/bin:/usr/bin |
Hi, Regarding how I call zrep. You see my statements in this ticket. So I do not call zrep with full path, but without path. However, there is only exactly 1 zrep on the machine. I moved it to /usr/local/sbin, which is part to my PATH variable. That is, for each new commit you do I do:
Regarding using env or not using env. I think this does not make a difference, as env is not (at least for me) changing my path, see the environment with and without usage of env:
But just for testing, I re-executed the whole script, now without using "env". To me the result looks identical
I also run yet another script which is still on a single machine, but source and dest filesystems are on different pools, however, the same error message comes when zrep tries to expire the snaps on the destination. |
env with arguments is supposed to act differently than without it.
what does
$ env env
say?
|
Regarding env, I feel the discussion drives away from the core of the problem. Back to the core problem. Going through your script I find that when expiring the origin side, when _expire() is called, it is called from within the running script, so with the PID that also did the previous sync and which is stored in the zfs properties as lock-pid of the destination as "received". Then in line 1741 you find that you also want to expire the destination side. That triggers these questions: Is it necessary to call a second process when the destination is on the same localhost machine than the origin? |
BTW, my refactored self-contained test script is now this:
|
Here are the two conceptual oddities in your case:
1. a lock is set *per filesystem*.
The remote side is a *separate filesystem*.
That is why it is fine to call expire on the destination in a separate
process. The second process acquires a lock on the target filesystem, does
the expire, and releases it.
That is why it *works* when called on a remote filesystem.
That is why I dont understand why it is *not* working on localhost
2. the latest git version of zrep has a debug output every time it tries to
acquire a fs lock.
As you can see in your output,
zrep_lock_fs: set lock on backup/z0
If you are using the same version of the script on both sides, you SHOULD
also be seeing
zrep_lock_fs: set lock on backup/zcopy/z0
but you are not.
Therefore it seems like it is calling an older version of zrep for the
second expire somehow.
|
I can probably not prove to you that I only run 1 version of zrep, and as origfs and destfs are on the same localhost, there is only 1 zrep script, so there is nothing like old/new script. I guess you would have to trust me on this. I was debugging your script. This is what I understand: Now we have 3 zrep:lock-pid properties:
Then inside _snapandsync next _sync is called, which zfs send/receive the snapshot @zrep_000001 to the destination filesystem. Now we have 6 zrep:lock-pid properties,
Now, still from zrep_sync, the _expire $srcfs is called After the expiration ran successfully on origpool/z0, next zrep_ssh localhost 'zrep expire destpool/zcopy/z0' is called. New process: zrep expire calls zrep_expire It then checks if "owning lock still exists", which is true, as the "received" lock-pid is the PID of the zrep sync process and this is still running as it has just spawned the zrep expire process. I guess this is a crucial point. The PID retrieved from the destination filesystem is the PID of the currently running "zrep sync" process on the original side. Here it makes a difference: on 2 different machines, that PID may also be in use by some arbitrary other process, but on a not so busy system chances are that that on the destination system there is no running process with this PID. In that case you would (falsly) interpret it as stale and just assign a new lock which then makes it appear to work. But here the process is still running, thus you return 1, i.e. zrep_lock_fs() fails. From all this, I could think the problem is that you probably did not expect to get the "received" property of "zrep:lock-pid", but your check more is inclided towards a "local" property. Hope this helps you finding the root cause. |
Thank you for your feedback. I will look into the PID issue.
Now something for you to do:
Edit the script, and add your own debug to zrep_lock_fs
.
Print out "Hi Carsten!" to stderr.
echo "HI Carsten!" >/dev/fd/2
If you dont see that after it says (expiring backup/copy/z0), then you need
to figure out what copy of zrep the remote expire is actually calling.
PS: I also bumped git version to add a little extra debug to zrep_lock_fs
|
PS: you are wrong about this:
On Sun, Apr 2, 2017 at 1:42 PM, Carsten Langer ***@***.***> wrote:
....
Then inside _snapandsync next _sync is called, which zfs send/receive the
snapshot @zrep_000001 to the destination filesystem.
Now we have 6 zrep:lock-pid properties,
- as "local" in orgpool/z0
- as "inherited" in ***@***.***_000000
- as "inherited" in ***@***.***_000001
- as "received" in destpool/zcopy/z0
- as "inherited" in ***@***.***_000000
- as "inherited" in ***@***.***_000001
That is, also for the top filesystem of the hierarchical backup on the
destination, the "destpool/zcopy/z0", we now have a zrep:lock-pid, as it
was copied over via the zfs send/receive.
because on SYNCS zrep does not use (send properties in snapshot) flag.
It does that only on init (see use of -p)
|
also:
On Sun, Apr 2, 2017 at 1:42 PM, Carsten Langer ***@***.***> wrote:
I can probably not prove to you that I only run 1 version of zrep, and as
origfs and destfs ar
Here it makes a difference: on 2 different machines, that PID may also be
in use by some arbitrary other process, but on a not so busy system chances
are that that on the destination system there is no running process with
this PID. In that case you would (falsly) interpret it as stale and just
assign a new lock which then makes it appear to work.
but if this were the case, then since you have DEBUG flag on, you should
see output from:
_errprint overiding stale lock on $1 from pid
$check
but you dont.
Therefore it is not doing that.
|
This is how it should look. Note the double "zrep_lock_fs" output. $# DEBUG=1 ZREP_R=-R zrep sync all |
Hi Phil, I added as first line in zrep_lock_fs() You write
Well, that was not theory. I actually observed it in my system. I added in zrep_sync() after _snapandsync and before _expire a "exit 1". This exited the script after "zfs send" and befor any expriation. Then I listed all properties and found them as described. You are right that the first time (from zrep init) zfs send runs with -p: BUT: I also use -R for the recursive sync. And ZFS's manpage states that this implies copying over all properties:
The man page continues ...
This explains why I see the zrep:loc-pid property with "source=received" on the destination filesystem. My feeling is that you did not expect that "zrep sync" would copy over any property, and it would not if the "-R" was not used. Without copying over the properties, the destination filesystem would not have the zrep:loc-pid property received from the source filesystem, and then your locking methodology would work. But with "-R" it does copy over the properties, at least in my version of ZFS (zfs and os versions see bottom of #44 (comment)). You then wrote:
It does indeed do that, i.e. it would do if there was not the other bug. Now that mentioned commit broke things for me. When I retested today with 2 machines, I got the following error: "ssh: Could not resolve hostname debug=1: Name or service not known", and the script brakes. Running with ksh -x I see:
So I guess that the commit for adding the DEBUG to the other side through ssh does not work. I therefore removed the "DEBUG=${DEBUG}" from that line again and instead uncommented the prepared DEBUG=1 in line 33 on both machines. Now both zrep run in DEBUG mode, and voila, I get the error message on the stale lock when running on 2 separate machines. It looks like complicated, but I think we now have 2 streams: I appreciate your active discussion in this topic. I hope you can fix the issues. Best regards |
ah... thanks for your detailed investigation. I did the -R check, but only for sync, not for init. most places I already used zfs get -s local. Please try the latest git. |
wait.. my comments dont make sense. But new version should still fix your problems I think :) |
or then again.. i may have broken it
|
Fun fact.. when you use "zfs get", an unset value is returned as "-" /smack zfs devs. git updated |
Nearly there. There is still one error in commit 1247e4a, which i commented there. With that proposed change, it runs for me on the 2 separate machines (both debug and sync). will now try on the single machine. |
I can confirm it also works on the single machine now (with the manual patch that I commented above). |
ah, thanks for the logic check!
Updated.
|
cool, finally all parts of the issue fixed. I close this issue. Thanks again. |
Thank you for your willingness and assistance in testing and debugging :) |
…s bug #44,and possibly others. Also, bump to version 1.7
This is follow up on #16, unfortunately I still get the error with latest v1.6.8.
OS and ZFS versions at the end.
I have this issue on the real data, but then I created some sample hierarchical dataset on pool called "backup" to reproduce the error more easily. Original filesystems are backup/z0, backup/z0/z1 and backup/z0/z2. All get recursively replicated to backup/zcopy/z0/...
Enrivonment:
zrep v1.6.8, as downloaded from https://raw.githubusercontent.com/bolthole/zrep/master/zrep on 2017-03-26.
The text was updated successfully, but these errors were encountered: