Sync daemon mode

[untitaker]

Long-running process that keeps connections open. Not sure if this would be that useful.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[geier]

there is an RFC that would probably make sense to support for such a daemon: RFC 6578 Collection Synchronization for Web Distributed Authoring and Versioning (WebDAV) but I didn't find anything on server support (apart from Apples CalDAV server).

Edit: Looks like sabredav supports it

[geier]

btw: I got an email saying that #24 was "Add hook for shell commands", what happened to that?

[untitaker]

On 2 April 2014 23:41:04 CEST, Christian Geier notifications@github.com wrote:

btw: I got an email saying that #24 was "Add hook for shell commands",
what happened to that?

I renamed it after realizing sync hooks don't make sense without a daemon mode.

[untitaker]

As far as i can see, the RFC you linked only adds the ability to fetch changes that happened since some datetime, which is also a performance improvement but doesn't relate to keeping connections open, so it can be implemented independently. I opened #26 for this.

[WhyNotHugo]

Not sure if this would be that useful.

None-plaintext passwords, something that can't really be achieved by merely running it via cron.

[untitaker]

vdirsyncer prompts for the password, you can pipe to it from a wrapper script. Not ideal, but possible.

[WhyNotHugo]

vdirsyncer prompts for the password, you can pipe to it from a wrapper script. Not ideal, but possible.

Which sort of brings us back to #228. 😁 (I don't want to have to interact with anything when running via cron).

[untitaker]

(I don't want to have to interact with anything when running via cron).

Right, but you can write a script like this:

local password="$(pass vdirsyncer)"
while true; do
    echo "$password" | vdirsyncer sync
    sleep 60
done

and run it as daemon. You now have loaded the password in memory, don't have to store it as plaintext on disk and vdirsyncer still authenticates properly.

EDIT: Don't do that, you can potentially use /proc/ to read the password while the pipe is active (with a lot of luck or good timing, but still).

[untitaker]

And yes, that still requires you to enter your master password once, but there's no way around that.

[untitaker]

It seems like this feature is inevitable for security, but I wonder if there is another user-friendly way that doesn't involve re-inventing cronjobs in Python.

BTW, if you have installed click-repl, you can pipe commands to vdirsyncer repl and it will keep the passwords in memory.

[WhyNotHugo]

A second upside on a daemon is that it can use something like libevent-python to instantly push locally-modified events (without re-scanning everything else), and only download an defined intervals.

[untitaker]

instantly push locally-modified events (without re-scanning everything else)

Push events probably induce an insane amount of complexity, so I'm not sure if
I'm going to care about implementing it. I am still encountering bugs in
offlineimap's IDLE support which is why I've turned it off.

[untitaker]

FWIW I'm currently using this: https://github.com/untitaker/dotfiles/blob/master/home/.scripts/vsy

It requires click-repl.

[untitaker]

I've set the milestone to 1.0. I'm currently using the workaround described above, so I have no personal incentive to actually fix this. But we definetly need a better (official) solution here rather than the combo of an undocumented feature, an unstable third-party package and a massive shell script.

[untitaker]

The script I use above no longer requires an undocumented hack. So it is possible develop a vdirsyncer daemon outside of vdirsyncer itself, and I think this would be a good idea to take a shortcut to stable 1.0.

[Decipher]

Adding my interest in getting a Daemon mode added, I'm currently in the process of putting together a Docker version of vdirsyncer (https://hub.docker.com/r/deciphered/vdirsyncer-docker/) so that I can run it as a service on my Synology NAS, and having a daemon is essential.

[untitaker]

@Decipher depending on what you would exactly expect from such a daemon a cronjob or similar might do the trick

[Decipher]

@untitaker That's what I intend on implementing to start with, yes, but the benefit of having a Daemon mode would be that the logic to determine if the previous sync is still running or how long since the last sync was completed could be baked in instead of having to be accounted for by each individual implementer.

I'm currently writing a simple shell script that will either be hooked up to a cron-job, or just be the entrypoint for the Docker container, that will do that logic.

I suppose the biggest issue I can see is that there are multiple use cases that the Daemon/my script should account for, which is evident based on your own script above:

1. Trigger sync of local file system change (your approach, only works if you are syncing local <-> remote)
2. Time based sync (my approach as I'm syncing remote <-> remote)
3. Other use cases I haven't yet considered.

[untitaker]

You can write a shell script that alternates between vdirsyncer sync and sleep and run that as a daemon. No need to check for overlapping executions then.

[untitaker]

FWIW I don't think I want this in vdirsyncer 1.0, but as a separate (unstable!) project for now.

[Decipher]

Yes, you are right, as if it is actively running vdirsyncer sync it won't continue the shell script until said process has finished. Clearly I was just overthinking it ;)

I still think that a daemon mode is something that would benefit the project, but understand the desire to not overcomplicate the the 1.0 release.

I do think putting it in a seperate project isn't the best approach though, as it limits the visibility of the feature. Maybe it doesn't even need to be a daemon per se, just a flag for sync that keeps it actively running until it's exited or errors.

Anyway, I'll continue with the script approach for the moment and will let you know once the docker project is in a usable state.

[WhyNotHugo]

Another bonus feature for daemon mode is "ignore one-time network errors" (or maybe just warn). For example, if the network connection fails once in a row ignore it, and only warn me on the second attempt.

I occasionally get cron emails due to a transient network error that's not there any more 5 minutes later. If I disable cron emails, I'll never find about about conflicts.

[untitaker]

The thing is, all of this (even the retry logic) can be implemented in a wrapper script. I don't really see the need for this to be in vdirsyncer anymore.

[WhyNotHugo]

On the topic of RFC 6578, it seems that Xandikos supports that too.

As for a daemon mode, I'm using autovdirsyncer, which is very basic but works. A sync is run automatically when a local file changes, and otherwise a sync happens every 15 minutes.

The remaining issue here is that events created on the server take up to 15 minutes to get pulled, and polling isn't the most efficient thing. JMAP addresses this, but we don't support JMAP yet (#266). This draft spec exists too, but doesn't really work for us since (a) apparently no-one supports it (b) systems running vdirsyncer are seldom reachable remotely.