Pre and Post commands / hooks

[user9209]

Configurable commands or scripts to run at various point in a file life cycle.

First: You did a great job! Thanks a lot!

Feature-Request: Pre and Post commands What does it means?

Why?

• Encrypt data
• Notify users
• check for requirements
• ...

Pull

• Global Pre-Pull--Command

  • Runs before downloading the sync
  • runs for each share not overwritting it
  • label:
    • %share-path%
    • %remote-device-id%
    • %local-device-id%
    • ...

• Share Pre-Pull--Command

  • Runs before downloading the sync
  • only a single share
  • label like Global

• Global Pull-File-Command

  • only at download
  • run it on each single file
  • runing on every share (if not overwritten in the share-setting)

• Share Pull-File-Command

  • only at download
  • run it on each single file
  • runing only on a single share (overwrittes global-setting)

• Global Post-Pull-Command

  • Runs after downloading the sync
  • runing on every share (if not overwritten in the share-setting)
  • runs for each share not overwritting it
  • label like Global-Pre-Pull

• Share Post-Pull--Command

  • Runs after downloading the sync
  • only a single share
  • label like Global-Pre-Pull

Push

• Global Post-Push--Command

  • Runs before uploading the sync
  • runs for each share not overwritting it
  • label:
    • %share-path%
    • %remote-device-id%
    • %local-device-id%
    • ...

• Share Pre-Push--Command

  • Runs before uploading the sync
  • only a single share
  • label like Global

• Global / Share Push-File-Command

  • only at upload
  • run it on each single file
  • runing on every share

• Global Post-Push-Command

  • Runs after uploading the sync
  • runing on every share (if not overwritten in the share-setting)
  • runs for each share not overwritting it
  • label like Global-Pre-Pull

• Share Post-Push--Command

  • Runs after uploading the sync
  • only a single share
  • label like Global-Pre-Pull

By default show global setting in share tab

• if values is changed, take the changed value and copy all others
  • important, as you do not know, whats is happening in the single shares if changing global settings
  • global setting can also used by set the value to <global>
  • setting a value to " " (empty string) means to do nothing! Do not use global setting!

For all:

• some labels like "%filename%", "%input-file%" = input file, %output-file% = ouput file name, "%hash%", %size%" = file size input file, ...
  • For using as myScriptDoingThinks --filname "%filename%" --hash %hash% --size %size% --ouput %output-file%
    • the client will replace all "%...%" with its values
    • if for example %output-file% is not set, file "%input-file% will be uploaded, if it is set file %output-file%

Labels:

• %additional-files% = if exist, push also files in each line <filename>.sha3\n<filename>.meta
• %filename% = filename no path
• %input-file% = filename read with absolute path
• %output-file% = filename write with absolute path (if not exist use %input-file%)
• %size% = file size input
• %hash% = hash of the block
• %device-id-local% = client id
• %device-id-remote% = id of extern partner(s)
• %email% = if support of user variables is insert so settings can take by the scripts like email
• %...%

Exit codes

• 0 = OK
• 1 = skip file
• 2 = about sync

Samples

Samples for Pre-Pull-Commands

• Mounting a device
• Checking for internet connection
• Checking for free space / existing file / ...
• Stopping a service using the shared files
• connect to vpn

Samples for Pull-File-Command

• Decrypt file
• check checksum of file
• unzip archivs
• unzip file
• run virus scan
• add data to a database, log, ...

Samples for Post-Pull-Commands

• check checksum of all file
• run virus scan
• send a notification to the user (popup)
• send a email notification
• disconnect from vpn
• run a deploy (e.g. copy to /var/www)
• run a backup

Samples for Pre-Push-Commands

• generate checksum of all file
• Checking for internet connection
• connect to vpn

Samples for Pull-Push-Command

• Ecrypt file
• generate checksum of file
• zip file
• add data to a database, log, ...

Samples for Post-Push-Commands

• Unmounting a device
• disconnect to vpn
• Starting a service again using the shared files
• send a notification to the user (popup)
• send a email notification

Modifikation or addition version of File-Command

• using pipes!
• files is read by the client, gets piped to the command (Standard-Inputstream), doing things on the stream (read, modify, add) and outputs a pipe (Standard-Outputstream)
• Warning: input size may not equal output size (adding cipher header, ...)
• Example (Push):
• encrypt-then-hash --hash %hash% --size %size% --device-id %device-id% -file-list %additional-files%
  1. Opens file stream to local source file
  2. Script
     1. Compares stream matching %hash%
     2. Encrypts stream and signing it with cert of %device-id%
     3. Generate a sha3-hash of the stream
     4. output stream
  3. pipe stream data to extern partners

Final

Many information, some complexity but would to a design, where you can do crazy things.

[calmh]

Thanks for a well written feature request. I don't expect this to be implemented in its entirety, but it's a good list to choose from.

[heX16]

@user9209 "Encrypt data": can the command change the contents of the source file? or creates a copy of the file in another directory with decrypted content?

[user9209]

@user9209 "Encrypt data": can the command change the contents of the source file? or creates a copy of the file in another directory with decrypted content?

As description above:

%input-file% = filename read with absolute path
%output-file% = filename write with absolute path (if not exist use %input-file%)

if %output-file% is not use run it in-place (but file size may change!)
if %output-file% is use a new file is used.

The best would be to use pipes (file is read only once from the hdd, all operations are performed and written only once to the hdd)

In-place is not always possible (e.g. output-size > input-size does not work whereas OTR is easy implement in-place).

So it depends on the implementation.
Use as less read and write processes to hdd in the pipeline.

[lulou]

Would it be possible to implement blocking synchronous scripts on a folder level?
for example 3 folders:
folder a
folder b
folder c

folder b contains copies of your syncthing configuration.
folder a and b contains files with your important data. the scripts within folder a and folder b read the content of folder b.
there are 2 possible outcomes here:
folder a has changed and folder b.
pause syncing of folder a (like litterally and endless loop until... from the script in folder a).
therefore trigger files in folder b are changed and the script in folder a returns. folder a is now synced.

you do not need all of the above stuff to implement all of the above stuff if you can go for blocking a folder sync with scripts.
depends on wether block exchange happens on a folder or global basis. i think.

further example 1, store encrypted files within the syncthing repo
folder e is not added to the syncthing repo
folder c contains the encrypted data, and can be touched by syncthing.
stuff is changed in c, post execute decryption and write it to e
add a hook to whatever you use to create your decrypted files and write to c if there is a change.
crypting stuff can take long, so i would not add that calculation process as overhead to the syncthing scan.

as i wrote within #6009 executing stuff, wich can be changed from the outside world could be a security concern
this feature has to be turned off by default, only accessible with a command line switch
if you depend on it add digital signatures to your files.

[danisztls]

Even something simpler would be extremely useful.

Use case:

I have a script that imports pictures from a dir and do all sorts of processing and outputs that to a export dir. Currently I'm using Syncthing to sync both folder from and to my phone. Ideally I would call my script after Syncthing syncs import and call Syncthing to sync back to phone after exporting is done. I could schedule the script execution periodically or monitor for file changes via inotify but processing while Syncthing is syncing may cause conflicts.

[desbma]

I have a similar need to run custom commands as "hooks" when certain events happen.

Previously I was using custom inotify scripts but they were unefficient and/or unreliable.

So I wrote a simple daemon to react to Syncthing events using the REST API, and run configurable commands: stfed.

It only supports 3 events (folder down sync finished, file down sync finished, local file conflict), but it works well for my use cases.

I'm posting this here in case this is useful for someone else, if not, its open source, so you can fork and/or complain in issues.