Skip to content

domachine/filesync

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

91 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Filesync (file synchronization utility)

1 Description

Filesync is a utility which makes it easy to keep two places synchron. It listens on the one place for changes and applies them to the other.

2 Roadmap

2.1 TESTED Package the program with CMake

2.2 Version 1.0 (current)

Basic functionality:

  • Synchronize two directories

State:

  • Branched

2.3 Version 2.0

New features:

2.4 Version 3.0

New features:

2.5 Sub-versions

All Sub-versions will be used to submit patches and bugfixes etc.

3 User Interface

3.1 Program options

OptionDescription
-r, --rsyncRsync executable path.
-s, --sourceThe path to the source directory.
-t, --targetThe path to the target directory.
-d, --depthThe depth to dive into the
directory.
-D, --daemonRuns filesync as daemon.
-p, --pid-fileSets the file in which the daemon writes its PID.
-l, --log-fileThe log-file to redirect messages to (default: stderr).
-c, --conf-fileA lua configuration file.

4 Features

4.1 Daemon functionality

Filesync can be used as a daemon using the initscript filesyncd. This is configured through a configuration either in $HOME/.filesyncrc, /etc/filesyncrc or in a file specified through the --config option. The configuration-file is a simple bash script which can set the following options:

FILESYNC_EXE=filesync_path
JOBS=(config_files ...)

config_files are lua scripts according to the Profile functionality.

4.2 Depth feature

Filesync will be able to limit the watching depth. Means the depth of diving recoursive into a directory.

  • -1 means unlimited depth
  • 0 means: visit only the given directory without watching any subdirectories.
  • n: Visit n subdirectory-levels.

5 Implementation

5.1 General structure

5.1.1 TESTED Boolean type

Implemented as enum.

enum BOOL
{
    FALSE,
    TRUE
}

5.1.2 TESTED watch-session struct

src/watch_session.h src/watch_session.c

struct watch_session *new_watch_session();

struct watch_session
{
    /** \brief The src directory which is watched. */
    char *src;

    /** \brief The target which will be synchronized. */
    char *target;

    /** \brief The path to the rsync-executable. */
    char *rsync_path;

    /** \brief The depth to watch the source directory. */
    int depth;

    /** \brief Inotify dscriptor. */
    int notify_descr;

    /** \brief The watch-table. */
    struct dir_watch *watch_table;

    /** \brief The watch filter mask. */
    uint32_t watch_mask;
};

void destroy_watch_session(struct watch_session *ws);

depth = -1 means unlimited depth. rsync-cmd-cache is a buffer for the prebuild (Build rsync command) rsync command.

5.1.2.1 OBSOLETE Free the hash structure

file:src/watch_session.c::/* TODO: Free the hash structure. */

5.1.2.2 TESTED Implement default watch-filter-mask.

file:src/watch_session.c::/* TODO: Fix this with default bitmask. */

5.1.2.3 TESTED Cache the source and the target length.

Also affects src/notify.c

5.1.3 TESTED Module testing system

5.1.4 TESTED Logging

The logging mechanism should be used to output various messages. Three levels are defined which should be used in appropriate situations.

LevelDescription
INFOA normal program message.
WARNOutputted if warnings are enabled.
ERRORAlways outputted (system critical errors).
DEBUGOnly outputted if debug is explicitely requested.

The levels are implemented using int constants which can be concatenated to a flag map.

Logging has a unique interface where it is easy to write to log files or use the standard error output etc. The logging interface is initialized using a initialization function that takes a pointer to the FILE object which will be used for logging. The second argument shows whether a newline should be appended to each output. The initialization function needs to be called at least once. Otherwise the program will abort on a logging request.

The initialization function initializes the logging interface and configures it.

void init_log(FILE *c, BOOL nl, int filter);

The second argument is of Boolean type. The third argument tells the interface which levels should be logged. The logging itself is done using the log\_msg() function. It takes the level and the string (as format string) to log.

void log_msg(LOG_LEVEL l, const char *fmt, ...);

The logging should be closed using the close function. This function closes the stored file descriptor (if != stderr and != stdout).

void close_log();

5.1.4.1 TODO [#C] Add possibility to set channel prefixes like [DEBUG] …

5.1.5 OBSOLETE [#A] Build rsync command

The rsync-cmd-cache field in the watch-session structure represents a template of the used rsync command. This can be done because we need only one command all the time with different parameters. But these parameters can be format sprintf. The building of the command should be done by a seperate function.

char *build_rsync_cmd_tmpl(const char *rsync_path,
                           const char *src);

5.1.6 TESTED [#A] Rsync proxy

src/rsync_proxy.h src/rsync_proxy.c

A proxy function that abstracts the rsync call.

int sync_file(struct watch_session *ws, const char *path);

path should be relative to the target path. The function should return a value != 0 if something went wrong. A value != 0 is a constant that can be looked up in a error table.

5.1.6.1 TODO [#C] Send rsync output through logging interface.

5.1.7 TESTED [#B] Command line parser

src/cmdparser.h src/cmdparser.c

A command line parser which parses the command line and fills the fields of a watch_session structure.

The command line mainly consists of a method that takes a pointer to a watch-session struct. This watch-session is filled with the information extracted from the command line. The watch session should be created with new_watch_session().

Errors and warnings are logged through the logging interface. A return-value != 0 indicates an error while parsing.

int parse_cmd_line(struct watch_session *ws, int argc, const char **argv);

5.1.7.1 TODO Make the verbosity level customizable

file:src/main.c::/* TODO: make the verbosity level cutomizable through command line. */

5.1.8 User authentication

The user authentication should be done with ssh-keys or standard-input. Filesync doesn’t support plain text authentication via the command line interface to avoid attackers to read the password via the process table.

5.1.9 TESTED [#B] Inotify watch loop

5.1.9.1 TESTED File Watch structure

src/notify.h src/notify.c

struct dir_watch
{
    int wd;

    /** \brief The path of the watched directory (relative to the top level). */
    char *path;

    /** \brief The depth-level of the watched directory. */
    int depth_level;

    /** \brief Needed to make this hashable. */
    UT_hash_handle hh;
};

5.1.9.2 TESTED Depth recording

src/main.c

5.1.9.2.1 Convention

Top-level (src directory) is level 0.

5.1.10 TESTED Exclude functionality

Makes it possible to exclude certain files from synchronization.

5.1.11 TESTED Do a clean shutdown after receiving SIGINT

5.2 TODO [#C] Memory error system

5.3 Profile functionality

This describes the possibility to declare filesync options within a config file (profile) and call filesync with this config file as configuration instruction.

A configuration file is a simple lua script which can set the following variables which will be interpreted by filesync.

src = "Your source directory"
target = "Your target"
exclude = "Exclude regex"
depth = -1

5.4 TESTED Daemon functionality

5.4.1 TESTED Initscript

5.4.2 TESTED Pid-file option

5.4.3 TESTED Log-file option

6 Plans

6.1 TODO Design filesync as daemon that is called for work

Filesync should be a daemon that runs in memory and waits for requests that tell him to watch a certain directory and synchronize it with another.

6.1.1 TODO Interface to communicate

Unix-Sockets will be used as communication interface.

6.1.2 TODO Communication protocol

Command (1 byte)ArgumentEOF
enum command {
    ADD_JOB,
    SHUTDOWN
};

The first byte of the message is the command byte. It specifies what the daemon should do. The rest of the message is reserved for arguments for the command.

6.1.2.1 Commands

CommandArgumentsDescription
ADD_JOB<src>\0<target>\0<depth>\0<watch-mask>This command
takes the src-directory, target-
directory, depth and watch-mask
seperated by null-bytes. It creates then
a new watch-session and launches it.
SHUTDOWN<none>This argument simply shuts the daemon down.

7 Copyright

Copyright (C) 2010, 2011 Dominik Burgdörfer <dominik.burgdoerfer@googlemail.com>

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published