File Sync Details

File Sync Details

This page covers the file synchronization functionality of the Soft Data Diode Tools. For basic setup, see the main README.

How File Sync Works

The file synchronization uses a best-effort delivery approach with a sliding window mechanism. This is fundamentally different from protocols like TCP or even traditional file transfer methods.

Best-Effort Delivery Model

Unlike reliable protocols, there is no guarantee that any given file will transfer successfully in any particular sync round, or that the receiver is even listening when the sender attempts transmission. The system is designed around the constraints of one-way communication.

Key Characteristics:

• Files are discovered during each sync interval
• Files are transmitted as a series of UDP packets
• If any packet is lost or corrupted, the entire file is discarded by the receiver
• Files get multiple opportunities to transfer across as many sync intervals as fit within the --max-file-age setting
• No active acknowledgment or retransmission mechanism exists

Sliding Window Approach

The sender maintains a sliding window of files to attempt transfer:

• New and modified files enter the window
• Files remain in the window for a configurable period (--max-file-age)
• Multiple transmission attempts increase the probability of success
• The window advances based on time, not successful transfer

Configuration Options

Sender Options

python sender/ddsender.py \
  --mode filesync \
  --sync-path /home/user/documents \
  --cloud-ip YOUR_CLOUD_IP \
  --cloud-port 5010 \
  --key "your-base64-key-here" \
  --sync-interval 60 \
  --max-file-age 1 \
  --preserve-path \
  --include-dotfiles

Parameters:

• --sync-interval: How often to scan for changes (seconds)
• --max-file-age: How long to keep trying to transfer a file (hours)
• --preserve-path: Preserve the full path right back to the root of the filesystem (defaults to truncating the --sync-path off each file path)
• --include-dotfiles: Include hidden files (disabled by default)

Receiver Options

python filereceiver/ddfilereceiver.py \
    --udp-host 1.2.3.4 \
    --udp-port 5010 \
    --http-host 127.0.0.1 \
    --http-port 8080 \
    --key "your-base64-key-here" \
    --output-dir /path/to/filesync 

File Transfer Process

1. Sender scans the sync directory at each interval
2. File size and modification time are used to detect changes
3. Files are split into UDP-sized fragments
4. Fragments are sent as encrypted UDP packets
5. Receiver attempts to reassemble complete files
6. Only complete files are written to the output directory

Limitations and Workarounds

Very Large Files

Very large files are particularly problematic because a single lost packet causes the entire file to be discarded.

Mitigation Strategies:

• Increase --max-file-age to give more transfer opportunities
• Increase --sync-interval to attempt transfers more frequently
• Increase network MTU (Jumbo Packets) and --max-packet-size if supported
• Schedule periodic runs with --max-file-age 0 to catch straggler files
• Consider splitting very large files before transmission

Incomplete Transfers

Files that don't complete transfer remain invisible to the receiver. There is no partial file storage or resume capability.

Workarounds:

• Use shorter sync intervals for faster detection of failed transfers
• Do a full sync on some schedule, like once overnight

Directory Structure Preservation

By default, the receiver just recreates the subset of the folder structure from wherever the path was on the sender. Preservation of the entire path right from the filesystem root requires additional configuration.

Older files copied into structure

If you copy an old file into the sender's sync path and the timestamp is preserved, it won't get picked up for transmission. You'll need to make sure any files get a timestamp that fits within the --max-file-age window.

Optimization Strategies

To increase your success rates

• Reduce --sync-interval to attempt transfers more frequently
• Increase --max-file-age to give files more opportunities
• Run intensive syncs during low-traffic periods
• For critical files, schedule dedicated sync runs with --max-file-age 0

For Resource Conservation

• Increase --sync-interval to reduce scanning overhead
• Use multiple sender instances with different paths and intervals

Known Limitations

• No delta/partial transfers - entire file retransmitted each attempt

• No file deletion propagation - deletions are not synced

• No real-time notification of successful transfers

• Symbolic links and special files are not handled

• Return to the main README for basic usage