rst
The SupRsync agent keeps a local directory synced with a remote server. It continuously copies over new files to its destination, verifying the copy by checking the md5sum and deleting the local files after a specified amount of time if the local and remote checksums match.
Below is an example of what the SupRsync configuration might look like on a smurf-server, with an instance copying g3 files and an instance copying smurf auxiliary files.
Below is an example configuration that copies files to the base directory /path/to/base/dir on a remote system. This will delete successfully transferred files after 7 days, or 604800 seconds:
{'agent-class': 'SupRsync',
'instance-id': 'timestream-sync',
'arguments':[
'--archive-name', 'timestreams',
'--remote-basedir', '/path/to/base/dir/timestreams',
'--db-path', '/data/so/dbs/suprsync.db',
'--ssh-host', '<user>@<hostname>',
'--ssh-key', '<path_to_ssh_key>',
'--delete-local-after', '604800',
'--max-copy-attempts', '10',
'--copy-timeout', '60',
'--cmd-timeout', '5',
'--suprsync-file-root', '/data/so/suprsync',
]},
{'agent-class': 'SupRsync',
'instance-id': 'smurf-sync',
'arguments':[
'--archive-name', 'smurf',
'--remote-basedir', '/path/to/base/dir/smurf',
'--db-path', '/data/so/dbs/suprsync.db',
'--ssh-host', '<user>@<hostname>',
'--ssh-key', '<path_to_ssh_key>',
'--delete-local-after', '604800',
'--max-copy-attempts', '10',
'--copy-timeout', '20',
'--cmd-timeout', '5',
'--suprsync-file-root', '/data/so/suprsync',
]},Note
Make sure you add the public-key corresponding to the ssh id file you will be using to the remote server using the ssh-copy-id function.
Note
If for some reason the user running the suprsync agent does not have file permissions, if the --delete-local-after param is set the agent will crash when trying to delete files. In such an environment, make sure this option is not used.
Below is a sample docker-compose entry for the SupRsync agents. Because the data we'll be transfering is owned by the cryo:smurf user, we set that as the user of the agent so it has the correct permissions. This is only possible because the cryo:smurf user is already built into the SuprSync docker:
ocs-timestream-sync:
image: simonsobs/socs:latest
hostname: ocs-docker
user: cryo:smurf
network_mode: host
container_name: ocs-timestream-sync
environment:
- INSTANCE_ID=timestream-sync
- SITE_HUB=ws://${CB_HOST}:8001/ws
- SITE_HTTP=http://${CB_HOST}:8001/call
volumes:
- ${OCS_CONFIG_DIR}:/config
- /data:/data
- /home/cryo/.ssh:/home/cryo/.ssh
ocs-smurf-sync:
image: simonsobs/socs:latest
hostname: ocs-docker
user: cryo:smurf
network_mode: host
container_name: ocs-smurf-sync
environment:
- INSTANCE_ID=smurf-sync
- SITE_HUB=ws://${CB_HOST}:8001/ws
- SITE_HTTP=http://${CB_HOST}:8001/call
volumes:
- ${OCS_CONFIG_DIR}:/config
- /data:/data
- /home/cryo/.ssh:/home/cryo/.sshNote
If copying to a remote host from a docker container it is required that the corresponding ssh-key is also mounted into the container, and that it belongs to the user inside of the container with permission 600. The configuration above works because the cryo user in the container is the same as the one on the host, so the .ssh directory is already properly configured. For instance if using an ocs user in the docker, you may want to add a .ssh directory for the ocs user on the host (with correct permissions) and mount that directory into the container.
The SupRsync agent works by monitoring a table of files to be copied in a sqlite database. The full table is described here<SupRsyncFiles>, but importantly it contains information such as the full local path, the remote path relative to a base directory determined by the SupRsync cfg, checksumming info, several timestamps, and an "archive" name which is used to determine which SupRsync agent should manage each file. This makes it possible for multiple SupRsync agents to monitor the same db, but write their archives to different base-directories or remote computers.
Note
This agent does not remove empty directories, since I can't think of a foolproof way to determine whether or not more files will be written to a given directory, and pysmurf output directories will likely have unregistered files that stick around even after copied files have been removed. I think it's probably best to have a separate cronjob make that determination and remove directory husks.
To add a file to the SupRsync database, you can use the SupRsyncFilesManager class as follows:
from socs.db.suprsync import SupRsyncFilesManager
local_abspath = '/path/to/local/file.g3'
remote_relpath = 'remote/path/file.g3'
archive = 'timestreams'
srfm = SupRsyncFilesManager('/path/to/suprsync.db')
srfm.add_file(local_abspath, remote_relpath, archive)The SupRsync agent will then copy the file /path/to/local/file.g3 to the <remote_basedir>/remote/path/file.g3 on the remote server (or locally if none is specified). If the --delete-local-after option is used, the original file will be deleted after the specified amount of time.
The primary use case of this agent is copying files from the smurf-server to a daq node or simons1. The pysmurf monitor agent now populates the files table with both pysmurf auxiliary files (using the archive name "smurf") and g3 timestream files (using the archive name "timestreams"), as described in this<pysmurf_monitor_suprsync_db> section. On the smurf-server we'll be running one SupRsync agent for each of these two archives.
For data packaging, file destinations are usually grouped into timecode directories, based on the first 5-digits of the ctime at which the file was created. Destination paths (or remote_paths in the db) will typically look like:
<remote_basedir>/<5-digit timecode>/<any-number-of-subdirs>/<filename>where the remote-basedir is set in the suprsync site config, and everything after is what is registered as the remote_path in the suprsync database.
With this schema, it is important to transmit information that will allow downstream processors to determine whether or not a timecode directory is complete, as opposed to not copied over yet.
The SupRsync database will now automatically detect the timecode directory of new files that are added, and will track whether a timecode directory is complete (meaning suprsync expects no new files will be added) and synced (that all files in the timecode directory have been synced).
Once a timecode directory is complete and fully synced, suprsync will write a finalization file:
<timestamp>_<archive_name>_<dir timecode>_finalized.yamlthat will contain information about the timecode directory, including the number of files that this suprsync instance has synced over, the sub-directories that this suprsync instance has added to, the instance-id of the suprsync agent, and the finalization time. The directory on the local host that these files will be written to is set using the --suprsync-file-root argument, and the path on the remote host will be automatically generated.
A timecode directory will be marked as complete if any of the following are true:
- There are files in the suprsync archive written to a newer timecode directory
tc_now - tc_directory > 1, or we are roughly 1 day past when the timecode directory ended.
For example, say we have a suprsync agent running on smurf-srv20 with instance-id smurf-sync-srv20, with suprsync-file-root = /data/so/suprsync. Suppose that at timestamp 1686152766 this agent detects that all files in the timecode directory 16750 of the smurf archive have been successfully synced. This agent will write the local file:
/data/so/suprsync/16861/smurf-sync-srv20/1686152766_smurf_16750_finalized.yamlThis file will be synced to the remote path:
<remote_basedir>/16861/suprsync/smurf-sync-srv20/1686152766_smurf_16750_finalized.yamlwhere it can be processed by downstream data packaging software.
socs.agents.suprsync.agent.SupRsync
socs.db.suprsync.SupRsyncFile
socs.db.suprsync.SupRsyncFilesManager
socs.db.suprsync.TimecodeDir