Daemon to continuously and incrementally synchronize a directory from remote object store to a local directory.
objinsync pull --exclude '**/__pycache__/**' s3://bucket/keyprefix ./localdir
When running in daemon mode (without
--once flag), a health check endpoint is
:8087/health and a prometheus metrics endpoint is served at
:8087/metrics. You can use
--status-addr to override the binding address.
Objinsync also comes with builtin Sentry integration. To enable it, set the
SENTRY_DSN environment variable.
You can also run objinsync in pull once mode, which behaves just like
aws s3 sync:
objinsync pull --once s3://bucket/keyprefix ./localdir
To use with Minio instead of S3, you can set
--disable-ssl flags for
pull command as you see fit.
--interval flags allows to configure the pull time interval, which is 5 seconds by default:
objinsync pull --interval 20s s3://bucket/keyprefix ./localdir
Enable debug logs by setting the
DEBUG environment variable
DEBUG=1 objinsync pull ...
Simply download the prebuilt single binary from release page or use
go get command:
go get github.com/scribd/objinsync
Pre-built docker images are available at https://github.com/orgs/scribd/packages/container/package/objinsync.
Run from source
AWS_REGION=us-east-2 go run main.go pull s3://qph-test-airflow-airflow-code/airflow_home/dags ./dags
To cut a release, push tag to remote in the format of