Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add an archive mode #16

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Add an archive mode #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3a7eb3b
watch: Add an archive mode
mirsal Dec 2, 2020
35d489b
README.md: Document the archive option
mirsal Dec 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,13 @@ docker run -d --name my-data-container \
elementar/s3-volume --force-restore /data s3://mybucket/someprefix
```

### Running in archive mode

If the `--archive` option is set, the container will run without performing an
initial restore, even if the data directory is empty. Additionally, running in
archive mode disables the delete flag on aws s3 sync requests, so remote content
will not be deleted even if it is not present locally.

### Using Compose and named volumes

Most of the time, you will use this image to sync data for another container.
Expand Down
23 changes: 19 additions & 4 deletions watch
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ cat <<-EOF

--force-restore restore even if local directory is not empty

--archive disable initial restore and do not delete
remote content that's missing locally

eg: $PROGNAME /data s3://bucket/dir
EOF
}
Expand All @@ -18,7 +21,7 @@ function error_exit {
exit 1
}

PARSED_OPTIONS=$(getopt -n "$0" -o f --long "force-restore" -- "$@")
PARSED_OPTIONS=$(getopt -n "$0" -o fa --long "force-restore,archive" -- "$@")
if [ $? -ne 0 ]; then
exit 1
fi
Expand All @@ -30,6 +33,10 @@ while true; do
FORCE_RESTORE="true"
shift;;

-a|--archive)
ARCHIVE="true"
shift;;

--)
shift
break;;
Expand All @@ -46,6 +53,12 @@ else
AWS=aws
fi

if [ ${ARCHIVE:false} == 'true' ]; then
DELETE="--delete"
else
DELETE=
fi

function restore {
if [ "$(ls -A $LOCAL)" ]; then
if [[ ${FORCE_RESTORE:false} == 'true' ]]; then
Expand All @@ -61,15 +74,15 @@ function restore {

function backup {
echo "backup $LOCAL => $REMOTE"
if ! $AWS s3 sync "$LOCAL" "$REMOTE" --delete; then
if ! $AWS s3 sync "$LOCAL" "$REMOTE" $DELETE; then
echo "backup failed" 1>&2
return 1
fi
}

function final_backup {
echo "backup $LOCAL => $REMOTE"
while ! $AWS s3 sync "$LOCAL" "$REMOTE" --delete; do
while ! $AWS s3 sync "$LOCAL" "$REMOTE" $DELETE; do
echo "backup failed, will retry" 1>&2
sleep 1
done
Expand All @@ -85,7 +98,9 @@ function idle {
done
}

restore
if [ ${ARCHIVE:false} == 'false' -o ${FORCE_RESTORE:false} == 'true' ]; then
restore
fi

trap final_backup SIGHUP SIGINT SIGTERM
trap "backup; idle" USR1
Expand Down