refactor sync-to-s3.sh per directory#623
Conversation
|
|
||
| set +e | ||
| $AWS_CLI s3 sync "${AWS_ARGS[@]}" "$DIRECTORY" "s3://$S3_SYNC_BUCKET/$SAFENAME/" | ||
| set -e |
There was a problem hiding this comment.
This final set -e does nothing, I intend for it to prevent human error keeping the flag active if the script was extended in the future.
| {%- for directory, entry in pillar.sync.directories|items %} | ||
| {%- set minute = (loop.index0 * 5) % 60 %} | ||
| {{minute}} 03,15 * * * root /home/sysadmin-tools/bin/sync-to-s3.sh {{ directory }} | ||
| {%- if entry and 'exclude' in entry %} --exclude "{{ entry.exclude }}"{% endif %} |
There was a problem hiding this comment.
Just checking: This puts the cron job onto two lines. Does cron read onto the second line?
There was a problem hiding this comment.
Also, maybe do (I think items is safe on None?):
{%- for option, value in entry|items %} --{{ option }} "{{ value }}"{% endfor %}
| AWS_ARGS=(--only-show-errors --delete) | ||
|
|
||
| set +e | ||
| $AWS_CLI s3 sync "$DIRECTORY" "s3://$S3_SYNC_BUCKET/$SAFENAME/" --only-show-errors --delete 2>&1 | grep -v "You did not provide the number of bytes specified by the Content-Length HTTP header" |
There was a problem hiding this comment.
Do we not need this anymore?
2>&1 | grep -v "You did not provide the number of bytes specified by the Content-Length HTTP header"
| echo "Unknown option" | ||
| exit 1 | ||
| ;; | ||
| *) |
There was a problem hiding this comment.
This seems a bit paranoid, as we control the Pillar data, and should only be setting real args. I think we can pass through args directly.
2 participants
This PR modifies the sync-to-s3.sh to sync one directory rather than many, this allows us to add directory specific features (exclude paths #618) without the feature being applied to all paths.
The main complexity of backing up directories has been moved into Salt creating a cron for each sync path and setting features as required - this should prove more extendable in the future.
Closes #618
Connected tasks: