Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use rsync to copy pack contents #415

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Use rsync to copy pack contents #415

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c7e9f09
Use rsync to copy pack contents
cognifloyd Apr 12, 2024
c87f49f
Don't preserve ownership when copying pack contents (#418)
cognifloyd May 9, 2024
c0774d6
More rsync/cp cleanups
cognifloyd May 9, 2024
dfaaca9
Merge branch 'master' into rsync
cognifloyd May 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
* Stop generating the DataStore Secret (#385) and checksum labels (#391) when existing secret provided or disabled (by @bmarick)
* Stop generating the checksum labels for Auth Secret (#392) when existing secret provided or disabled (by @bmarick)
* Use `image.pullPolicy` for all containers including init containers that use `image.utilityImage`. (#397) (by @jk464)
* Use `rsync` to copy pack contents when available, falling back to `cp`. (#414) (by @cognifloyd)

## v1.0.0
* Bump to latest CircleCI orb versions (kubernetes@1.3.1 and helm@3.0.0 by @ZoeLeah)
Expand Down
30 changes: 22 additions & 8 deletions templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -343,9 +343,14 @@ Merge packs and virtualenvs from st2 with those from st2packs images
command:
- 'sh'
- '-ec'
- |
/bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
/bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
- >
if command rsync; then
rsync -a /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-a is equivalent to -rlptgoD: https://www.mankier.com/1/rsync#--archive

  • -r: copy directories recursively
  • -l: recreate symlinks in destination
  • -p: permissions
  • -t: modification times
  • -g: copy gid (group id) by name
  • -o: copy oid (owner id) by name
  • -D: devices (character and block) and specials (sockets, fifos, etc)

Of those, I'm confident we need -rlp or -rlE (where -E is short for --executability, a subset of -p). Depending on which user is running this, we might also want -og. The other parts of -a are probably not necessary.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'll push a commit that switches to -rlptD based on rsync...Stealthii:stackstorm-k8s:rsync

rsync -a /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared;
else
/bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-a is equivalent to -dR --preserve=all:

  • -d: is equivalent to -P --preserve=links
  • -P: no dereference / never follow symbolic links in SOURCE
  • -R: copy directories recursively
  • --preserve=all: preserve file attributes
    • mode
    • ownership for user and group
    • timestamps
    • links for hard links
    • context for security context
    • xattr for extended attributes

Of these we need -dR --preserve=mode (we need mode to maintain the execute bits of action files). Depending on which user is copying the file, we might also need --preserve=ownership.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'll push a commit that switches to -RP --preserve=mode,timestamps,links,xattr based on rsync...Stealthii:stackstorm-k8s:rsync

/bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared;
fi
{{- with .securityContext | default $.Values.st2actionrunner.securityContext | default $.Values.securityContext }}
{{/* st2actionrunner is likely the most permissive so use that if defined. */}}
securityContext: {{- toYaml . | nindent 8 }}
Expand All @@ -365,9 +370,14 @@ Merge packs and virtualenvs from st2 with those from st2packs images
command:
- 'sh'
- '-ec'
- |
/bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
/bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
- >
if command rsync; then
rsync -a /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
rsync -a /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared;
else
/bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
/bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
fi
{{- with .Values.st2actionrunner.securityContext | default .Values.securityContext }}
{{/* st2actionrunner is likely the most permissive so use that if defined. */}}
securityContext: {{- toYaml . | nindent 8 }}
Expand All @@ -386,8 +396,12 @@ Merge packs and virtualenvs from st2 with those from st2packs images
command:
- 'sh'
- '-ec'
- |
/bin/cp -aR /opt/stackstorm/configs/. /opt/stackstorm/configs-shared
- >
if command rsync; then
rsync -a /opt/stackstorm/configs/. /opt/stackstorm/configs-shared;
else
/bin/cp -aR /opt/stackstorm/configs/. /opt/stackstorm/configs-shared;
fi
{{- with .Values.st2actionrunner.securityContext | default .Values.securityContext }}
{{/* st2actionrunner is likely the most permissive so use that if defined. */}}
securityContext: {{- toYaml . | nindent 8 }}
Expand Down