Skip to content

fix apply chown permissions in parallel for large workspace #73

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 16, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -375,7 +375,9 @@ pipeline {
// Build Docker container for push to LS Repo
stage('Build-Single') {
when {
environment name: 'MULTIARCH', value: 'false'
expression {
env.MULTIARCH == 'false' || params.PACKAGE_CHECK == 'true'
}
environment name: 'EXIT_STATUS', value: ''
}
steps {
Expand All @@ -400,7 +402,10 @@ pipeline {
// Build MultiArch Docker containers for push to LS Repo
stage('Build-Multi') {
when {
environment name: 'MULTIARCH', value: 'true'
allOf {
environment name: 'MULTIARCH', value: 'true'
expression { params.PACKAGE_CHECK == 'false' }
}
environment name: 'EXIT_STATUS', value: ''
}
parallel {
Expand Down Expand Up @@ -505,7 +510,7 @@ pipeline {
sh '''#! /bin/bash
set -e
TEMPDIR=$(mktemp -d)
if [ "${MULTIARCH}" == "true" ]; then
if [ "${MULTIARCH}" == "true" ] && [ "${PACKAGE_CHECK}" == "false" ]; then
LOCAL_CONTAINER=${IMAGE}:amd64-${META_TAG}
else
LOCAL_CONTAINER=${IMAGE}:${META_TAG}
Expand Down Expand Up @@ -566,7 +571,7 @@ pipeline {
steps {
sh '''#! /bin/bash
echo "Packages were updated. Cleaning up the image and exiting."
if [ "${MULTIARCH}" == "true" ]; then
if [ "${MULTIARCH}" == "true" ] && [ "${PACKAGE_CHECK}" == "false" ]; then
docker rmi ${IMAGE}:amd64-${META_TAG}
else
docker rmi ${IMAGE}:${META_TAG}
Expand All @@ -590,7 +595,7 @@ pipeline {
steps {
sh '''#! /bin/bash
echo "There are no package updates. Cleaning up the image and exiting."
if [ "${MULTIARCH}" == "true" ]; then
if [ "${MULTIARCH}" == "true" ] && [ "${PACKAGE_CHECK}" == "false" ]; then
docker rmi ${IMAGE}:amd64-${META_TAG}
else
docker rmi ${IMAGE}:${META_TAG}
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,7 @@ Once registered you can define the dockerfile to use with `-f Dockerfile.aarch64

## Versions

* **16.09.21:** - Fix slow `chown` on large workspace (contents of workspace folder no longer chowned).
* **11.07.21:** - Bump node to 14 to fix builds
* **08.05.21:** - Fix doc link
* **04.02.20:** - Allow setting gui password via hash using env var `HASHED_PASSWORD`.
Expand Down
1 change: 1 addition & 0 deletions readme-vars.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ app_setup_block: |

# changelog
changelogs:
- { date: "16.09.21:", desc: "Fix slow `chown` on large workspace (contents of workspace folder no longer chowned)." }
- { date: "11.07.21:", desc: "Bump node to 14 to fix builds" }
- { date: "08.05.21:", desc: "Fix doc link" }
- { date: "04.02.20:", desc: "Allow setting gui password via hash using env var `HASHED_PASSWORD`." }
Expand Down
19 changes: 17 additions & 2 deletions root/etc/cont-init.d/30-config
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,20 @@ if [ -n "${SUDO_PASSWORD}" ] || [ -n "${SUDO_PASSWORD_HASH}" ]; then
fi

# permissions
chown -R abc:abc \
/config
if [ -f "/usr/bin/find" ] && [ -f "/usr/bin/xargs" ]; then
CORES=$(nproc --all)

# Split workload between config and workspace
echo "setting permissions::configuration"
find /config -path /config/workspace -prune -false -o -type d -print0 | \
xargs --null -r --max-args=1 --max-procs=$((CORES*2*8)) \
chown -R abc:abc

echo "setting permissions::workspace"
chown abc:abc /config/workspace
else
# Set permissions on data mount
# do not decend into the workspace
chown -R abc:abc "$(ls /config -I workspace)"
chown abc:abc /config/workspace
fi