csm-backup in docker with dotnet

[ruXlab]

Hello here

I'm trying to get up and running backups on the linux via docker but stumbled upon this issue:

SCM Backup 1.4.0.ae7edc3
https://scm-backup.org/
Backing up config files
Getting repositories for source 'xxx-bitbucket' from hoster 'bitbucket'
Trying to find SCMs on this machine...
Found Git 2.20.1 
Backing up source: xxx
  Git: https://bitbucket.org/xxx
fatal: Unable to create temporary file '/scm-backup-out/xxx-bitbucket/xxxx#proj/repo/objects/pack/tmp_pack_XXXXXX': Permission denied
fatal: index-pack failed

Backup failed!
The application will close in 0 seconds!

I don't see what is wrong with permissions because if I do docker exec in this container I can easily create file:

# touch /scm-backup-out/xxx-bitbucket/xxxx#proj/repo/objects/pack//tmp_pack_XXXXXX
# ls -l /scm-backup-out/xxx-bitbucket/xxxx#proj/repo/objects/pack//tmp_pack_XXXXXX
-rw-r--r-- 1 1000 1000 0 Apr 26 13:11 /scm-backup-out/xxx-bitbucket/xxxx#proj/repo/objects/pack/tmp_pack_XXXXXX

Also, git clone inside container works fine as expected

---

My Dockerfile:

FROM mcr.microsoft.com/dotnet/core/sdk:3.1

WORKDIR /scm-backup

# Install .NET core and SCM Backup
RUN apt-get update && \
    apt-get -y install curl sudo wget unzip apt-transport-https git && \
    wget https://github.com/christianspecht/scm-backup/releases/download/1.4.0/scm-backup-1.4.0.ae7edc3.zip -O scm-backup.zip && \
    unzip scm-backup.zip

CMD dotnet ScmBackup.dll

Use:

➜  docker build . -t scm-backup
➜  docker run -it -v `pwd`/bitbucket-backups:/scm-backup-out:rw -v `pwd`/settings.yml:/scm-backup/settings.yml scm-backup 

[christianspecht]

Sorry, but I don't have any Linux and Docker experience.
I'm developing this on Windows and my only "contact" with Docker so far was setting up some CI on Bitbucket Pipelines, which happens to use Docker containers under the hood.

So I don't have the knowledge to directly help you with any permission issues.
And SCM Backup just calls the Git executable, so there's nothing special happening in SCM Backup that I could "fix".
That error message sounds like something deep from the bowels of Git.

---

Also, git clone inside container works fine as expected

SCM Backup uses git fetch, not git clone. Here are the exact parameters.

So for your example, it would execute this:

git -C "/scm-backup-out/xxx-bitbucket/xxxx#proj/repo/" fetch --force --prune https://bitbucket.org/xxxx/proj/ refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*

Does this work in your Docker container?

[ruXlab]

@christianspecht thanks for prompt response

I tried to reproduce what your code does but in Docker:

1. Create folder mkdir "/scm-backup-out/xxx-bitbucket/xxxx#proj/repo/" -p
2. I noted git init is used, so I did git init --bare in that folder
3. Run command above and it failed!

git -C "/scm-backup-out/xxx-bitbucket/xxxx#proj/repo/" fetch --force --prune https://github.com/christianspecht/scm-backup.git refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*
remote: Enumerating objects: 324, done.
remote: Counting objects: 100% (324/324), done.
remote: Compressing objects: 100% (194/194), done.
fatal: Unable to create temporary file '/scm-backup-out/xxx-bitbucket/xxxx#proj/repo/objects/pack/tmp_pack_XXXXXX': Permission denied
fatal: index-pack failed

So, you were right about issue - it was caused by git and/or docker themself. Will look into this later

Thank you!

---

It's very interesting - git clone works as expected

git clone https://github.com/christianspecht/scm-backup.git 
Cloning into 'scm-backup'...
remote: Enumerating objects: 324, done.
remote: Counting objects: 100% (324/324), done.
remote: Compressing objects: 100% (194/194), done.
remote: Total 3884 (delta 205), reused 212 (delta 128), pack-reused 3560
Receiving objects: 100% (3884/3884), 486.04 KiB | 1.36 MiB/s, done.
Resolving deltas: 100% (2756/2756), done.

So I'd assume it has to do with git init and the way it creates git index files

---

git init --bare produces valid permissions (uid 1000)

root@fa3ca74f5265:/scm-backup-out/xxx-bitbucket# tree -u    
.
`-- [1000    ]  xxxx#proj
    `-- [1000    ]  repo
        |-- [1000    ]  FETCH_HEAD
        |-- [1000    ]  HEAD
        |-- [1000    ]  branches
        |-- [1000    ]  config
        |-- [1000    ]  description
        |-- [1000    ]  hooks
        |   |-- [1000    ]  applypatch-msg.sample
        |   |-- [1000    ]  commit-msg.sample
        |   |-- [1000    ]  fsmonitor-watchman.sample
        |   |-- [1000    ]  post-update.sample
        |   |-- [1000    ]  pre-applypatch.sample
        |   |-- [1000    ]  pre-commit.sample
        |   |-- [1000    ]  pre-push.sample
        |   |-- [1000    ]  pre-rebase.sample
        |   |-- [1000    ]  pre-receive.sample
        |   |-- [1000    ]  prepare-commit-msg.sample
        |   `-- [1000    ]  update.sample
        |-- [1000    ]  info
        |   `-- [1000    ]  exclude
        |-- [1000    ]  objects
        |   |-- [1000    ]  info
        |   `-- [1000    ]  pack
        |       `-- [1000    ]  eeeeeeee
        `-- [1000    ]  refs
            |-- [1000    ]  heads
            `-- [1000    ]  tags

11 directories, 17 files

[christianspecht]

FYI: someone else got it to work in Docker, see #51

[juris]

Hi there, here is a sample of working Dockerfile:

FROM alpine:3.13

ARG YQ_VERSION=4.7.1
ARG SCM_FILE=scm-backup-1.5.3.7c34699.zip
ARG DOTNET_FILE=dotnet-runtime-3.1.14-linux-musl-x64.tar.gz
ARG DOTNET_FILE_SHA512=77856f6183ef7ee561fe8510e89148566972ae21e3915242ae4ac7ef987b1aa78cda09bb06fdae96cd03758975dc5eb0e8652dea79b96db327ac5de2a4d41961

ENV SCM_ROOT=/opt/scm-backup
ENV DOTNET_ROOT=/opt/dotnet
ENV PATH=${PATH}:${DOTNET_ROOT}

WORKDIR ${SCM_ROOT}

RUN apk add --no-cache \
 # Install deps
        bash icu-libs krb5-libs \
        libgcc libintl libssl1.1 \
        libstdc++ zlib curl git \
 # Install dotnet runtime
 && curl -fsL -o ${DOTNET_FILE} https://download.visualstudio.microsoft.com/download/pr/e7f88aae-a10e-4b49-8ccd-31f2df82cf3f/35fb27716a3b7543ef2889f0e4c19949/${DOTNET_FILE} \
 && echo "${DOTNET_FILE_SHA512}  ${DOTNET_FILE}" > ${DOTNET_FILE}.sha512 \
 && sha512sum -c ${DOTNET_FILE}.sha512 \
 && mkdir -p ${DOTNET_ROOT} \
 && tar zxf ${DOTNET_FILE} -C ${DOTNET_ROOT} \
 # Install scm-backup
 && curl -fsL -o ${SCM_FILE} https://github.com/christianspecht/scm-backup/releases/download/1.5.3/${SCM_FILE} \
 && unzip ${SCM_FILE} \
 && rm -f ${SCM_FILE} ${DOTNET_FILE} ${DOTNET_FILE}.sha512 \
 # Install yq (config templating)
 && curl -fSL https://github.com/mikefarah/yq/releases/download/v${YQ_VERSION}/yq_linux_amd64 -o /usr/local/bin/yq \
 && chmod +x /usr/local/bin/yq

COPY entrypoint.sh /usr/local/bin
COPY settings.yml .
RUN chmod +x /usr/local/bin/entrypoint.sh

ENTRYPOINT ["entrypoint.sh"]

Very raw entrypoint.sh for Dockerfile with variable replacement:

#!/usr/bin/env bash

yq e -i '(.sources[] | select(.title == "bitbucket_backup").name) = env(BITBUCKET_TEAM)' settings.yml
yq e -i '(.sources[] | select(.title == "bitbucket_backup").authName) = env(BITBUCKET_USER)' settings.yml
yq e -i '(.sources[] | select(.title == "bitbucket_backup").password) = env(BITBUCKET_PASS)' settings.yml

exec dotnet ScmBackup.dll

Sample settings.yml template with bitbucket config:

localFolder: 'backup'

sources:
  - title: bitbucket_backup
    hoster: bitbucket
    type: org
    name: team
    authName: username
    password: app_password

Just create all 3 files in a single directory and run docker build . -t scm-backup
Then you can use a container and mount your target backup repo to /opt/scm-backup/backup
E.g.

docker build . -t scm-backup
docker run -v ${PWD}:/opt/scm-backup/backup -e BITBUCKET_USER=mybbusername -e BITBUCKET_PASS=mybbpassword -e BITBUCKET_TEAM=mybbteam scm-backup

[christianspecht]

Closing this for now.

I just added a new section to the documentation which lists all Dockerfiles / images that I know of:
https://docs.scm-backup.org/en/latest/install/#docker