Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

csm-backup in docker with dotnet #49

Closed
ruXlab opened this issue Apr 26, 2020 · 5 comments
Closed

csm-backup in docker with dotnet #49

ruXlab opened this issue Apr 26, 2020 · 5 comments

Comments

@ruXlab
Copy link

ruXlab commented Apr 26, 2020

Hello here

I'm trying to get up and running backups on the linux via docker but stumbled upon this issue:

SCM Backup 1.4.0.ae7edc3
https://scm-backup.org/
Backing up config files
Getting repositories for source 'xxx-bitbucket' from hoster 'bitbucket'
Trying to find SCMs on this machine...
Found Git 2.20.1 
Backing up source: xxx
  Git: https://bitbucket.org/xxx
fatal: Unable to create temporary file '/scm-backup-out/xxx-bitbucket/xxxx#proj/repo/objects/pack/tmp_pack_XXXXXX': Permission denied
fatal: index-pack failed

Backup failed!
The application will close in 0 seconds!

I don't see what is wrong with permissions because if I do docker exec in this container I can easily create file:

# touch /scm-backup-out/xxx-bitbucket/xxxx#proj/repo/objects/pack//tmp_pack_XXXXXX
# ls -l /scm-backup-out/xxx-bitbucket/xxxx#proj/repo/objects/pack//tmp_pack_XXXXXX
-rw-r--r-- 1 1000 1000 0 Apr 26 13:11 /scm-backup-out/xxx-bitbucket/xxxx#proj/repo/objects/pack/tmp_pack_XXXXXX

Also, git clone inside container works fine as expected


My Dockerfile:

FROM mcr.microsoft.com/dotnet/core/sdk:3.1

WORKDIR /scm-backup

# Install .NET core and SCM Backup
RUN apt-get update && \
    apt-get -y install curl sudo wget unzip apt-transport-https git && \
    wget https://github.com/christianspecht/scm-backup/releases/download/1.4.0/scm-backup-1.4.0.ae7edc3.zip -O scm-backup.zip && \
    unzip scm-backup.zip

CMD dotnet ScmBackup.dll

Use:

➜  docker build . -t scm-backup
➜  docker run -it -v `pwd`/bitbucket-backups:/scm-backup-out:rw -v `pwd`/settings.yml:/scm-backup/settings.yml scm-backup 
@christianspecht
Copy link
Owner

Sorry, but I don't have any Linux and Docker experience.
I'm developing this on Windows and my only "contact" with Docker so far was setting up some CI on Bitbucket Pipelines, which happens to use Docker containers under the hood.

So I don't have the knowledge to directly help you with any permission issues.
And SCM Backup just calls the Git executable, so there's nothing special happening in SCM Backup that I could "fix".
That error message sounds like something deep from the bowels of Git.


Also, git clone inside container works fine as expected

SCM Backup uses git fetch, not git clone. Here are the exact parameters.

So for your example, it would execute this:

git -C "/scm-backup-out/xxx-bitbucket/xxxx#proj/repo/" fetch --force --prune https://bitbucket.org/xxxx/proj/ refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*

Does this work in your Docker container?

@ruXlab
Copy link
Author

ruXlab commented Apr 26, 2020

@christianspecht thanks for prompt response

I tried to reproduce what your code does but in Docker:

  1. Create folder mkdir "/scm-backup-out/xxx-bitbucket/xxxx#proj/repo/" -p
  2. I noted git init is used, so I did git init --bare in that folder
  3. Run command above and it failed!
git -C "/scm-backup-out/xxx-bitbucket/xxxx#proj/repo/" fetch --force --prune https://github.com/christianspecht/scm-backup.git refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*
remote: Enumerating objects: 324, done.
remote: Counting objects: 100% (324/324), done.
remote: Compressing objects: 100% (194/194), done.
fatal: Unable to create temporary file '/scm-backup-out/xxx-bitbucket/xxxx#proj/repo/objects/pack/tmp_pack_XXXXXX': Permission denied
fatal: index-pack failed

So, you were right about issue - it was caused by git and/or docker themself. Will look into this later

Thank you!


It's very interesting - git clone works as expected

git clone https://github.com/christianspecht/scm-backup.git 
Cloning into 'scm-backup'...
remote: Enumerating objects: 324, done.
remote: Counting objects: 100% (324/324), done.
remote: Compressing objects: 100% (194/194), done.
remote: Total 3884 (delta 205), reused 212 (delta 128), pack-reused 3560
Receiving objects: 100% (3884/3884), 486.04 KiB | 1.36 MiB/s, done.
Resolving deltas: 100% (2756/2756), done.

So I'd assume it has to do with git init and the way it creates git index files


git init --bare produces valid permissions (uid 1000)

root@fa3ca74f5265:/scm-backup-out/xxx-bitbucket# tree -u    
.
`-- [1000    ]  xxxx#proj
    `-- [1000    ]  repo
        |-- [1000    ]  FETCH_HEAD
        |-- [1000    ]  HEAD
        |-- [1000    ]  branches
        |-- [1000    ]  config
        |-- [1000    ]  description
        |-- [1000    ]  hooks
        |   |-- [1000    ]  applypatch-msg.sample
        |   |-- [1000    ]  commit-msg.sample
        |   |-- [1000    ]  fsmonitor-watchman.sample
        |   |-- [1000    ]  post-update.sample
        |   |-- [1000    ]  pre-applypatch.sample
        |   |-- [1000    ]  pre-commit.sample
        |   |-- [1000    ]  pre-push.sample
        |   |-- [1000    ]  pre-rebase.sample
        |   |-- [1000    ]  pre-receive.sample
        |   |-- [1000    ]  prepare-commit-msg.sample
        |   `-- [1000    ]  update.sample
        |-- [1000    ]  info
        |   `-- [1000    ]  exclude
        |-- [1000    ]  objects
        |   |-- [1000    ]  info
        |   `-- [1000    ]  pack
        |       `-- [1000    ]  eeeeeeee
        `-- [1000    ]  refs
            |-- [1000    ]  heads
            `-- [1000    ]  tags

11 directories, 17 files

@christianspecht
Copy link
Owner

FYI: someone else got it to work in Docker, see #51

@ruXlab ruXlab mentioned this issue Apr 28, 2020
@juris
Copy link

juris commented Apr 30, 2021

Hi there, here is a sample of working Dockerfile:

FROM alpine:3.13

ARG YQ_VERSION=4.7.1
ARG SCM_FILE=scm-backup-1.5.3.7c34699.zip
ARG DOTNET_FILE=dotnet-runtime-3.1.14-linux-musl-x64.tar.gz
ARG DOTNET_FILE_SHA512=77856f6183ef7ee561fe8510e89148566972ae21e3915242ae4ac7ef987b1aa78cda09bb06fdae96cd03758975dc5eb0e8652dea79b96db327ac5de2a4d41961

ENV SCM_ROOT=/opt/scm-backup
ENV DOTNET_ROOT=/opt/dotnet
ENV PATH=${PATH}:${DOTNET_ROOT}

WORKDIR ${SCM_ROOT}

RUN apk add --no-cache \
 # Install deps
        bash icu-libs krb5-libs \
        libgcc libintl libssl1.1 \
        libstdc++ zlib curl git \
 # Install dotnet runtime
 && curl -fsL -o ${DOTNET_FILE} https://download.visualstudio.microsoft.com/download/pr/e7f88aae-a10e-4b49-8ccd-31f2df82cf3f/35fb27716a3b7543ef2889f0e4c19949/${DOTNET_FILE} \
 && echo "${DOTNET_FILE_SHA512}  ${DOTNET_FILE}" > ${DOTNET_FILE}.sha512 \
 && sha512sum -c ${DOTNET_FILE}.sha512 \
 && mkdir -p ${DOTNET_ROOT} \
 && tar zxf ${DOTNET_FILE} -C ${DOTNET_ROOT} \
 # Install scm-backup
 && curl -fsL -o ${SCM_FILE} https://github.com/christianspecht/scm-backup/releases/download/1.5.3/${SCM_FILE} \
 && unzip ${SCM_FILE} \
 && rm -f ${SCM_FILE} ${DOTNET_FILE} ${DOTNET_FILE}.sha512 \
 # Install yq (config templating)
 && curl -fSL https://github.com/mikefarah/yq/releases/download/v${YQ_VERSION}/yq_linux_amd64 -o /usr/local/bin/yq \
 && chmod +x /usr/local/bin/yq

COPY entrypoint.sh /usr/local/bin
COPY settings.yml .
RUN chmod +x /usr/local/bin/entrypoint.sh

ENTRYPOINT ["entrypoint.sh"]

Very raw entrypoint.sh for Dockerfile with variable replacement:

#!/usr/bin/env bash

yq e -i '(.sources[] | select(.title == "bitbucket_backup").name) = env(BITBUCKET_TEAM)' settings.yml
yq e -i '(.sources[] | select(.title == "bitbucket_backup").authName) = env(BITBUCKET_USER)' settings.yml
yq e -i '(.sources[] | select(.title == "bitbucket_backup").password) = env(BITBUCKET_PASS)' settings.yml

exec dotnet ScmBackup.dll

Sample settings.yml template with bitbucket config:

localFolder: 'backup'

sources:
  - title: bitbucket_backup
    hoster: bitbucket
    type: org
    name: team
    authName: username
    password: app_password

Just create all 3 files in a single directory and run docker build . -t scm-backup
Then you can use a container and mount your target backup repo to /opt/scm-backup/backup
E.g.

docker build . -t scm-backup
docker run -v ${PWD}:/opt/scm-backup/backup -e BITBUCKET_USER=mybbusername -e BITBUCKET_PASS=mybbpassword -e BITBUCKET_TEAM=mybbteam scm-backup

christianspecht added a commit to christianspecht/scm-backup-docs that referenced this issue Aug 25, 2021
@christianspecht
Copy link
Owner

Closing this for now.

I just added a new section to the documentation which lists all Dockerfiles / images that I know of:
https://docs.scm-backup.org/en/latest/install/#docker

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants