Add git-lfs to image #654

jameshearttech · 2022-12-22T22:39:40Z

I'm requesting git-lfs be added to the image. This issue came up back in 2019 #174. The issue was closed without adding git-lfs to the image. I tested v3.6.2 by adding git-lfs to the image and I'm syncing LFS files from BitBucket without any additional configuration.

This is my dockerfile.

FROM k8s.gcr.io/git-sync/git-sync:v3.6.2
USER root
RUN apt-get update
RUN apt-get -y install git-lfs
USER 65533:65533

This is my container spec within my deployment in k8s.

spec:
  template:
    spec:
      containers:
        - name: git-sync-lfs
          image: example-docker-hub/git-sync-lfs:v3.6.2
          args:
            - -ssh
            - -repo=git@bitbucket.org:example-private-repo/lfs.git
            - -root=/data
            - -dest=lfs
            - -branch=master
            - -depth=1
            - -wait=60
          resources:
            limits:
              cpu: 200m
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: lfs-data
              mountPath: /data
            - name: git-secret
              mountPath: /etc/git-secret
          securityContext:
            runAsUser: 65533

The text was updated successfully, but these errors were encountered:

thockin · 2022-12-22T23:25:04Z

I would want an e2e test for this - how do I force this to be used so I know it works?

jameshearttech · 2022-12-23T00:56:10Z

Do existing tests not cover this? Can I not just add git-lfs package in a Dockerfile using apt-get then run tests on new image?

My background is sysadmin. I'm new to devops. I know some bash, docker, and k8s. You want me to write a test to prove that it works? Not sure how to do that.

I forked the repo. I'm looking at Dockerfile.in and test_e2e.sh. I'm not sure where to add git-lfs package in Dockerfile.in. Do I pass it an image:tag as an argument to run tests?

thockin · 2022-12-23T01:17:55Z

If you can show me how to craft an e2e setup which specifically proves LFS works, that would be great. You can see in that file that each testcase is a little snip of shell code which sets up a scenario. What I don't quiote grok is how to set up a repo to use LFS and how to *prove* that it works.

…

On Thu, Dec 22, 2022 at 4:56 PM James Ringer ***@***.***> wrote: Do existing tests not cover this? My background is sysadmin. I'm new to devops. I know some bash. You want me to write a test to prove that it works? Not sure how to do that. I went ahead and forked the repo. I'm looking at test_e2e.sh. How can I help? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

jameshearttech · 2022-12-23T01:23:16Z

Does this help? GitHub - Configuring Git LFS

GitHub provides some storage and bandwidth for LFS for free. GitHub - About Storage and Bandwidth

We can do a call and work through it together if you'd like. I set it up on BitBucket, but it's probably similar on GitHub. The BitBucket repo I have setup is private, but I could setup a public repo if that helps.

Afaik the only thing that will be different after adding the git-lfs package will be that if a repo contains lfs files they will be synced rather than the text files that point to them. This makes me think existing tests should be fine, but I'm far from and expert.

thockin · 2022-12-23T03:00:24Z

To do an e2e I need some way to replicate it locally or at least without any auth to an external service. Possible?

…

On Thu, Dec 22, 2022, 5:23 PM James Ringer ***@***.***> wrote: Does this help? GitHub - Configuring Git LFS <https://docs.github.com/en/repositories/working-with-files/managing-large-files/configuring-git-large-file-storage> We can do a call and work through it together if that helps. I set it up on BitBucket, but it's probably similar on GitHub. — Reply to this email directly, view it on GitHub <#654 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVG5E3GCH32QE7KMKMDWOT5I7ANCNFSM6AAAAAATHFYSJ4> . You are receiving this because you commented.Message ID: ***@***.***>

jameshearttech · 2022-12-23T03:28:17Z

If you setup a public GitHub repo with LFS I don't think you need authentication.

jameshearttech · 2022-12-23T15:09:57Z

Do the existing tests use a git repo to test? Maybe just add an LFS file to that repo?

thockin · 2022-12-23T18:37:16Z

The existing e2e tests use a `file://` repo or an SSH repo (still backed by files). It looks like we can build an LFS test-server binary ( https://github.com/git-lfs/lfs-test-server) to run tests against, but there's not a standard apt pkg for it, so we'll have to pull and build it as part of the test. Look in _test_tools. That's not impossible, it just makes this a little more involved. At least it is written in Go, so we can drop it into tools/go.mod. :) If you have bandwidth, it's appreciated, otherwise I'll get to this eventually, I hope. :)

…

On Fri, Dec 23, 2022 at 7:10 AM James Ringer ***@***.***> wrote: Do the existing tests use a git repo to test? Maybe just add an LFS file to that repo? — Reply to this email directly, view it on GitHub <#654 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVFGCCNKCP5ONWDXENDWOW6E7ANCNFSM6AAAAAATHFYSJ4> . You are receiving this because you commented.Message ID: ***@***.***>

jameshearttech · 2022-12-24T22:49:58Z

In my fork I have setup lfs-test-server using the sshd folder as an example.

_test_tools
├── lfs-test-server
│   ├── Dockerfile
│   ├── README.md
│   └── run.sh
├── ncsvr
│   ├── Dockerfile
│   ├── README.md
│   └── ncsvr.sh
└── sshd
    ├── Dockerfile
    ├── README.md
    ├── sshd.sh
    └── sshd_config

I am using env vars LFS_USERNAME and LFS_PASSWORD to configure lfs-test-server management username and password. Looking at sshd example I'm assuming we don't want these committed to the repository. Not sure how I'm going to do that. If you don't set these the management interface is not accessible. Also, it looks like you have to login and create a user to use lfs-test-server. Not how I'm going to work around that either. Any ideas?

thockin · 2022-12-26T23:12:32Z

Ugh, what a cumbersome piece of software to automate. How about something like this Dockerfile:

FROM alpine AS base                                                                                                                                   

RUN mkdir -p /out/etc/apk && cp -r /etc/apk/* /out/etc/apk/
RUN apk add --no-cache --initdb -p /out \
    alpine-baselayout \
    apk-tools \
    curl \
    && true

###############

FROM scratch

ENTRYPOINT []
WORKDIR /

COPY --from=base /out/ /

COPY lfs-test-server /

RUN LFS_ADMINUSER=admin \
    LFS_ADMINPASS=admin \
    timeout 2 \
    /lfs-test-server & \
    sleep 0.5 \
    && curl -s localhost:8080/mgmt/add -u admin:admin -X POST -F name=e2e -F password=e2e

ENTRYPOINT ["/lfs-test-server"]

I did a quick test and that SEEMS to work, as long as lfs-test-server is built with CGO_ENABLED=0 (static binary).

Now you can run the image in docker, get the IP (see ssh examples) and us username "e2e" password "e2e".

jameshearttech · 2022-12-27T00:34:47Z

I take it from your comment committing the credentials in git is not an issue? I saw in sshd that the certs were in a volume that was not committed.

thockin · 2022-12-27T00:39:25Z

That's because it's not easy to make "trivial" SSH keys, whereas this case is super easy :)

…

On Mon, Dec 26, 2022 at 4:34 PM James Ringer ***@***.***> wrote: I take it from your comment committing the credentials in git is not an issue? I saw in sshd that the certs were in a volume that was not committed. — Reply to this email directly, view it on GitHub <#654 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVFVFQ4LEKCP6F7JCZLWPI2TDANCNFSM6AAAAAATHFYSJ4> . You are receiving this because you commented.Message ID: ***@***.***>

jameshearttech · 2022-12-27T00:43:37Z

That makes sense. I just wanted to check before moving forward. Btw adding the user with curl that way is just what I was looking for. I'll keep working on this. Thanks.

jameshearttech · 2023-01-26T05:57:08Z

Haven't circled back to this yet, but I will. Please don't let the issue go stale and close. Thanks!

thockin · 2023-02-26T21:22:14Z

Hi. I am hoping to have a v4.0.0 release candidate some time in the next month or so - would love to have this in. No pressure, we can always add it later.

jameshearttech · 2023-02-26T21:30:01Z

I'll put some time toward this over the next few weeks.

jameshearttech · 2023-03-12T10:28:06Z

I'm working on test_e2e.sh. I added a section to init lfs-test-server.

# Init Git LFS Test Server

# Clone lfs-test-server to _test_tools
git clone https://github.com/git-lfs/lfs-test-server ./_test_tools/lfs-test-server

# Overwrite lfs-test-server Dockerfile
tee ./_test_tools/lfs-test-server/Dockerfile <<EOF
FROM golang:1.18
WORKDIR /go/src/github.com/git-lfs/lfs-test-server
COPY . .
RUN go build
EXPOSE 8080
ENV LFS_ADMINUSER=admin
ENV LFS_ADMINPASS=admin
RUN /go/src/github.com/git-lfs/lfs-test-server/lfs-test-server & \
    sleep 0.5 \
    && curl -s localhost:8080/mgmt/add -u admin:admin -X POST -F name=e2e -F password=e2e
ENTRYPOINT ["/go/src/github.com/git-lfs/lfs-test-server/lfs-test-server"]
EOF

# Build the lfs-test-server image
docker build ./_test_tools/lfs-test-server -t example.com/lfs-test-server:latest

# Run the lfs-test-server container
docker run --rm -d -p 8080:8080/tcp example.com/lfs-test-server:latest

The shell snippet works. Idk if that's what you had in mind. I'm also looking at modifying the init_repo function.

function init_repo() {
    rm -rf "$REPO"
    mkdir -p "$REPO"
    git -C "$REPO" init -q -b "$MAIN_BRANCH"
    touch "$REPO"/file
    git -C "$REPO" add file
    git -C "$REPO" commit -aqm "init file"

    cd "$REPO"
    git lfs install
    git lfs track "*.jpg"
    cd ..
    echo "[lfs]" > "$REPO"/.lfsconfig
    echo "    url = \"http://localhost:8080/\"" >> "$REPO"/.lfsconfig
    git -C "$REPO" add .lfsconfig .gitattributes
    git -C "$REPO" commit -aqm "lfs config"

    touch "$REPO"/lfs.jpg
    git -C "$REPO" add lfs.jpg
    git -C "$REPO" commit -aqm "add lfs.jpg"

    rm -rf "$REPO2"
    cp -r "$REPO" "$REPO2"
}

To confirm the lfs tracked file is sent to lfs-test-server we need to git push, but for that we need a remote.

thockin · 2023-03-12T15:32:37Z

I don't think you need to do this in the common init_repo, but in the specific LFS test case?

…

On Sun, Mar 12, 2023, 3:28 AM James Ringer ***@***.***> wrote: I'm working on test_e2e.sh. I added a section to init lfs-test-server. # Init Git LFS Test Server # Clone lfs-test-server to _test_tools git clone https://github.com/git-lfs/lfs-test-server ./_test_tools/lfs-test-server # Overwrite lfs-test-server Dockerfile tee ./_test_tools/lfs-test-server/Dockerfile <<EOF FROM golang:1.18 WORKDIR /go/src/github.com/git-lfs/lfs-test-server COPY . . RUN go build EXPOSE 8080 ENV LFS_ADMINUSER=admin ENV LFS_ADMINPASS=admin RUN /go/src/github.com/git-lfs/lfs-test-server/lfs-test-server & \ sleep 0.5 \ && curl -s localhost:8080/mgmt/add -u admin:admin -X POST -F name=e2e -F password=e2e ENTRYPOINT ["/go/src/github.com/git-lfs/lfs-test-server/lfs-test-server"] EOF # Build the lfs-test-server image docker build ./_test_tools/lfs-test-server -t example.com/lfs-test-server:latest # Run the lfs-test-server container docker run --rm -d -p 8080:8080/tcp example.com/lfs-test-server:latest The shell snippet works. Idk if that's what you had in mind. I'm also looking at modifying the init_repo function. function init_repo() { rm -rf "$REPO" mkdir -p "$REPO" git -C "$REPO" init -q -b "$MAIN_BRANCH" touch "$REPO"/file git -C "$REPO" add file git -C "$REPO" commit -aqm "init file" cd "$REPO" git lfs install git lfs track "*.jpg" cd .. echo "[lfs]" > "$REPO"/.lfsconfig echo " url = \"http://localhost:8080/\"" >> "$REPO"/.lfsconfig git -C "$REPO" add .lfsconfig .gitattributes git -C "$REPO" commit -aqm "lfs config" touch "$REPO"/lfs.jpg git -C "$REPO" add lfs.jpg git -C "$REPO" commit -aqm "add lfs.jpg" rm -rf "$REPO2" cp -r "$REPO" "$REPO2" } To confirm the lfs tracked file is sent to lfs-test-server we need to git push, but for that we need a remote. — Reply to this email directly, view it on GitHub <#654 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVAHFJNHWUSATBMSGLDW3WQMDANCNFSM6AAAAAATHFYSJ4> . You are receiving this because you commented.Message ID: ***@***.***>

jameshearttech · 2023-03-13T19:00:34Z

I've been looking at the repo trying to understand how to do this, but I don't quite follow how everything works. Would it be better if I put the script at /test-lfs.sh?

thockin · 2023-03-14T01:46:05Z

Do you have a branch pushed to GitHub? I can clone it later this week and see what I see.

…

On Mon, Mar 13, 2023, 12:00 PM James Ringer ***@***.***> wrote: I've been looking at the repo trying to understand how to do this, but I don't quite follow how everything works. Would it be better if I put the script at /test-lfs.sh? — Reply to this email directly, view it on GitHub <#654 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVDC4EU4YWEQ22QIPHDW35VF3ANCNFSM6AAAAAATHFYSJ4> . You are receiving this because you commented.Message ID: ***@***.***>

jameshearttech · 2023-03-17T15:41:40Z

Tbh, there's not much to see at this point. After working on it a bit last weekend I have a better idea of which direction to go. I'm going to work on it again this weekend a bit more, so we'll see how it goes.

jameshearttech · 2023-03-20T04:40:11Z

I worked with it a bit more and made a little progress. I got a lfs-test-server container along with a git-server-docker container setup and was able to push a commit to git-server-docker containing a jpg that was pushed to lfs-test-server.

I'll throw a bit more time at it next weekend. I feel it's starting to come together, but there is still a ways to go. Ultimately, I want to automate the setup of an environment where we have a repo in git-server-docker and a large file in lfs-test-server such that when we clone the repo with git-sync it pulls the large file with the repo.

Are we on the same page with that goal?

thockin · 2023-03-21T22:01:38Z

I think that's right - enough to prove that LFS was activated and works.

…

On Sun, Mar 19, 2023 at 9:40 PM James Ringer ***@***.***> wrote: I worked with it a bit more and made a little progress. I got a lfs-test-server container along with a git-server-docker container setup and was able to push a commit to git-server-docker containing a jpg that was pushed to lfs-test-server. I'll throw a bit more time at it next weekend. I feel it's starting to come together, but there is still a ways to go. Ultimately, I want to automate the setup of an environment where we have a repo in git-server-docker and a large file in lfs-test-server such that when we clone the repo with git-sync it pulls the large file with the repo. Are we on the same page with that goal? — Reply to this email directly, view it on GitHub <#654 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVEEDNZECMWQG2CD7UTW47NTLANCNFSM6AAAAAATHFYSJ4> . You are receiving this because you commented.Message ID: ***@***.***>

thockin · 2023-04-07T17:26:16Z

Circling around to open topics - any progress here?

jameshearttech · 2023-04-07T17:29:44Z

I have not had time the past couple of weeks. Tomorrow I'll have some time to pick this up again. Idk how far I'll get, but I'll keep at it until it's done.

jameshearttech · 2023-04-21T19:05:08Z

I talked with my manager and received approval to start dedicating some of my work time to OSS. This is at the top of my list!

thockin · 2023-04-21T19:20:18Z

That's awesome! Congrats.

jameshearttech · 2023-04-24T15:17:26Z

I updated my fork. It's not working yet and it's rough, but if you want to take a look.

jameshearttech · 2023-06-13T15:19:28Z

@thockin I have not forgotten about this, but I'm on a modified schedule until the end of June with no time for oss. I'll circle back to this in July. Sorry I'm slow and inexperienced.

k8s-triage-robot · 2024-01-22T11:34:03Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-02-21T11:40:33Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2024-03-22T12:27:53Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot · 2024-03-22T12:27:57Z

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen

Mark this issue as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jameshearttech · 2024-03-22T15:29:57Z

Bad bot.

thockin · 2024-03-25T16:08:49Z

I can re-open if you still want to tackle this, but the bot just sweeps up debris which is often left behind by well-intentioned people who just can't find/make the time to finish (IOW, me, frequently).

jameshearttech mentioned this issue Mar 17, 2023

Add git-lfs to image and e2e test #699

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 22, 2024

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 21, 2024

k8s-ci-robot closed this as not planned Won't fix, can't repro, duplicate, stale Mar 22, 2024

thockin reopened this Mar 25, 2024

thockin removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Mar 25, 2024

Add git-lfs to image #654

Add git-lfs to image #654

Comments

jameshearttech commented Dec 22, 2022

thockin commented Dec 22, 2022

jameshearttech commented Dec 23, 2022 • edited

thockin commented Dec 23, 2022 via email

jameshearttech commented Dec 23, 2022 • edited

thockin commented Dec 23, 2022 via email

jameshearttech commented Dec 23, 2022

jameshearttech commented Dec 23, 2022

thockin commented Dec 23, 2022 via email

jameshearttech commented Dec 24, 2022 • edited

thockin commented Dec 26, 2022 • edited

jameshearttech commented Dec 27, 2022 • edited

thockin commented Dec 27, 2022 via email

jameshearttech commented Dec 27, 2022

jameshearttech commented Jan 26, 2023

thockin commented Feb 26, 2023

jameshearttech commented Feb 26, 2023

jameshearttech commented Mar 12, 2023

thockin commented Mar 12, 2023 via email

jameshearttech commented Mar 13, 2023

thockin commented Mar 14, 2023 via email

jameshearttech commented Mar 17, 2023

jameshearttech commented Mar 20, 2023 • edited

thockin commented Mar 21, 2023 via email

thockin commented Apr 7, 2023

jameshearttech commented Apr 7, 2023

jameshearttech commented Apr 21, 2023 • edited

thockin commented Apr 21, 2023

jameshearttech commented Apr 24, 2023 • edited

jameshearttech commented Jun 13, 2023

k8s-triage-robot commented Jan 22, 2024

k8s-triage-robot commented Feb 21, 2024

k8s-triage-robot commented Mar 22, 2024

k8s-ci-robot commented Mar 22, 2024

jameshearttech commented Mar 22, 2024

thockin commented Mar 25, 2024

jameshearttech commented Dec 23, 2022 •

edited

jameshearttech commented Dec 23, 2022 •

edited

jameshearttech commented Dec 24, 2022 •

edited

thockin commented Dec 26, 2022 •

edited

jameshearttech commented Dec 27, 2022 •

edited

jameshearttech commented Mar 20, 2023 •

edited

jameshearttech commented Apr 21, 2023 •

edited

jameshearttech commented Apr 24, 2023 •

edited