diff --git a/.kanban/changes/ABC-2/README.md b/.kanban/changes/ABC-2/README.md new file mode 100644 index 00000000..38f135ad --- /dev/null +++ b/.kanban/changes/ABC-2/README.md @@ -0,0 +1,52 @@ +# ABC-2 Audit Trail + +## Summary + +ABC-2 implements repository clone caching for GitHub issue #138. + +Requirement: + +> "Что бы один и тот же репозиторий лежал бы в кеше и мы бы грузили данные из кеша + git pull под нужную нам ветку" + +Final behavior: + +- Project containers mount the shared cache volume at `/home/dev/.docker-git/.cache`. +- Repository cache mirrors are stored under `/home/dev/.docker-git/.cache/git-mirrors/.git`. +- Warm-cache clones use the refreshed bare mirror as the clone source. +- The mirror is used only after a successful authenticated refresh from the real remote. +- Before using a mirror as clone source, the generated entrypoint verifies and repairs bare mirror `HEAD` to an existing `refs/heads/*` ref. +- PR/MR refs still fetch the requested ref from the authenticated upstream URL after clone. + +## Commits + +- `1cb29d1 fix(core): clone repositories from warm mirror cache` +- `90b98a8 fix(core): guard warm mirror clone reuse` + +## Changed Files + +- `packages/lib/src/core/templates-entrypoint/tasks.ts` +- `packages/app/src/lib/core/templates-entrypoint/tasks.ts` +- `packages/lib/tests/core/templates.test.ts` + +## Invariants + +- `refresh_success(cache, remote) -> may_clone_from(cache)` +- `refresh_failure(cache, remote) -> clone_source = authenticated_remote` +- `may_clone_from(cache) -> exists(cache.HEAD) && cache.HEAD in refs/heads/*` +- `repoUrl equality -> same mirror key` +- `requested repoRef preserved in final working tree` + +## Review Closure + +The first implementation introduced two P1 risks: + +- A mirror bootstrapped from an `issue-*` fallback could retain `HEAD` pointing to a local-only branch that later gets pruned. +- A private or stale mirror could be used after authenticated refresh failure, bypassing remote access checks. + +Commit `90b98a8` closes both risks: + +- Cache source assignment now lives only in the successful refresh branch. +- The mirror `HEAD` is validated with `show-ref` and repaired with `symbolic-ref` before use. + +See `review.md` and `verification.md` for details. + diff --git a/.kanban/changes/ABC-2/files.md b/.kanban/changes/ABC-2/files.md new file mode 100644 index 00000000..9dd9869d --- /dev/null +++ b/.kanban/changes/ABC-2/files.md @@ -0,0 +1,23 @@ +# ABC-2 File Trace + +## Runtime Template + +`packages/lib/src/core/templates-entrypoint/tasks.ts` + +- Defines clone-cache initialization and mirror refresh. +- Ensures mirror source is used only after successful refresh. +- Repairs mirror `HEAD` before using it as clone source. + +`packages/app/src/lib/core/templates-entrypoint/tasks.ts` + +- Synchronized application copy of the runtime template. + +## Test Coverage + +`packages/lib/tests/core/templates.test.ts` + +- Captures generated shell invariants for clone-cache behavior. +- Guards against broad remote refs. +- Guards against reintroducing cache use after refresh failure. +- Guards mirror `HEAD` validation/repair before cache source reuse. + diff --git a/.kanban/changes/ABC-2/review.md b/.kanban/changes/ABC-2/review.md new file mode 100644 index 00000000..7272a5c3 --- /dev/null +++ b/.kanban/changes/ABC-2/review.md @@ -0,0 +1,56 @@ +# ABC-2 Review Notes + +## Issue Alignment + +The implementation matches issue #138 by keeping a single shared bare mirror per repository URL and using that mirror for warm clone data. The generated runtime still refreshes the mirror from the authenticated remote first, which preserves access checks and remote freshness. + +## Risk Review + +### P1: Invalid Mirror HEAD + +Risk: + +An `issue-*` fallback clone can create a local branch and bootstrap the bare mirror with `HEAD` pointing at that local-only branch. A later mirror refresh can prune that ref, after which cloning directly from the mirror can produce an unborn or empty checkout. + +Resolution: + +The entrypoint now computes `CACHE_HEAD_REF`, verifies it exists with: + +```bash +git --git-dir "$CACHE_REPO_DIR" show-ref --verify --quiet "$CACHE_HEAD_REF" +``` + +If it is missing, the entrypoint selects the first existing branch from: + +```bash +refs/heads/main refs/heads/master refs/heads +``` + +Then it repairs `HEAD` via: + +```bash +git --git-dir "$CACHE_REPO_DIR" symbolic-ref HEAD "$CACHE_HEAD_REF" +``` + +The cache is used as clone source only if this succeeds. + +### P1: Cache Use After Auth/Refresh Failure + +Risk: + +Using a warm mirror after `git fetch` fails can bypass private repo access checks and return stale data. + +Resolution: + +`CLONE_SOURCE_REPO_URL="$CACHE_REPO_DIR"` is assigned only inside the successful mirror refresh branch. On refresh failure, the clone source remains `CLONE_SOURCE_REPO_URL="$AUTH_REPO_URL"`. + +## Regression Coverage + +`packages/lib/tests/core/templates.test.ts` asserts: + +- mirror refresh uses branch/tag-only refspecs; +- clone source defaults to `$AUTH_REPO_URL`; +- `$CACHE_REPO_DIR` becomes clone source only in the successful fetch path; +- mirror `HEAD` is checked and repaired before use; +- `--reference-if-able` is not used by the warm-cache path. + diff --git a/.kanban/changes/ABC-2/verification.md b/.kanban/changes/ABC-2/verification.md new file mode 100644 index 00000000..67869f52 --- /dev/null +++ b/.kanban/changes/ABC-2/verification.md @@ -0,0 +1,46 @@ +# ABC-2 Verification + +## Passed Checks + +The following checks were run in the workspace and passed: + +```bash +bun run --cwd packages/lib test -- core/templates.test.ts +bun run --cwd packages/lib typecheck +bun run --cwd packages/app typecheck +bun run --cwd packages/app build:docker-git +bun run typecheck +bun run check +``` + +## Docker Runtime Verification + +The stock clone-cache e2e script was run against the reachable docker-git +controller in this remote-Docker environment: + +```bash +DOCKER_GIT_API_URL=http://172.18.0.3:3336 \ +DOCKER_GIT_API_CONTAINER_NAME=docker-git-api-cloudflared \ +DOCKER_GIT_E2E_CLONE_CACHE_TIMEOUT=900s \ + bash scripts/e2e/clone-cache.sh +``` + +Result: + +```text +e2e/clone-cache: cache reuse verified for https://github.com/octocat/Hello-World/issues/1 +``` + +Environment notes: + +- `DOCKER_HOST=tcp://host.docker.internal:2375` requires an explicit `DOCKER_GIT_API_URL`. +- The controller container is named `docker-git-api-cloudflared`; setting `DOCKER_GIT_API_CONTAINER_NAME` lets the e2e helper inspect the nested project Docker daemon. +- A shorter `300s` first attempt expired while cold-pulling/building the base runtime image, before clone-cache assertions could run. + +## Current Workspace State + +At archive time: + +- working tree was clean before creating this audit trail; +- final code commits were present on branch `vk/2562-github-138`; +- archive artifacts are stored under `.kanban/changes/ABC-2`. diff --git a/packages/app/src/lib/core/templates-entrypoint/tasks.ts b/packages/app/src/lib/core/templates-entrypoint/tasks.ts index 1889eb05..6cd9b498 100644 --- a/packages/app/src/lib/core/templates-entrypoint/tasks.ts +++ b/packages/app/src/lib/core/templates-entrypoint/tasks.ts @@ -130,6 +130,7 @@ const cloneCacheRefreshRefspecs = "'+refs/heads/*:refs/heads/*' '+refs/tags/*:re const renderCloneCacheInit = (config: TemplateConfig): string => ` CLONE_CACHE_ARGS="" + CLONE_SOURCE_REPO_URL="$AUTH_REPO_URL" CACHE_REPO_DIR="" CACHE_ROOT="/home/${config.sshUser}/.docker-git/.cache/git-mirrors" if command -v sha256sum >/dev/null 2>&1; then @@ -146,11 +147,21 @@ const renderCloneCacheInit = (config: TemplateConfig): string => chown 1000:1000 "$CACHE_ROOT" || true if [[ -d "$CACHE_REPO_DIR" ]]; then if su - ${config.sshUser} -c "git --git-dir '$CACHE_REPO_DIR' rev-parse --is-bare-repository >/dev/null 2>&1"; then - if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git --git-dir '$CACHE_REPO_DIR' fetch --progress --prune '$AUTH_REPO_URL' ${cloneCacheRefreshRefspecs}"; then + if su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git --git-dir '$CACHE_REPO_DIR' fetch --progress --prune '$AUTH_REPO_URL' ${cloneCacheRefreshRefspecs}"; then + CACHE_HEAD_REF="$(git --git-dir "$CACHE_REPO_DIR" symbolic-ref -q HEAD 2>/dev/null || true)" + if [[ -z "$CACHE_HEAD_REF" ]] || ! git --git-dir "$CACHE_REPO_DIR" show-ref --verify --quiet "$CACHE_HEAD_REF"; then + CACHE_HEAD_REF="$(git --git-dir "$CACHE_REPO_DIR" for-each-ref --format='%(refname)' refs/heads/main refs/heads/master refs/heads | head -n 1 || true)" + fi + if [[ -n "$CACHE_HEAD_REF" ]] && git --git-dir "$CACHE_REPO_DIR" symbolic-ref HEAD "$CACHE_HEAD_REF"; then + CLONE_SOURCE_REPO_URL="$CACHE_REPO_DIR" + CLONE_CACHE_ARGS="--no-local" + echo "[clone-cache] using mirror: $CACHE_REPO_DIR" + else + echo "[clone-cache] mirror has no usable HEAD for $REPO_URL" + fi + else echo "[clone-cache] mirror refresh failed for $REPO_URL" fi - CLONE_CACHE_ARGS="--reference-if-able '$CACHE_REPO_DIR' --dissociate" - echo "[clone-cache] using mirror: $CACHE_REPO_DIR" else echo "[clone-cache] invalid mirror removed: $CACHE_REPO_DIR" rm -rf "$CACHE_REPO_DIR" @@ -170,19 +181,19 @@ const renderCloneBodyRef = (config: TemplateConfig): string => String.raw` if [[ -n "$REPO_REF" ]]; then if [[ "$REPO_REF" == refs/pull/* || "$REPO_REF" == refs/merge-requests/* ]]; then REF_BRANCH="$(printf "%s" "$REPO_REF" | sed -E 's#^refs/pull/([^/]+)/head$#pr-\1#; s#^refs/merge-requests/([^/]+)/head$#mr-\1#')" - if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS '$AUTH_REPO_URL' '$TARGET_DIR'"; then + if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS '$CLONE_SOURCE_REPO_URL' '$TARGET_DIR'"; then echo "[clone] git clone failed for $REPO_URL" CLONE_OK=0 else - if ! su - ${config.sshUser} -c "cd '$TARGET_DIR' && GIT_TERMINAL_PROMPT=0 git fetch --progress origin '$REPO_REF':'$REF_BRANCH' && git checkout '$REF_BRANCH'"; then + if ! su - ${config.sshUser} -c "cd '$TARGET_DIR' && GIT_TERMINAL_PROMPT=0 git fetch --progress '$AUTH_REPO_URL' '$REPO_REF':'$REF_BRANCH' && git checkout '$REF_BRANCH'"; then echo "[clone] git fetch failed for $REPO_REF" CLONE_OK=0 fi fi else - if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS --branch '$REPO_REF' '$AUTH_REPO_URL' '$TARGET_DIR'"; then + if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS --branch '$REPO_REF' '$CLONE_SOURCE_REPO_URL' '$TARGET_DIR'"; then echo "[clone] branch '$REPO_REF' missing; retrying without --branch" - if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS '$AUTH_REPO_URL' '$TARGET_DIR'"; then + if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS '$CLONE_SOURCE_REPO_URL' '$TARGET_DIR'"; then echo "[clone] git clone failed for $REPO_URL" CLONE_OK=0 elif [[ "$REPO_REF" == issue-* ]]; then @@ -194,7 +205,7 @@ const renderCloneBodyRef = (config: TemplateConfig): string => fi fi else - if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS '$AUTH_REPO_URL' '$TARGET_DIR'"; then + if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS '$CLONE_SOURCE_REPO_URL' '$TARGET_DIR'"; then echo "[clone] git clone failed for $REPO_URL" CLONE_OK=0 fi diff --git a/packages/lib/src/core/templates-entrypoint/tasks.ts b/packages/lib/src/core/templates-entrypoint/tasks.ts index 1889eb05..6cd9b498 100644 --- a/packages/lib/src/core/templates-entrypoint/tasks.ts +++ b/packages/lib/src/core/templates-entrypoint/tasks.ts @@ -130,6 +130,7 @@ const cloneCacheRefreshRefspecs = "'+refs/heads/*:refs/heads/*' '+refs/tags/*:re const renderCloneCacheInit = (config: TemplateConfig): string => ` CLONE_CACHE_ARGS="" + CLONE_SOURCE_REPO_URL="$AUTH_REPO_URL" CACHE_REPO_DIR="" CACHE_ROOT="/home/${config.sshUser}/.docker-git/.cache/git-mirrors" if command -v sha256sum >/dev/null 2>&1; then @@ -146,11 +147,21 @@ const renderCloneCacheInit = (config: TemplateConfig): string => chown 1000:1000 "$CACHE_ROOT" || true if [[ -d "$CACHE_REPO_DIR" ]]; then if su - ${config.sshUser} -c "git --git-dir '$CACHE_REPO_DIR' rev-parse --is-bare-repository >/dev/null 2>&1"; then - if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git --git-dir '$CACHE_REPO_DIR' fetch --progress --prune '$AUTH_REPO_URL' ${cloneCacheRefreshRefspecs}"; then + if su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git --git-dir '$CACHE_REPO_DIR' fetch --progress --prune '$AUTH_REPO_URL' ${cloneCacheRefreshRefspecs}"; then + CACHE_HEAD_REF="$(git --git-dir "$CACHE_REPO_DIR" symbolic-ref -q HEAD 2>/dev/null || true)" + if [[ -z "$CACHE_HEAD_REF" ]] || ! git --git-dir "$CACHE_REPO_DIR" show-ref --verify --quiet "$CACHE_HEAD_REF"; then + CACHE_HEAD_REF="$(git --git-dir "$CACHE_REPO_DIR" for-each-ref --format='%(refname)' refs/heads/main refs/heads/master refs/heads | head -n 1 || true)" + fi + if [[ -n "$CACHE_HEAD_REF" ]] && git --git-dir "$CACHE_REPO_DIR" symbolic-ref HEAD "$CACHE_HEAD_REF"; then + CLONE_SOURCE_REPO_URL="$CACHE_REPO_DIR" + CLONE_CACHE_ARGS="--no-local" + echo "[clone-cache] using mirror: $CACHE_REPO_DIR" + else + echo "[clone-cache] mirror has no usable HEAD for $REPO_URL" + fi + else echo "[clone-cache] mirror refresh failed for $REPO_URL" fi - CLONE_CACHE_ARGS="--reference-if-able '$CACHE_REPO_DIR' --dissociate" - echo "[clone-cache] using mirror: $CACHE_REPO_DIR" else echo "[clone-cache] invalid mirror removed: $CACHE_REPO_DIR" rm -rf "$CACHE_REPO_DIR" @@ -170,19 +181,19 @@ const renderCloneBodyRef = (config: TemplateConfig): string => String.raw` if [[ -n "$REPO_REF" ]]; then if [[ "$REPO_REF" == refs/pull/* || "$REPO_REF" == refs/merge-requests/* ]]; then REF_BRANCH="$(printf "%s" "$REPO_REF" | sed -E 's#^refs/pull/([^/]+)/head$#pr-\1#; s#^refs/merge-requests/([^/]+)/head$#mr-\1#')" - if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS '$AUTH_REPO_URL' '$TARGET_DIR'"; then + if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS '$CLONE_SOURCE_REPO_URL' '$TARGET_DIR'"; then echo "[clone] git clone failed for $REPO_URL" CLONE_OK=0 else - if ! su - ${config.sshUser} -c "cd '$TARGET_DIR' && GIT_TERMINAL_PROMPT=0 git fetch --progress origin '$REPO_REF':'$REF_BRANCH' && git checkout '$REF_BRANCH'"; then + if ! su - ${config.sshUser} -c "cd '$TARGET_DIR' && GIT_TERMINAL_PROMPT=0 git fetch --progress '$AUTH_REPO_URL' '$REPO_REF':'$REF_BRANCH' && git checkout '$REF_BRANCH'"; then echo "[clone] git fetch failed for $REPO_REF" CLONE_OK=0 fi fi else - if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS --branch '$REPO_REF' '$AUTH_REPO_URL' '$TARGET_DIR'"; then + if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS --branch '$REPO_REF' '$CLONE_SOURCE_REPO_URL' '$TARGET_DIR'"; then echo "[clone] branch '$REPO_REF' missing; retrying without --branch" - if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS '$AUTH_REPO_URL' '$TARGET_DIR'"; then + if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS '$CLONE_SOURCE_REPO_URL' '$TARGET_DIR'"; then echo "[clone] git clone failed for $REPO_URL" CLONE_OK=0 elif [[ "$REPO_REF" == issue-* ]]; then @@ -194,7 +205,7 @@ const renderCloneBodyRef = (config: TemplateConfig): string => fi fi else - if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS '$AUTH_REPO_URL' '$TARGET_DIR'"; then + if ! su - ${config.sshUser} -c "GIT_TERMINAL_PROMPT=0 git clone --progress $CLONE_CACHE_ARGS '$CLONE_SOURCE_REPO_URL' '$TARGET_DIR'"; then echo "[clone] git clone failed for $REPO_URL" CLONE_OK=0 fi diff --git a/packages/lib/tests/core/templates.test.ts b/packages/lib/tests/core/templates.test.ts index b357a86e..64671119 100644 --- a/packages/lib/tests/core/templates.test.ts +++ b/packages/lib/tests/core/templates.test.ts @@ -299,11 +299,29 @@ describe("renderEntrypoint clone cache", () => { const entrypoint = renderEntrypoint(makeTemplateConfig()) expect(entrypoint).toContain("git --git-dir '$CACHE_REPO_DIR' fetch") + expect(entrypoint).toContain('CLONE_SOURCE_REPO_URL="$AUTH_REPO_URL"') + expect(entrypoint).toContain('CLONE_SOURCE_REPO_URL="$CACHE_REPO_DIR"') + expect(entrypoint).toContain('CLONE_CACHE_ARGS="--no-local"') + expect(entrypoint).toContain("if su - dev -c \"GIT_TERMINAL_PROMPT=0 git --git-dir '$CACHE_REPO_DIR' fetch") + expect(entrypoint).toContain('CACHE_HEAD_REF="$(git --git-dir "$CACHE_REPO_DIR" symbolic-ref -q HEAD') + expect(entrypoint).toContain('git --git-dir "$CACHE_REPO_DIR" show-ref --verify --quiet "$CACHE_HEAD_REF"') + expect(entrypoint).toContain("for-each-ref --format='%(refname)' refs/heads/main refs/heads/master refs/heads") + expect(entrypoint).toContain('git --git-dir "$CACHE_REPO_DIR" symbolic-ref HEAD "$CACHE_HEAD_REF"') + expect(entrypoint).toContain("git clone --progress $CLONE_CACHE_ARGS '$CLONE_SOURCE_REPO_URL' '$TARGET_DIR'") expect(entrypoint).toContain("'+refs/heads/*:refs/heads/*'") expect(entrypoint).toContain("'+refs/tags/*:refs/tags/*'") + expect(entrypoint).toContain("git fetch --progress '$AUTH_REPO_URL' '$REPO_REF':'$REF_BRANCH'") expect(entrypoint).not.toContain("'+refs/*:refs/*'") expect(entrypoint).not.toContain("'+refs/pull/*:refs/pull/*'") expect(entrypoint).not.toContain("'+refs/merge-requests/*:refs/merge-requests/*'") + expect(entrypoint).not.toContain("--reference-if-able") + expect(entrypoint).not.toContain("if ! su - dev -c \"GIT_TERMINAL_PROMPT=0 git --git-dir '$CACHE_REPO_DIR' fetch") + + const refreshFailureBlock = entrypoint.slice( + entrypoint.indexOf('echo "[clone-cache] mirror refresh failed for $REPO_URL"'), + entrypoint.indexOf('echo "[clone-cache] invalid mirror removed: $CACHE_REPO_DIR"') + ) + expect(refreshFailureBlock).not.toContain('CLONE_SOURCE_REPO_URL="$CACHE_REPO_DIR"') }) it("preserves branch/tag-only clone-cache refspecs for generated configs", () => {