GitHub Clone & Import Pipelines

GitHub Clone & Import Pipelines

Two sibling core services handle GitHub repository acquisition. src/core/github-clone.ts is the desktop/main-process clone helper used through the github:clone IPC handler. src/core/github-import.ts is the Electron-free Server Edition import service used through the server's github:import RPC. Both consume a short-lived, credential-bearing clone URL minted by Termsprawl Cloud, validate it, shallow-clone with git clone --depth 1, guard the destination, and redact secret material from errors.

Module Map

Path	Responsibility	Key exports / behavior
src/core/github-clone.ts	Desktop twin of the import service. Clones a cloud-minted clone URL into a caller-provided absolute destination.	cloneRepo, GitHubCloneDeps, CloneRequest, CloneResult
src/core/github-import.ts	Server Edition import service. Validates owner/repo, asks the cloud for a one-shot clone URL, shallow-clones into destRoot/<repoName>, and lists repo suggestions.	importGitHubRepo, listSuggestedRepos, GitHubImportDeps, ImportRequest, ImportResult, SuggestedRepo
src/core/github-clone.test.ts	Unit coverage for the local clone helper with injected spawnFn.	Happy path, URL refusal, destination guard, empty-dir allowance, redacted stderr
src/core/github-import.test.ts	Unit coverage for import and suggestion listing with injected fetchFn/spawnFn.	No network or real git; token/URL leak assertions
scripts/github-import-e2e.sh	E2E proof for the Phase 17 GitHub import path. Boots a fake cloud plus a real ts-space container and drives the server RPC over WebSocket.	github:suggest, github:import, workspace snapshot, credential-leak checks, re-import refusal

Call Chains

Desktop local clone

1. Termsprawl Cloud mints a short-lived bearer clone URL.
2. The desktop main process invokes cloneRepo(deps, { url, dest }) via the github:clone IPC handler.
3. cloneRepo validates the URL with isAllowedCloneUrl: only https: on github.com.
4. It refuses a non-empty existing destination; an existing empty directory is allowed because git tolerates it.
5. It spawns git clone --depth 1 <url> <dest> through the injected spawnFn or default execFile.
6. On failure, it redacts the URL from git stderr before throwing.
7. It returns { path: dest, ok: true }; the URL and credential are never returned to the renderer.

Server Edition import

1. The server RPC github:import calls importGitHubRepo(deps, { fullName, destRoot }).
2. fullName must match the strict owner/repo shape and cannot contain traversal segments.
3. The destination is resolved as join(destRoot, basename(fullName).replace(/\.git$/, '')); a non-empty existing destination is refused.
4. The service sends POST /api/v1/github/import-url with Authorization: Bearer <bootToken> and JSON { fullName }.
5. A 404 becomes github_not_connected; other non-OK responses use the cloud error string when present, otherwise a generic status message.
6. The returned { url } is validated with the same https/github.com allowlist before git runs.
7. git clone --depth 1 <cloneUrl> <dest> runs through the injected/default spawn.
8. The result is { name, path, fullName }; the clone URL and boot token are not returned.

Boot repo suggestions

The E2E script asserts a boot broadcast named github:suggest. The core computation is listSuggestedRepos(deps, projectNames):

1. GET /api/v1/github/repos with the boot token.
2. Normalize either a raw array or { repos: [...] }.
3. Drop repos whose names are already represented in projectNames.
4. Also treat the last path segment of an existing project cwd or remote URL as taken, plus the suffix of any clone_url.
5. Never throw: network errors, non-OK responses, malformed JSON, and non-array bodies all degrade to [].

flowchart TD
  Desktop["github:clone IPC handler"] --> Clone["cloneRepo"]
  Clone --> V1["Validate https + github.com"]
  V1 --> D1["Refuse non-empty dest"]
  D1 --> G1["git clone --depth 1 url dest"]
  G1 --> R1["Return path, ok true"]

  Server["github:import RPC"] --> Import["importGitHubRepo"]
  Import --> V2["Validate owner/repo fullName"]
  V2 --> D2["Resolve destRoot/repoName; refuse non-empty"]
  D2 --> Broker["POST /api/v1/github/import-url<br/>Bearer bootToken"]
  Broker --> Cloud["Termsprawl Cloud"]
  Cloud --> Mint["Mint short-lived bearer clone URL"]
  Mint --> V3["Validate https + github.com"]
  V3 --> G2["git clone --depth 1 url dest"]
  G2 --> R2["Return name, path, fullName"]

  Boot["Boot suggestion path"] --> Suggest["listSuggestedRepos"]
  Suggest --> List["GET /api/v1/github/repos<br/>Bearer bootToken"]
  List --> Filter["Filter against projectNames"]

Loading

Key nodes: both clone paths converge on the same allowlist and shallow-clone command, but only the server path brokers through /api/v1/github/import-url. The suggestion path is read-only and best-effort. Validation and destination guards always run before git is spawned.

Key State & Data Shapes

Shape	Meaning
CloneRequest	{ url, dest }; url carries the GitHub credential, dest is the absolute clone destination.
CloneResult	{ path, ok: true }; no URL or token.
ImportRequest	{ fullName, destRoot }; fullName is validated owner/repo.
ImportResult	{ name, path, fullName }; name is the last path segment minus .git.
SuggestedRepo	{ fullName, name, private, updatedAt }; safe metadata for UI suggestions.
CloudRepo	Internal snake_case cloud shape: full_name, clone_url, private, updated_at.
GitHubCloneDeps	Injectable spawnFn; default is execFile.
GitHubImportDeps	cloudApi, bootToken, injectable fetchFn, injectable spawnFn.

Boundary Conditions

• URL allowlist: only https: and hostname github.com are accepted. http, other hosts, scp-style URLs, file://, and non-URL strings are refused before git runs.
• Credential redaction: invalid-URL errors expose only a parsed origin or empty string. Git failures replace the full URL with <clone-url> in stderr.
• Destination guard: a non-empty existing directory is refused. An existing empty directory is allowed.
• Full-name validation: FULL_NAME_RE is ^[\w.-]+\/[\w.-]+$ with exactly one slash; . and .. owner/repo segments are rejected.
• Cloud errors: import maps any 404 to github_not_connected; other non-OK responses use body.error only when it is a string.
• Suggestion degradation: listSuggestedRepos never throws; every failure mode returns [].
• Filtering: suggestions are filtered case-insensitively by project name, last segment of project paths/URLs, and clone URL suffix.
• Duplication: isAllowedCloneUrl, safeOrigin-style origin handling, and defaultSpawn are duplicated between the desktop clone and server import modules; policy changes must be mirrored or extracted.

E2E Coverage

scripts/github-import-e2e.sh boots a fake cloud with /api/v1/github/repos and /api/v1/github/import-url, then starts a real ts-space container and drives the server's WebSocket bridge. The script's stated assertions are:

1. Boot github:suggest lists repos not yet on the canvas.
2. github:import clones the repo for real, with git objects on the volume.
3. The imported project exists in the workspace with the clone as cwd.
4. The credential never appears in container logs, the response, or /data.
5. Re-import of the same repo is refused because the project cwd is already known.

The fake cloud returns repo listings as { repos: REPOS } and mints { url, expiresIn: 90 } from the import-url broker. The visible import-url response is an https://github.com/... URL, matching the strict allowlist that importGitHubRepo enforces. The drive script sends github:import, then requests workspace:snapshot after the import response.

Extension Points

• Inject spawnFn in either core service to test without git or substitute a different process runner.
• Inject fetchFn in github-import.ts to test cloud behavior or replace the transport.
• Configure cloudApi and bootToken through GitHubImportDeps for Server Edition boot.
• Update IMPORT_URL_PATH and REPOS_PATH for broker endpoint changes.
• Extend CloudRepo normalization when the cloud response shape changes; the array/{ repos } handling is centralized in extractReposArray.
• Adjust isAllowedCloneUrl only with awareness that the clone URL is a credential-bearing secret and currently restricted to https://github.com.
• Keep auth header construction aligned: importGitHubRepo builds the Bearer header inline while listSuggestedRepos uses authHeaders().

Sources: src/core/github-clone.ts, src/core/github-import.ts, src/core/github-clone.test.ts, src/core/github-import.test.ts, scripts/github-import-e2e.sh.