diff --git a/README.md b/README.md index d7366bc3b..c8a6093a3 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,7 @@ The **Ambient Code Platform** is an AI automation platform that combines Claude - **Intelligent Agentic Sessions**: AI-powered automation for analysis, research, content creation, and development tasks - **Multi-Agent Workflows**: Specialized AI agents model realistic software team dynamics +- **Git Provider Support**: Native integration with GitHub and GitLab (SaaS and self-hosted) - **Kubernetes Native**: Built with Custom Resources, Operators, and proper RBAC for enterprise deployment - **Real-time Monitoring**: Live status updates and job execution tracking @@ -35,6 +36,42 @@ The platform consists of containerized microservices orchestrated via Kubernetes 5. **Result Storage**: Analysis results stored back in Custom Resource status 6. **UI Updates**: Frontend displays real-time progress and completed results +## Git Provider Support + +### Supported Providers + +**GitHub**: +- ✅ GitHub.com (public and private repositories) +- ✅ GitHub Enterprise Server +- ✅ GitHub App authentication +- ✅ Personal Access Token authentication + +**GitLab** (v1.1.0+): +- ✅ GitLab.com (SaaS) +- ✅ Self-hosted GitLab (Community & Enterprise editions) +- ✅ Personal Access Token authentication +- ✅ HTTPS and SSH URL formats +- ✅ Custom domains and ports + +### Key Features + +- **Automatic Provider Detection**: Repositories automatically identified as GitHub or GitLab from URL +- **Multi-Provider Projects**: Use GitHub and GitLab repositories in the same project +- **Secure Token Storage**: All credentials encrypted in Kubernetes Secrets +- **Provider-Specific Error Handling**: Clear, actionable error messages for each platform + +### Getting Started with GitLab + +1. **Create Personal Access Token**: [GitLab PAT Setup Guide](docs/gitlab-token-setup.md) +2. **Connect Account**: Settings → Integrations → GitLab +3. **Configure Repository**: Add GitLab repository URL to project settings +4. **Create Sessions**: AgenticSessions work seamlessly with GitLab repos + +**Documentation**: +- [GitLab Integration Guide](docs/gitlab-integration.md) - Complete user guide +- [GitLab Token Setup](docs/gitlab-token-setup.md) - Step-by-step PAT creation +- [Self-Hosted GitLab](docs/gitlab-self-hosted.md) - Enterprise configuration + ## Prerequisites ### Required Tools @@ -489,11 +526,18 @@ See [e2e/README.md](e2e/README.md) for detailed documentation, troubleshooting, ## Support & Documentation +### Deployment & Configuration - **Deployment Guide**: [docs/OPENSHIFT_DEPLOY.md](docs/OPENSHIFT_DEPLOY.md) - **OAuth Setup**: [docs/OPENSHIFT_OAUTH.md](docs/OPENSHIFT_OAUTH.md) - **Architecture Details**: [diagrams/](diagrams/) - **API Documentation**: Available in web interface after deployment +### GitLab Integration +- **GitLab Integration Guide**: [docs/gitlab-integration.md](docs/gitlab-integration.md) +- **GitLab Token Setup**: [docs/gitlab-token-setup.md](docs/gitlab-token-setup.md) +- **Self-Hosted GitLab**: [docs/gitlab-self-hosted.md](docs/gitlab-self-hosted.md) +- **GitLab Testing**: [docs/gitlab-testing-procedures.md](docs/gitlab-testing-procedures.md) + ## Legacy vTeam References While the project is now branded as **Ambient Code Platform**, the name "vTeam" still appears in various technical components for backward compatibility and to avoid breaking changes. You will encounter "vTeam" or "vteam" in: diff --git a/components/backend/git/operations.go b/components/backend/git/operations.go index 8f3ada8dc..69022267f 100644 --- a/components/backend/git/operations.go +++ b/components/backend/git/operations.go @@ -22,6 +22,9 @@ import ( "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/client-go/dynamic" "k8s.io/client-go/kubernetes" + + "ambient-code-backend/gitlab" + "ambient-code-backend/types" ) // Package-level dependencies (set from main package) @@ -29,6 +32,7 @@ var ( GetProjectSettingsResource func() schema.GroupVersionResource GetGitHubInstallation func(context.Context, string) (interface{}, error) GitHubTokenManager interface{} // *GitHubTokenManager from main package + GetBackendNamespace func() string ) // ProjectSettings represents the project configuration @@ -109,6 +113,55 @@ func GetGitHubToken(ctx context.Context, k8sClient *kubernetes.Clientset, dynCli return string(token), nil } +// GetGitLabToken retrieves a GitLab Personal Access Token for a user +func GetGitLabToken(ctx context.Context, k8sClient *kubernetes.Clientset, project, userID string) (string, error) { + if k8sClient == nil { + log.Printf("Cannot read GitLab token: k8s client is nil") + return "", fmt.Errorf("no GitLab credentials available. Please connect your GitLab account") + } + + // GitLab tokens are stored in the project namespace (multi-tenant isolation) + // This matches the GitHub PAT pattern using ambient-non-vertex-integrations + secret, err := k8sClient.CoreV1().Secrets(project).Get(ctx, "gitlab-user-tokens", v1.GetOptions{}) + if err != nil { + log.Printf("Failed to get gitlab-user-tokens secret in %s: %v", project, err) + return "", fmt.Errorf("no GitLab credentials available. Please connect your GitLab account in this project") + } + + if secret.Data == nil { + log.Printf("Secret gitlab-user-tokens exists but Data is nil") + return "", fmt.Errorf("no GitLab credentials available. Please connect your GitLab account") + } + + token, ok := secret.Data[userID] + if !ok { + log.Printf("Secret gitlab-user-tokens has no token for user %s", userID) + return "", fmt.Errorf("no GitLab credentials available. Please connect your GitLab account") + } + + if len(token) == 0 { + log.Printf("Secret gitlab-user-tokens has token for user %s but value is empty", userID) + return "", fmt.Errorf("no GitLab credentials available. Please connect your GitLab account") + } + + log.Printf("Using GitLab token for user %s from gitlab-user-tokens secret", userID) + return string(token), nil +} + +// GetGitToken retrieves a Git token based on the repository provider +func GetGitToken(ctx context.Context, k8sClient *kubernetes.Clientset, dynClient dynamic.Interface, repoURL, project, userID string) (string, error) { + provider := types.DetectProvider(repoURL) + + switch provider { + case types.ProviderGitHub: + return GetGitHubToken(ctx, k8sClient, dynClient, project, userID) + case types.ProviderGitLab: + return GetGitLabToken(ctx, k8sClient, project, userID) + default: + return "", fmt.Errorf("unsupported repository provider for URL: %s", repoURL) + } +} + // getSecretKeys returns a list of keys from a secret's Data map for debugging func getSecretKeys(data map[string][]byte) []string { keys := make([]string, 0, len(data)) @@ -119,38 +172,77 @@ func getSecretKeys(data map[string][]byte) []string { } // CheckRepoSeeding checks if a repo has been seeded by verifying .claude/commands/ and .specify/ exist -func CheckRepoSeeding(ctx context.Context, repoURL string, branch *string, githubToken string) (bool, map[string]interface{}, error) { - owner, repo, err := ParseGitHubURL(repoURL) - if err != nil { - return false, nil, err - } - +// Supports both GitHub and GitLab repositories +func CheckRepoSeeding(ctx context.Context, repoURL string, branch *string, token string) (bool, map[string]interface{}, error) { branchName := "main" if branch != nil && strings.TrimSpace(*branch) != "" { branchName = strings.TrimSpace(*branch) } - claudeExists, err := checkGitHubPathExists(ctx, owner, repo, branchName, ".claude", githubToken) - if err != nil { - return false, nil, fmt.Errorf("failed to check .claude: %w", err) - } + provider := types.DetectProvider(repoURL) - // Check for .claude/commands directory (spec-kit slash commands) - claudeCommandsExists, err := checkGitHubPathExists(ctx, owner, repo, branchName, ".claude/commands", githubToken) - if err != nil { - return false, nil, fmt.Errorf("failed to check .claude/commands: %w", err) - } + var claudeExists, claudeCommandsExists, claudeAgentsExists, specifyExists bool + var err error - // Check for .claude/agents directory - claudeAgentsExists, err := checkGitHubPathExists(ctx, owner, repo, branchName, ".claude/agents", githubToken) - if err != nil { - return false, nil, fmt.Errorf("failed to check .claude/agents: %w", err) - } + switch provider { + case types.ProviderGitHub: + var owner, repo string + owner, repo, err = ParseGitHubURL(repoURL) + if err != nil { + return false, nil, err + } - // Check for .specify directory (from spec-kit) - specifyExists, err := checkGitHubPathExists(ctx, owner, repo, branchName, ".specify", githubToken) - if err != nil { - return false, nil, fmt.Errorf("failed to check .specify: %w", err) + claudeExists, err = checkGitHubPathExists(ctx, owner, repo, branchName, ".claude", token) + if err != nil { + return false, nil, fmt.Errorf("failed to check .claude: %w", err) + } + + claudeCommandsExists, err = checkGitHubPathExists(ctx, owner, repo, branchName, ".claude/commands", token) + if err != nil { + return false, nil, fmt.Errorf("failed to check .claude/commands: %w", err) + } + + claudeAgentsExists, err = checkGitHubPathExists(ctx, owner, repo, branchName, ".claude/agents", token) + if err != nil { + return false, nil, fmt.Errorf("failed to check .claude/agents: %w", err) + } + + specifyExists, err = checkGitHubPathExists(ctx, owner, repo, branchName, ".specify", token) + if err != nil { + return false, nil, fmt.Errorf("failed to check .specify: %w", err) + } + + case types.ProviderGitLab: + var parsed *types.ParsedGitLabRepo + parsed, err = gitlab.ParseGitLabURL(repoURL) + if err != nil { + return false, nil, fmt.Errorf("invalid GitLab URL: %w", err) + } + + client := gitlab.NewClient(parsed.APIURL, token) + + claudeExists, err = checkGitLabPathExists(ctx, client, parsed.ProjectID, branchName, ".claude") + if err != nil { + return false, nil, fmt.Errorf("failed to check .claude: %w", err) + } + + claudeCommandsExists, err = checkGitLabPathExists(ctx, client, parsed.ProjectID, branchName, ".claude/commands") + if err != nil { + return false, nil, fmt.Errorf("failed to check .claude/commands: %w", err) + } + + claudeAgentsExists, err = checkGitLabPathExists(ctx, client, parsed.ProjectID, branchName, ".claude/agents") + if err != nil { + return false, nil, fmt.Errorf("failed to check .claude/agents: %w", err) + } + + specifyExists, err = checkGitLabPathExists(ctx, client, parsed.ProjectID, branchName, ".specify") + if err != nil { + return false, nil, fmt.Errorf("failed to check .specify: %w", err) + } + + default: + return false, nil, fmt.Errorf("unsupported repository provider for URL: %s", repoURL) } details := map[string]interface{}{ @@ -165,6 +257,24 @@ func CheckRepoSeeding(ctx context.Context, repoURL string, branch *string, githu return isSeeded, details, nil } +// checkGitLabPathExists checks if a path exists in a GitLab repository +func checkGitLabPathExists(ctx context.Context, client *gitlab.Client, projectID, branch, path string) (bool, error) { + // Try to get the tree for this path + entries, err := client.GetAllTreeEntries(ctx, projectID, branch, path) + if err != nil { + // Check if it's a 404 error (path doesn't exist) + if gitlabErr, ok := err.(*types.GitLabAPIError); ok { + if gitlabErr.StatusCode == 404 { + return false, nil + } + } + return false, err + } + + // Path exists if we got entries + return len(entries) > 0 || entries != nil, nil +} + // ParseGitHubURL extracts owner and repo from a GitHub URL func ParseGitHubURL(gitURL string) (owner, repo string, err error) { gitURL = strings.TrimSuffix(gitURL, ".git") @@ -256,7 +366,7 @@ type Workflow interface { // PerformRepoSeeding performs the actual seeding operations // wf parameter should implement the Workflow interface // Returns: branchExisted (bool), error -func PerformRepoSeeding(ctx context.Context, wf Workflow, branchName, githubToken, agentURL, agentBranch, agentPath, specKitRepo, specKitVersion, specKitTemplate string) (bool, error) { +func PerformRepoSeeding(ctx context.Context, wf Workflow, branchName, token, agentURL, agentBranch, agentPath, specKitRepo, specKitVersion, specKitTemplate string) (bool, error) { umbrellaRepo := wf.GetUmbrellaRepo() if umbrellaRepo == nil { return false, fmt.Errorf("workflow has no spec repo") @@ -266,21 +376,46 @@ func PerformRepoSeeding(ctx context.Context, wf Workflow, branchName, githubToke return false, fmt.Errorf("branchName is required") } + // Validate push access to spec repo before starting + log.Printf("Validating push access to spec repo: %s", umbrellaRepo.GetURL()) + if err := validatePushAccess(ctx, umbrellaRepo.GetURL(), token); err != nil { + return false, fmt.Errorf("spec repo access validation failed: %w", err) + } + + // Validate push access to all supporting repos before starting + supportingRepos := wf.GetSupportingRepos() + if len(supportingRepos) > 0 { + log.Printf("Validating push access to %d supporting repos", len(supportingRepos)) + for i, repo := range supportingRepos { + if err := validatePushAccess(ctx, repo.GetURL(), token); err != nil { + return false, fmt.Errorf("supporting repo #%d (%s) access validation failed: %w", i+1, repo.GetURL(), err) + } + } + } + umbrellaDir, err := os.MkdirTemp("", "umbrella-*") if err != nil { return false, fmt.Errorf("failed to create temp dir for spec repo: %w", err) } - defer os.RemoveAll(umbrellaDir) + defer func() { + if err := os.RemoveAll(umbrellaDir); err != nil { + log.Printf("Warning: failed to cleanup temp directory %s: %v", umbrellaDir, err) + } + }() agentSrcDir, err := os.MkdirTemp("", "agents-*") if err != nil { return false, fmt.Errorf("failed to create temp dir for agent source: %w", err) } - defer os.RemoveAll(agentSrcDir) + defer func() { + if err := os.RemoveAll(agentSrcDir); err != nil { + log.Printf("Warning: failed to cleanup temp directory %s: %v", agentSrcDir, err) + } + }() // Clone umbrella repo with authentication log.Printf("Cloning umbrella repo: %s", umbrellaRepo.GetURL()) - authenticatedURL, err := InjectGitHubToken(umbrellaRepo.GetURL(), githubToken) + authenticatedURL, err := InjectGitToken(umbrellaRepo.GetURL(), token) if err != nil { return false, fmt.Errorf("failed to prepare spec repo URL: %w", err) } @@ -318,9 +453,30 @@ func PerformRepoSeeding(ctx context.Context, wf Workflow, branchName, githubToke } // Check if feature branch already exists remotely - cmd = exec.CommandContext(ctx, "git", "-C", umbrellaDir, "ls-remote", "--heads", "origin", branchName) + // Use authenticated URL directly to avoid issues with shallow clone remote setup + cmd = exec.CommandContext(ctx, "git", "ls-remote", "--heads", authenticatedURL, fmt.Sprintf("refs/heads/%s", branchName)) lsRemoteOut, lsRemoteErr := cmd.CombinedOutput() - branchExistsRemotely := lsRemoteErr == nil && strings.TrimSpace(string(lsRemoteOut)) != "" + log.Printf("DEBUG: ls-remote for branch '%s': error=%v, output='%s'", branchName, lsRemoteErr, string(lsRemoteOut)) + + // Check if branch exists by looking for actual git ref (ignoring warnings) + // Valid output format: "\trefs/heads/" + branchExistsRemotely := false + if lsRemoteErr == nil { + lines := strings.Split(string(lsRemoteOut), "\n") + for _, line := range lines { + trimmed := strings.TrimSpace(line) + // Skip empty lines and warning messages + if trimmed == "" || strings.HasPrefix(trimmed, "warning:") { + continue + } + // Check if line contains the branch ref + if strings.Contains(trimmed, fmt.Sprintf("refs/heads/%s", branchName)) { + branchExistsRemotely = true + break + } + } + } + log.Printf("DEBUG: branchExistsRemotely=%v", branchExistsRemotely) if branchExistsRemotely { // Branch exists - check it out instead of creating new @@ -579,11 +735,10 @@ func PerformRepoSeeding(ctx context.Context, wf Workflow, branchName, githubToke // Create feature branch in all supporting repos // Push access will be validated by the actual git operations - if they fail, we'll get a clear error - supportingRepos := wf.GetSupportingRepos() if len(supportingRepos) > 0 { log.Printf("Creating feature branch %s in %d supporting repos", branchName, len(supportingRepos)) for i, repo := range supportingRepos { - if err := createBranchInRepo(ctx, repo, branchName, githubToken); err != nil { + if err := createBranchInRepo(ctx, repo, branchName, token); err != nil { return false, fmt.Errorf("failed to create branch in supporting repo #%d (%s): %w", i+1, repo.GetURL(), err) } } @@ -592,11 +747,24 @@ func PerformRepoSeeding(ctx context.Context, wf Workflow, branchName, githubToke return branchExistsRemotely, nil } +// sanitizeURLForError removes credentials from a URL for safe error logging +func sanitizeURLForError(rawURL string) string { + u, err := url.Parse(rawURL) + if err != nil { + // If URL can't be parsed, just return a generic message + return "[invalid URL format]" + } + // Remove any embedded credentials + u.User = nil + return u.String() +} + // InjectGitHubToken injects a GitHub token into a git URL for authentication func InjectGitHubToken(gitURL, token string) (string, error) { u, err := url.Parse(gitURL) if err != nil { - return "", fmt.Errorf("invalid git URL: %w", err) + // Sanitize URL before including in error message + return "", fmt.Errorf("invalid git URL (%s): %w", sanitizeURLForError(gitURL), err) } if u.Scheme != "https" { @@ -607,6 +775,179 @@ func InjectGitHubToken(gitURL, token string) (string, error) { return u.String(), nil } +// InjectGitLabToken injects a GitLab token into a git URL for authentication +func InjectGitLabToken(gitURL, token string) (string, error) { + u, err := url.Parse(gitURL) + if err != nil { + // Sanitize URL before including in error message + return "", fmt.Errorf("invalid git URL (%s): %w", sanitizeURLForError(gitURL), err) + } + + if u.Scheme != "https" { + return gitURL, nil + } + + // GitLab uses oauth2:token@ format + u.User = url.UserPassword("oauth2", token) + return u.String(), nil +} + +// InjectGitToken injects a Git token into a URL based on the repository provider +func InjectGitToken(gitURL, token string) (string, error) { + provider := types.DetectProvider(gitURL) + + switch provider { + case types.ProviderGitHub: + return InjectGitHubToken(gitURL, token) + case types.ProviderGitLab: + return InjectGitLabToken(gitURL, token) + default: + return "", fmt.Errorf("unsupported repository provider for URL: %s", gitURL) + } +} + +// DetectPushError analyzes git push error output and provides user-friendly error messages +func DetectPushError(repoURL, stderr, stdout string) error { + provider := types.DetectProvider(repoURL) + + // Common error patterns + stderrLower := strings.ToLower(stderr) + stdoutLower := strings.ToLower(stdout) + combined := stderrLower + " " + stdoutLower + + // Check for authentication/permission errors + if strings.Contains(combined, "403") || strings.Contains(combined, "forbidden") { + if provider == types.ProviderGitLab { + return fmt.Errorf("GitLab push failed: Insufficient permissions. Ensure your GitLab token has 'write_repository' scope. You can update your token by reconnecting your GitLab account with the required permissions") + } else if provider == types.ProviderGitHub { + return fmt.Errorf("GitHub push failed: Insufficient permissions. Check that your GitHub App installation has write access to this repository") + } + return fmt.Errorf("Push failed: Insufficient permissions (403 Forbidden)") + } + + // Check for authentication failures + if strings.Contains(combined, "401") || strings.Contains(combined, "unauthorized") || strings.Contains(combined, "authentication failed") { + if provider == types.ProviderGitLab { + return fmt.Errorf("GitLab push failed: Authentication failed. Your GitLab token may be invalid or expired. Please reconnect your GitLab account") + } else if provider == types.ProviderGitHub { + return fmt.Errorf("GitHub push failed: Authentication failed. Check your GitHub App installation") + } + return fmt.Errorf("Push failed: Authentication failed (401 Unauthorized)") + } + + // Check for network errors + if strings.Contains(combined, "could not resolve host") || strings.Contains(combined, "connection refused") { + return fmt.Errorf("Push failed: Unable to connect to %s. Check network connectivity", extractHostFromURL(repoURL)) + } + + // Check for rate limiting + if strings.Contains(combined, "429") || strings.Contains(combined, "rate limit") { + if provider == types.ProviderGitLab { + return fmt.Errorf("GitLab push failed: Rate limit exceeded. Please wait a few minutes before retrying") + } + return fmt.Errorf("Push failed: API rate limit exceeded. Please wait before retrying") + } + + // Check for repository not found + if strings.Contains(combined, "404") || strings.Contains(combined, "not found") || strings.Contains(combined, "repository not found") { + return fmt.Errorf("Push failed: Repository not found. Verify the repository URL: %s", repoURL) + } + + // Return original error if no pattern matched + errMsg := strings.TrimSpace(stderr) + if errMsg == "" { + errMsg = strings.TrimSpace(stdout) + } + if len(errMsg) > 500 { + errMsg = errMsg[:500] + "..." + } + return fmt.Errorf("Push failed: %s", errMsg) +} + +// extractHostFromURL extracts the host from a git URL for error messages +func extractHostFromURL(gitURL string) string { + if strings.HasPrefix(gitURL, "git@") { + // SSH format: git@host:owner/repo + parts := strings.Split(gitURL, "@") + if len(parts) > 1 { + hostParts := strings.Split(parts[1], ":") + if len(hostParts) > 0 { + return hostParts[0] + } + } + } else { + // HTTPS format + u, err := url.Parse(gitURL) + if err == nil && u.Host != "" { + return u.Host + } + } + return "repository host" +} + +// ConstructBranchURL constructs a web URL to view a branch based on the provider +func ConstructBranchURL(repoURL, branch string) (string, error) { + provider := types.DetectProvider(repoURL) + + switch provider { + case types.ProviderGitHub: + return ConstructGitHubBranchURL(repoURL, branch) + case types.ProviderGitLab: + return ConstructGitLabBranchURL(repoURL, branch) + default: + return "", fmt.Errorf("unsupported provider for URL: %s", repoURL) + } +} + +// ConstructGitHubBranchURL constructs a GitHub web URL for a branch +func ConstructGitHubBranchURL(repoURL, branch string) (string, error) { + owner, repo, err := ParseGitHubURL(repoURL) + if err != nil { + return "", err + } + + // Clean repo name (remove .git if present) + repo = strings.TrimSuffix(repo, ".git") + + return fmt.Sprintf("https://github.com/%s/%s/tree/%s", owner, repo, branch), nil +} + +// ConstructGitLabBranchURL constructs a GitLab web URL for a branch +func ConstructGitLabBranchURL(repoURL, branch string) (string, error) { + parsed, err := gitlab.ParseGitLabURL(repoURL) + if err != nil { + return "", err + } + + // GitLab branch URL format: https://gitlab.com/owner/repo/-/tree/branch + return fmt.Sprintf("https://%s/%s/%s/-/tree/%s", parsed.Host, parsed.Owner, parsed.Repo, branch), nil +} + +// GetRepositoryWebURL returns the main web URL for a repository +func GetRepositoryWebURL(repoURL string) (string, error) { + provider := types.DetectProvider(repoURL) + + switch provider { + case types.ProviderGitHub: + owner, repo, err := ParseGitHubURL(repoURL) + if err != nil { + return "", err + } + repo = strings.TrimSuffix(repo, ".git") + return fmt.Sprintf("https://github.com/%s/%s", owner, repo), nil + + case types.ProviderGitLab: + parsed, err := gitlab.ParseGitLabURL(repoURL) + if err != nil { + return "", err + } + return fmt.Sprintf("https://%s/%s/%s", parsed.Host, parsed.Owner, parsed.Repo), nil + + default: + return "", fmt.Errorf("unsupported provider for URL: %s", repoURL) + } +} + // DeriveRepoFolderFromURL extracts the repo folder from a Git URL func DeriveRepoFolderFromURL(u string) string { s := strings.TrimSpace(u) @@ -675,21 +1016,26 @@ func PushRepo(ctx context.Context, repoDir, commitMessage, outputRepoURL, branch defer resp.Body.Close() switch resp.StatusCode { case 200: - var ghUser struct { - Login string `json:"login"` - Name string `json:"name"` - Email string `json:"email"` - } - if json.Unmarshal([]byte(fmt.Sprintf("%v", resp.Body)), &ghUser) == nil { - if gitUserName == "" && ghUser.Name != "" { - gitUserName = ghUser.Name - } else if gitUserName == "" && ghUser.Login != "" { - gitUserName = ghUser.Login + body, err := io.ReadAll(resp.Body) + if err == nil { + var ghUser struct { + Login string `json:"login"` + Name string `json:"name"` + Email string `json:"email"` } - if gitUserEmail == "" && ghUser.Email != "" { - gitUserEmail = ghUser.Email + if err := json.Unmarshal(body, &ghUser); err == nil { + if gitUserName == "" && ghUser.Name != "" { + gitUserName = ghUser.Name + } else if gitUserName == "" && ghUser.Login != "" { + gitUserName = ghUser.Login + } + if gitUserEmail == "" && ghUser.Email != "" { + gitUserEmail = ghUser.Email + } + log.Printf("gitPushRepo: fetched GitHub user name=%q email=%q", gitUserName, gitUserEmail) + } else { + log.Printf("Failed to parse GitHub user info: %v", err) } - log.Printf("gitPushRepo: fetched GitHub user name=%q email=%q", gitUserName, gitUserEmail) } case 403: log.Printf("gitPushRepo: GitHub API /user returned 403 (token lacks 'read:user' scope, using fallback identity)") @@ -764,7 +1110,8 @@ func PushRepo(ctx context.Context, repoDir, commitMessage, outputRepoURL, branch sout = sout[:2000] + "..." } log.Printf("gitPushRepo: push failed url=%q ref=%q err=%v stderr.snip=%q stdout.snip=%q", outputRepoURL, ref, err, serr, sout) - return "", fmt.Errorf("push failed: %s", errOut) + // Use enhanced error detection for user-friendly messages + return "", DetectPushError(outputRepoURL, errOut, out) } if len(out) > 2000 { @@ -939,10 +1286,231 @@ func CheckBranchExists(ctx context.Context, repoURL, branchName, githubToken str return false, fmt.Errorf("GitHub API error: %s (body: %s)", resp.Status, string(body)) } +// validatePushAccess checks if the user has push access to a repository (supports both GitHub and GitLab) +func validatePushAccess(ctx context.Context, repoURL, token string) error { + provider := types.DetectProvider(repoURL) + + switch provider { + case types.ProviderGitHub: + return validateGitHubPushAccess(ctx, repoURL, token) + case types.ProviderGitLab: + return validateGitLabPushAccess(ctx, repoURL, token) + default: + return fmt.Errorf("unsupported repository provider for URL: %s", repoURL) + } +} + +// validateGitHubPushAccess checks if the user has push access to a GitHub repository +func validateGitHubPushAccess(ctx context.Context, repoURL, githubToken string) error { + owner, repo, err := ParseGitHubURL(repoURL) + if err != nil { + return fmt.Errorf("invalid GitHub repository URL: %w", err) + } + + // Use GitHub API to check repository permissions + log.Printf("Validating push access to GitHub repo %s with token (len=%d)", repoURL, len(githubToken)) + apiURL := fmt.Sprintf("https://api.github.com/repos/%s/%s", owner, repo) + + req, err := http.NewRequestWithContext(ctx, "GET", apiURL, nil) + if err != nil { + return fmt.Errorf("failed to create request: %w", err) + } + + req.Header.Set("Authorization", "Bearer "+githubToken) + req.Header.Set("Accept", "application/vnd.github.v3+json") + + resp, err := http.DefaultClient.Do(req) + if err != nil { + return fmt.Errorf("failed to check repository access: %w", err) + } + defer resp.Body.Close() + + // Read response body once + body, err := io.ReadAll(resp.Body) + if err != nil { + return fmt.Errorf("failed to read response body: %w", err) + } + + if resp.StatusCode == http.StatusNotFound { + return fmt.Errorf("repository %s/%s not found or you don't have access to it", owner, repo) + } + + if resp.StatusCode == http.StatusTooManyRequests { + resetTime := resp.Header.Get("X-RateLimit-Reset") + if resetTime != "" { + return fmt.Errorf("GitHub API rate limit exceeded. Rate limit will reset at %s. Please try again later", resetTime) + } + return fmt.Errorf("GitHub API rate limit exceeded. Please try again later") + } + + if resp.StatusCode != http.StatusOK { + return fmt.Errorf("GitHub API error: %s (body: %s)", resp.Status, string(body)) + } + + // Parse response to check permissions + var repoInfo struct { + Permissions struct { + Push bool `json:"push"` + } `json:"permissions"` + } + + if err := json.Unmarshal(body, &repoInfo); err != nil { + return fmt.Errorf("failed to parse repository info: %w (body: %s)", err, string(body)) + } + + if !repoInfo.Permissions.Push { + return fmt.Errorf("you don't have push access to %s. Please fork the repository or use a repository you have write access to", repoURL) + } + + log.Printf("Validated push access to GitHub repo %s", repoURL) + return nil +} + +// validateGitLabPushAccess checks if the user has push access to a GitLab repository +func validateGitLabPushAccess(ctx context.Context, repoURL, gitlabToken string) error { + parsed, err := gitlab.ParseGitLabURL(repoURL) + if err != nil { + return fmt.Errorf("invalid GitLab repository URL: %w", err) + } + + // Use GitLab API to check repository permissions + log.Printf("Validating push access to GitLab repo %s with token (len=%d)", repoURL, len(gitlabToken)) + + // Get project details to check permissions + // Note: parsed.ProjectID is already URL-encoded, don't double-encode it + apiURL := fmt.Sprintf("%s/projects/%s", parsed.APIURL, parsed.ProjectID) + + req, err := http.NewRequestWithContext(ctx, "GET", apiURL, nil) + if err != nil { + return fmt.Errorf("failed to create request: %w", err) + } + + req.Header.Set("Authorization", "Bearer "+gitlabToken) + req.Header.Set("Accept", "application/json") + + resp, err := http.DefaultClient.Do(req) + if err != nil { + return fmt.Errorf("failed to check repository access: %w", err) + } + defer resp.Body.Close() + + // Read response body once + body, err := io.ReadAll(resp.Body) + if err != nil { + return fmt.Errorf("failed to read response body: %w", err) + } + + if resp.StatusCode == http.StatusNotFound { + return fmt.Errorf("repository %s/%s not found or you don't have access to it. Verify the repository URL and your GitLab token permissions", parsed.Owner, parsed.Repo) + } + + if resp.StatusCode == http.StatusUnauthorized || resp.StatusCode == http.StatusForbidden { + return fmt.Errorf("authentication failed for GitLab repository. Ensure your GitLab token has 'api' and 'write_repository' scopes") + } + + if resp.StatusCode == http.StatusTooManyRequests { + return fmt.Errorf("GitLab API rate limit exceeded. Please wait a few minutes before retrying") + } + + if resp.StatusCode != http.StatusOK { + return fmt.Errorf("GitLab API error: %s (body: %s)", resp.Status, string(body)) + } + + // Parse response to check permissions and ownership + var projectInfo struct { + Visibility string `json:"visibility"` + Namespace struct { + Kind string `json:"kind"` + Path string `json:"path"` + } `json:"namespace"` + Permissions struct { + ProjectAccess *struct { + AccessLevel int `json:"access_level"` + } `json:"project_access"` + GroupAccess *struct { + AccessLevel int `json:"access_level"` + } `json:"group_access"` + } `json:"permissions"` + } + + if err := json.Unmarshal(body, &projectInfo); err != nil { + return fmt.Errorf("failed to parse project info: %w (body: %s)", err, string(body)) + } + + // For public repositories, GitLab may return null permissions + // In this case, verify access by checking if we can get the authenticated user's info + // and if the namespace matches + if projectInfo.Permissions.ProjectAccess == nil && projectInfo.Permissions.GroupAccess == nil { + log.Printf("GitLab repo %s has null permissions (likely public repo), verifying access via user info", repoURL) + + // Get authenticated user info to verify token and check namespace ownership + userReq, err := http.NewRequestWithContext(ctx, "GET", parsed.APIURL+"/user", nil) + if err != nil { + return fmt.Errorf("failed to create user info request: %w", err) + } + userReq.Header.Set("Authorization", "Bearer "+gitlabToken) + userReq.Header.Set("Accept", "application/json") + + userResp, err := http.DefaultClient.Do(userReq) + if err != nil { + return fmt.Errorf("failed to get user info: %w", err) + } + defer userResp.Body.Close() + + if userResp.StatusCode != http.StatusOK { + return fmt.Errorf("unable to verify repository access. Token may not have sufficient permissions") + } + + userBody, err := io.ReadAll(userResp.Body) + if err != nil { + return fmt.Errorf("failed to read user info: %w", err) + } + + var userInfo struct { + Username string `json:"username"` + } + if err := json.Unmarshal(userBody, &userInfo); err != nil { + return fmt.Errorf("failed to parse user info: %w", err) + } + + // For user namespaces, check if the authenticated user owns the namespace + if projectInfo.Namespace.Kind == "user" && projectInfo.Namespace.Path == userInfo.Username { + log.Printf("Validated push access to GitLab repo %s (owner: %s)", repoURL, userInfo.Username) + return nil + } + + // For public repos not owned by the user, we cannot guarantee push access + // but if the token is valid and scoped correctly, assume access based on visibility + if projectInfo.Visibility == "public" { + log.Printf("Warning: GitLab repo %s is public but permissions are null. Assuming push access based on valid token", repoURL) + return nil + } + + return fmt.Errorf("unable to verify push access to %s. Repository may require explicit permissions", repoURL) + } + + // GitLab access levels: 10=Guest, 20=Reporter, 30=Developer, 40=Maintainer, 50=Owner + // Need at least Developer (30) to push + hasAccess := false + if projectInfo.Permissions.ProjectAccess != nil && projectInfo.Permissions.ProjectAccess.AccessLevel >= 30 { + hasAccess = true + } + if projectInfo.Permissions.GroupAccess != nil && projectInfo.Permissions.GroupAccess.AccessLevel >= 30 { + hasAccess = true + } + + if !hasAccess { + return fmt.Errorf("you don't have push access to %s. You need at least Developer (30) access level. Please check your permissions in GitLab", repoURL) + } + + log.Printf("Validated push access to GitLab repo %s", repoURL) + return nil +} + // createBranchInRepo creates a feature branch in a supporting repository // Follows the same pattern as umbrella repo seeding but without adding files // Note: This function assumes push access has already been validated by the caller -func createBranchInRepo(ctx context.Context, repo GitRepo, branchName, githubToken string) error { +func createBranchInRepo(ctx context.Context, repo GitRepo, branchName, token string) error { repoURL := repo.GetURL() if repoURL == "" { return fmt.Errorf("repository URL is empty") @@ -952,9 +1520,13 @@ func createBranchInRepo(ctx context.Context, repo GitRepo, branchName, githubTok if err != nil { return fmt.Errorf("failed to create temp dir: %w", err) } - defer os.RemoveAll(repoDir) + defer func() { + if err := os.RemoveAll(repoDir); err != nil { + log.Printf("Warning: failed to cleanup temp directory %s: %v", repoDir, err) + } + }() - authenticatedURL, err := InjectGitHubToken(repoURL, githubToken) + authenticatedURL, err := InjectGitToken(repoURL, token) if err != nil { return fmt.Errorf("failed to prepare repo URL: %w", err) } diff --git a/components/backend/gitlab/client.go b/components/backend/gitlab/client.go new file mode 100644 index 000000000..c3a69711b --- /dev/null +++ b/components/backend/gitlab/client.go @@ -0,0 +1,434 @@ +package gitlab + +import ( + "context" + "encoding/json" + "fmt" + "io" + "log" + "net/http" + "net/url" + "os" + "strconv" + "time" + + "ambient-code-backend/types" + "github.com/google/uuid" +) + +const ( + // DefaultMaxPaginationPages is the default limit for pagination loops + DefaultMaxPaginationPages = 100 +) + +// Client represents a GitLab API client +type Client struct { + httpClient *http.Client + baseURL string + token string +} + +// NewClient creates a new GitLab API client with 15-second timeout +func NewClient(baseURL, token string) *Client { + return &Client{ + httpClient: &http.Client{ + Timeout: 15 * time.Second, + }, + baseURL: baseURL, + token: token, + } +} + +// doRequest performs an HTTP request with GitLab authentication +// Includes standardized logging and request ID tracking for debugging +func (c *Client) doRequest(ctx context.Context, method, path string, body io.Reader) (*http.Response, error) { + url := c.baseURL + path + + // Generate unique request ID for tracking + requestID := uuid.New().String() + + // Log request start (with redacted URL) + startTime := time.Now() + LogInfo("[ReqID: %s] GitLab API request: %s %s", requestID, method, RedactURL(url)) + + req, err := http.NewRequestWithContext(ctx, method, url, body) + if err != nil { + LogError("[ReqID: %s] Failed to create request: %v", requestID, err) + return nil, fmt.Errorf("failed to create request: %w", err) + } + + // Add GitLab authentication header + req.Header.Set("Authorization", "Bearer "+c.token) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("X-Request-ID", requestID) // Include request ID in headers for GitLab correlation + + resp, err := c.httpClient.Do(req) + duration := time.Since(startTime) + + if err != nil { + LogError("[ReqID: %s] GitLab API request failed after %v: %v", requestID, duration, err) + return nil, fmt.Errorf("request failed: %w", err) + } + + // Log response with status and timing + LogInfo("[ReqID: %s] GitLab API response: %d %s (took %v)", + requestID, resp.StatusCode, http.StatusText(resp.StatusCode), duration) + + // Log warning for non-2xx responses + if resp.StatusCode < 200 || resp.StatusCode >= 300 { + LogWarning("[ReqID: %s] GitLab API returned non-success status: %d", requestID, resp.StatusCode) + } + + return resp, nil +} + +// ParseErrorResponse parses a GitLab API error response and returns a structured error +func ParseErrorResponse(resp *http.Response) *types.GitLabAPIError { + defer resp.Body.Close() + + // Extract request ID from response headers if present + requestID := resp.Header.Get("X-Request-ID") + if requestID == "" { + requestID = resp.Request.Header.Get("X-Request-ID") // Fallback to request header + } + + body, err := io.ReadAll(resp.Body) + if err != nil { + LogError("[ReqID: %s] Failed to read GitLab error response: %v", requestID, err) + return &types.GitLabAPIError{ + StatusCode: resp.StatusCode, + Message: "Failed to read error response from GitLab API", + Remediation: "Please try again or contact support if the issue persists", + RawError: err.Error(), + RequestID: requestID, + } + } + + // Try to parse GitLab error format + var gitlabError struct { + Message string `json:"message"` + Error string `json:"error"` + } + + if err := json.Unmarshal(body, &gitlabError); err == nil { + apiErr := MapGitLabAPIError(resp.StatusCode, gitlabError.Message, gitlabError.Error, string(body)) + apiErr.RequestID = requestID + LogError("[ReqID: %s] GitLab API error: %s (status: %d)", requestID, apiErr.Message, resp.StatusCode) + return apiErr + } + + // Fallback to generic error with raw body + apiErr := MapGitLabAPIError(resp.StatusCode, "", "", string(body)) + apiErr.RequestID = requestID + LogError("[ReqID: %s] GitLab API error (status: %d): %s", requestID, resp.StatusCode, string(body)) + return apiErr +} + +// MapGitLabAPIError maps HTTP status codes to user-friendly error messages +func MapGitLabAPIError(statusCode int, message, errorType, rawBody string) *types.GitLabAPIError { + apiError := &types.GitLabAPIError{ + StatusCode: statusCode, + RawError: rawBody, + } + + switch statusCode { + case 401: + apiError.Message = "GitLab token is invalid or expired" + apiError.Remediation = "Please reconnect your GitLab account with a valid Personal Access Token" + + case 403: + apiError.Message = "GitLab token lacks required permissions" + if message != "" { + apiError.Message = fmt.Sprintf("GitLab error: %s", message) + } + apiError.Remediation = "Ensure your token has 'api', 'read_repository', and 'write_repository' scopes and try again" + + case 404: + apiError.Message = "GitLab repository not found" + apiError.Remediation = "Verify the repository URL and your access permissions" + + case 429: + apiError.Message = "GitLab API rate limit exceeded" + apiError.Remediation = "Please wait a few minutes before retrying. GitLab.com allows 300 requests per minute" + + case 500, 502, 503, 504: + apiError.Message = "GitLab API is experiencing issues" + apiError.Remediation = "Please try again in a few minutes or contact support if the issue persists" + + default: + if message != "" { + apiError.Message = fmt.Sprintf("GitLab API error: %s", message) + } else { + apiError.Message = fmt.Sprintf("GitLab API returned status code %d", statusCode) + } + apiError.Remediation = "Please check your request and try again" + } + + return apiError +} + +// CheckResponse checks an HTTP response for errors and returns a GitLabAPIError if found +func CheckResponse(resp *http.Response) error { + if resp.StatusCode >= 200 && resp.StatusCode < 300 { + return nil + } + + return ParseErrorResponse(resp) +} + +// PaginationInfo contains pagination metadata from GitLab API responses +type PaginationInfo struct { + TotalPages int + NextPage int + PrevPage int + PerPage int + Total int + CurrentPage int +} + +// extractPaginationInfo extracts pagination info from response headers +func extractPaginationInfo(resp *http.Response) *PaginationInfo { + info := &PaginationInfo{} + + // GitLab uses X-Total-Pages, X-Next-Page, X-Per-Page headers + if totalPages := resp.Header.Get("X-Total-Pages"); totalPages != "" { + fmt.Sscanf(totalPages, "%d", &info.TotalPages) + } + if nextPage := resp.Header.Get("X-Next-Page"); nextPage != "" { + fmt.Sscanf(nextPage, "%d", &info.NextPage) + } + if prevPage := resp.Header.Get("X-Prev-Page"); prevPage != "" { + fmt.Sscanf(prevPage, "%d", &info.PrevPage) + } + if perPage := resp.Header.Get("X-Per-Page"); perPage != "" { + fmt.Sscanf(perPage, "%d", &info.PerPage) + } + if total := resp.Header.Get("X-Total"); total != "" { + fmt.Sscanf(total, "%d", &info.Total) + } + if page := resp.Header.Get("X-Page"); page != "" { + fmt.Sscanf(page, "%d", &info.CurrentPage) + } + + return info +} + +// GetBranches retrieves all branches for a GitLab repository with pagination support +func (c *Client) GetBranches(ctx context.Context, projectID string, page, perPage int) ([]types.GitLabBranch, *PaginationInfo, error) { + if perPage == 0 { + perPage = 100 // Max page size for GitLab API + } + + path := fmt.Sprintf("/projects/%s/repository/branches?page=%d&per_page=%d", projectID, page, perPage) + + resp, err := c.doRequest(ctx, "GET", path, nil) + if err != nil { + return nil, nil, err + } + defer resp.Body.Close() + + if err := CheckResponse(resp); err != nil { + return nil, nil, err + } + + body, err := io.ReadAll(resp.Body) + if err != nil { + return nil, nil, fmt.Errorf("failed to read branches response: %w", err) + } + + var branches []types.GitLabBranch + if err := json.Unmarshal(body, &branches); err != nil { + return nil, nil, fmt.Errorf("failed to parse branches response: %w", err) + } + + pagination := extractPaginationInfo(resp) + + return branches, pagination, nil +} + +// getMaxPaginationPages returns the configured maximum pagination pages +// Can be overridden via GITLAB_MAX_PAGINATION_PAGES environment variable +func getMaxPaginationPages() int { + if envVal := os.Getenv("GITLAB_MAX_PAGINATION_PAGES"); envVal != "" { + if val, err := strconv.Atoi(envVal); err == nil && val > 0 { + return val + } + log.Printf("Warning: Invalid GITLAB_MAX_PAGINATION_PAGES value '%s', using default %d", envVal, DefaultMaxPaginationPages) + } + return DefaultMaxPaginationPages +} + +// GetAllBranches retrieves all branches across all pages +// Pagination limit can be configured via GITLAB_MAX_PAGINATION_PAGES environment variable +func (c *Client) GetAllBranches(ctx context.Context, projectID string) ([]types.GitLabBranch, error) { + var allBranches []types.GitLabBranch + page := 1 + perPage := 100 + maxPages := getMaxPaginationPages() + + for { + branches, pagination, err := c.GetBranches(ctx, projectID, page, perPage) + if err != nil { + return nil, err + } + + allBranches = append(allBranches, branches...) + + // Check if there are more pages + if pagination.NextPage == 0 || len(branches) == 0 { + break + } + + page = pagination.NextPage + + // Safety limit to prevent infinite loops (configurable) + if page > maxPages { + log.Printf("Warning: Repository %s has more than %d pages of branches, truncating results", projectID, maxPages) + return allBranches, fmt.Errorf("exceeded pagination limit (%d pages). Increase GITLAB_MAX_PAGINATION_PAGES if needed", maxPages) + } + + // Log warning when approaching limit + if page > maxPages-10 { + log.Printf("Warning: Pagination for repository %s is approaching limit (page %d/%d)", projectID, page, maxPages) + } + } + + return allBranches, nil +} + +// GetTree retrieves the directory tree for a GitLab repository +func (c *Client) GetTree(ctx context.Context, projectID, ref, path string, page, perPage int) ([]types.GitLabTreeEntry, *PaginationInfo, error) { + if perPage == 0 { + perPage = 100 + } + + // Build the API path + apiPath := fmt.Sprintf("/projects/%s/repository/tree?ref=%s&page=%d&per_page=%d", + projectID, ref, page, perPage) + + if path != "" && path != "/" { + apiPath += "&path=" + path + } + + resp, err := c.doRequest(ctx, "GET", apiPath, nil) + if err != nil { + return nil, nil, err + } + defer resp.Body.Close() + + if err := CheckResponse(resp); err != nil { + return nil, nil, err + } + + body, err := io.ReadAll(resp.Body) + if err != nil { + return nil, nil, fmt.Errorf("failed to read tree response: %w", err) + } + + var entries []types.GitLabTreeEntry + if err := json.Unmarshal(body, &entries); err != nil { + return nil, nil, fmt.Errorf("failed to parse tree response: %w", err) + } + + pagination := extractPaginationInfo(resp) + + return entries, pagination, nil +} + +// GetAllTreeEntries retrieves all tree entries across all pages +func (c *Client) GetAllTreeEntries(ctx context.Context, projectID, ref, path string) ([]types.GitLabTreeEntry, error) { + var allEntries []types.GitLabTreeEntry + page := 1 + perPage := 100 + + for { + entries, pagination, err := c.GetTree(ctx, projectID, ref, path, page, perPage) + if err != nil { + return nil, err + } + + allEntries = append(allEntries, entries...) + + if pagination.NextPage == 0 || len(entries) == 0 { + break + } + + page = pagination.NextPage + + // Safety limit + if page > 100 { + return nil, fmt.Errorf("exceeded pagination limit (100 pages)") + } + } + + return allEntries, nil +} + +// GitLabFileContent represents the response from GitLab file content API +type GitLabFileContent struct { + FileName string `json:"file_name"` + FilePath string `json:"file_path"` + Size int `json:"size"` + Encoding string `json:"encoding"` + Content string `json:"content"` + ContentSHA string `json:"content_sha256"` + Ref string `json:"ref"` + BlobID string `json:"blob_id"` + CommitID string `json:"commit_id"` + LastCommitID string `json:"last_commit_id"` +} + +// GetFileContents retrieves the contents of a file from a GitLab repository +func (c *Client) GetFileContents(ctx context.Context, projectID, filePath, ref string) (*GitLabFileContent, error) { + // URL encode the file path using url.PathEscape for safe encoding + encodedPath := url.PathEscape(filePath) + + path := fmt.Sprintf("/projects/%s/repository/files/%s?ref=%s", projectID, encodedPath, ref) + + resp, err := c.doRequest(ctx, "GET", path, nil) + if err != nil { + return nil, err + } + defer resp.Body.Close() + + if err := CheckResponse(resp); err != nil { + return nil, err + } + + body, err := io.ReadAll(resp.Body) + if err != nil { + return nil, fmt.Errorf("failed to read file content response: %w", err) + } + + var fileContent GitLabFileContent + if err := json.Unmarshal(body, &fileContent); err != nil { + return nil, fmt.Errorf("failed to parse file content response: %w", err) + } + + return &fileContent, nil +} + +// GetRawFileContents retrieves the raw contents of a file (without base64 encoding) +func (c *Client) GetRawFileContents(ctx context.Context, projectID, filePath, ref string) ([]byte, error) { + // URL encode the file path using url.PathEscape for safe encoding + encodedPath := url.PathEscape(filePath) + + path := fmt.Sprintf("/projects/%s/repository/files/%s/raw?ref=%s", projectID, encodedPath, ref) + + resp, err := c.doRequest(ctx, "GET", path, nil) + if err != nil { + return nil, err + } + defer resp.Body.Close() + + if err := CheckResponse(resp); err != nil { + return nil, err + } + + body, err := io.ReadAll(resp.Body) + if err != nil { + return nil, fmt.Errorf("failed to read raw file content: %w", err) + } + + return body, nil +} diff --git a/components/backend/gitlab/connection.go b/components/backend/gitlab/connection.go new file mode 100644 index 000000000..1d6e924b5 --- /dev/null +++ b/components/backend/gitlab/connection.go @@ -0,0 +1,190 @@ +package gitlab + +import ( + "context" + "fmt" + "strconv" + "time" + + "k8s.io/client-go/kubernetes" + + "ambient-code-backend/k8s" + "ambient-code-backend/types" +) + +// ConnectionManager handles GitLab connection operations +type ConnectionManager struct { + clientset *kubernetes.Clientset + namespace string +} + +// NewConnectionManager creates a new connection manager +func NewConnectionManager(clientset *kubernetes.Clientset, namespace string) *ConnectionManager { + return &ConnectionManager{ + clientset: clientset, + namespace: namespace, + } +} + +// StoreGitLabConnection stores a GitLab connection (metadata in ConfigMap, token in Secret) +func (cm *ConnectionManager) StoreGitLabConnection(ctx context.Context, userID, token, instanceURL string) (*types.GitLabConnection, error) { + // Validate token and get user information + result, err := ValidateGitLabToken(ctx, token, instanceURL) + if err != nil { + return nil, fmt.Errorf("failed to validate token: %w", err) + } + + if !result.Valid { + return nil, fmt.Errorf("invalid token: %s", result.ErrorMessage) + } + + // Create connection metadata + connection := &types.GitLabConnection{ + UserID: userID, + GitLabUserID: strconv.Itoa(result.User.ID), + InstanceURL: instanceURL, + Username: result.User.Username, + UpdatedAt: time.Now(), + } + + // Store token in Kubernetes Secret + if err := k8s.StoreGitLabToken(ctx, cm.clientset, cm.namespace, userID, token); err != nil { + return nil, fmt.Errorf("failed to store token: %w", err) + } + + // Store connection metadata in ConfigMap + if err := k8s.StoreGitLabConnection(ctx, cm.clientset, cm.namespace, connection); err != nil { + // Cleanup: try to delete the token we just stored + _ = k8s.DeleteGitLabToken(ctx, cm.clientset, cm.namespace, userID) + return nil, fmt.Errorf("failed to store connection: %w", err) + } + + LogInfo("GitLab connection stored for user %s (GitLab user: %s)", userID, result.User.Username) + + return connection, nil +} + +// GetGitLabConnection retrieves a GitLab connection for a user +func (cm *ConnectionManager) GetGitLabConnection(ctx context.Context, userID string) (*types.GitLabConnection, error) { + connection, err := k8s.GetGitLabConnection(ctx, cm.clientset, cm.namespace, userID) + if err != nil { + return nil, err + } + + return connection, nil +} + +// GetGitLabConnectionWithToken retrieves both connection metadata and token +func (cm *ConnectionManager) GetGitLabConnectionWithToken(ctx context.Context, userID string) (*types.GitLabConnection, string, error) { + // Get connection metadata + connection, err := k8s.GetGitLabConnection(ctx, cm.clientset, cm.namespace, userID) + if err != nil { + return nil, "", err + } + + // Get token + token, err := k8s.GetGitLabToken(ctx, cm.clientset, cm.namespace, userID) + if err != nil { + return nil, "", fmt.Errorf("connection exists but token not found: %w", err) + } + + return connection, token, nil +} + +// UpdateGitLabConnection updates an existing GitLab connection +func (cm *ConnectionManager) UpdateGitLabConnection(ctx context.Context, userID, token, instanceURL string) (*types.GitLabConnection, error) { + // This is essentially the same as storing a new connection + // It will overwrite the existing one + return cm.StoreGitLabConnection(ctx, userID, token, instanceURL) +} + +// DeleteGitLabConnection removes a GitLab connection (both metadata and token) +func (cm *ConnectionManager) DeleteGitLabConnection(ctx context.Context, userID string) error { + // Delete token from Secret + if err := k8s.DeleteGitLabToken(ctx, cm.clientset, cm.namespace, userID); err != nil { + LogWarning("Failed to delete token for user %s: %v", userID, err) + // Continue with ConfigMap deletion even if Secret deletion fails + } + + // Delete connection metadata from ConfigMap + if err := k8s.DeleteGitLabConnection(ctx, cm.clientset, cm.namespace, userID); err != nil { + return fmt.Errorf("failed to delete connection: %w", err) + } + + LogInfo("GitLab connection deleted for user %s", userID) + + return nil +} + +// HasGitLabConnection checks if a user has a GitLab connection +func (cm *ConnectionManager) HasGitLabConnection(ctx context.Context, userID string) (bool, error) { + return k8s.HasGitLabConnection(ctx, cm.clientset, cm.namespace, userID) +} + +// GetConnectionStatus retrieves the connection status for a user +func (cm *ConnectionManager) GetConnectionStatus(ctx context.Context, userID string) (*ConnectionStatus, error) { + // Check if connection exists + hasConnection, err := cm.HasGitLabConnection(ctx, userID) + if err != nil { + return nil, err + } + + if !hasConnection { + return &ConnectionStatus{ + Connected: false, + }, nil + } + + // Get connection details + connection, err := cm.GetGitLabConnection(ctx, userID) + if err != nil { + return nil, err + } + + // Check if token exists + hasToken, err := k8s.HasGitLabToken(ctx, cm.clientset, cm.namespace, userID) + if err != nil { + return nil, err + } + + return &ConnectionStatus{ + Connected: true, + Username: connection.Username, + InstanceURL: connection.InstanceURL, + GitLabUserID: connection.GitLabUserID, + UpdatedAt: connection.UpdatedAt, + HasToken: hasToken, + }, nil +} + +// ConnectionStatus represents the status of a GitLab connection +type ConnectionStatus struct { + Connected bool `json:"connected"` + Username string `json:"username,omitempty"` + InstanceURL string `json:"instanceUrl,omitempty"` + GitLabUserID string `json:"gitlabUserId,omitempty"` + UpdatedAt time.Time `json:"updatedAt,omitempty"` + HasToken bool `json:"hasToken"` +} + +// ValidateExistingConnection validates that an existing connection still works +func (cm *ConnectionManager) ValidateExistingConnection(ctx context.Context, userID string) (bool, error) { + // Get connection and token + connection, token, err := cm.GetGitLabConnectionWithToken(ctx, userID) + if err != nil { + return false, err + } + + // Validate the token is still valid + result, err := ValidateGitLabToken(ctx, token, connection.InstanceURL) + if err != nil { + return false, err + } + + return result.Valid, nil +} + +// ListConnections returns all GitLab connections in the namespace +func (cm *ConnectionManager) ListConnections(ctx context.Context) ([]*types.GitLabConnection, error) { + return k8s.ListGitLabConnections(ctx, cm.clientset, cm.namespace) +} diff --git a/components/backend/gitlab/doc.go b/components/backend/gitlab/doc.go new file mode 100644 index 000000000..30924ed8e --- /dev/null +++ b/components/backend/gitlab/doc.go @@ -0,0 +1,7 @@ +// Package gitlab provides GitLab API integration for vTeam. +// This package implements GitLab repository operations including: +// - URL parsing and normalization +// - Token validation and management +// - API client for GitLab v4 endpoints +// - Connection management for GitLab.com and self-hosted instances +package gitlab diff --git a/components/backend/gitlab/logger.go b/components/backend/gitlab/logger.go new file mode 100644 index 000000000..228e9bd59 --- /dev/null +++ b/components/backend/gitlab/logger.go @@ -0,0 +1,86 @@ +package gitlab + +import ( + "fmt" + "log" + "net/url" + "regexp" +) + +// TokenRedactionPlaceholder is used to replace sensitive tokens in logs +const TokenRedactionPlaceholder = "[REDACTED]" + +// RedactToken removes sensitive token information from a string +func RedactToken(s string) string { + // GitLab PAT format: glpat-xxxxxxxxxxxxx + gitlabPATPattern := regexp.MustCompile(`glpat-[a-zA-Z0-9_-]+`) + s = gitlabPATPattern.ReplaceAllString(s, TokenRedactionPlaceholder) + + // GitLab CI token format: gitlab-ci-token + gitlabCIPattern := regexp.MustCompile(`gitlab-ci-token:\s*[a-zA-Z0-9_-]+`) + s = gitlabCIPattern.ReplaceAllString(s, "gitlab-ci-token: "+TokenRedactionPlaceholder) + + // Bearer tokens in Authorization headers + bearerPattern := regexp.MustCompile(`Bearer\s+[a-zA-Z0-9_-]+`) + s = bearerPattern.ReplaceAllString(s, "Bearer "+TokenRedactionPlaceholder) + + // OAuth2 tokens in URLs: oauth2:TOKEN@ + oauthURLPattern := regexp.MustCompile(`oauth2:[^@]+@`) + s = oauthURLPattern.ReplaceAllString(s, "oauth2:"+TokenRedactionPlaceholder+"@") + + // Generic token pattern in URLs + tokenURLPattern := regexp.MustCompile(`://[^:]+:[^@]+@`) + s = tokenURLPattern.ReplaceAllString(s, "://"+TokenRedactionPlaceholder+":"+TokenRedactionPlaceholder+"@") + + return s +} + +// LogInfo logs an informational message with token redaction +func LogInfo(format string, args ...interface{}) { + message := fmt.Sprintf(format, args...) + redacted := RedactToken(message) + log.Printf("[GitLab] INFO: %s", redacted) +} + +// LogWarning logs a warning message with token redaction +func LogWarning(format string, args ...interface{}) { + message := fmt.Sprintf(format, args...) + redacted := RedactToken(message) + log.Printf("[GitLab] WARNING: %s", redacted) +} + +// LogError logs an error message with token redaction +func LogError(format string, args ...interface{}) { + message := fmt.Sprintf(format, args...) + redacted := RedactToken(message) + log.Printf("[GitLab] ERROR: %s", redacted) +} + +// RedactURL removes sensitive information from a Git URL +// Handles both GitLab (oauth2:token@) and GitHub (x-access-token:token@) formats +func RedactURL(gitURL string) string { + // Parse the URL properly instead of string splitting + parsedURL, err := url.Parse(gitURL) + if err != nil { + // If parsing fails, fall back to regex-based redaction + return RedactToken(gitURL) + } + + // Check if URL contains user info (credentials) + if parsedURL.User != nil { + // Redact the entire userinfo part (handles oauth2:token, x-access-token:token, etc.) + parsedURL.User = url.User(TokenRedactionPlaceholder) + } + + return parsedURL.String() +} + +// SanitizeErrorMessage removes sensitive information from error messages +func SanitizeErrorMessage(err error) string { + if err == nil { + return "" + } + + message := err.Error() + return RedactToken(message) +} diff --git a/components/backend/gitlab/mappers.go b/components/backend/gitlab/mappers.go new file mode 100644 index 000000000..3563a0778 --- /dev/null +++ b/components/backend/gitlab/mappers.go @@ -0,0 +1,61 @@ +package gitlab + +import ( + "ambient-code-backend/types" +) + +// MapGitLabBranchToCommon converts a GitLabBranch to a common Branch type +func MapGitLabBranchToCommon(gitlabBranch types.GitLabBranch) types.Branch { + return types.Branch{ + Name: gitlabBranch.Name, + Protected: gitlabBranch.Protected, + Default: gitlabBranch.Default, + Commit: types.CommitInfo{ + SHA: gitlabBranch.Commit.ID, + Message: gitlabBranch.Commit.Title, + Author: gitlabBranch.Commit.AuthorName, + Timestamp: gitlabBranch.Commit.CommittedDate.Format("2006-01-02T15:04:05Z07:00"), + }, + } +} + +// MapGitLabBranchesToCommon converts multiple GitLab branches to common format +func MapGitLabBranchesToCommon(gitlabBranches []types.GitLabBranch) []types.Branch { + branches := make([]types.Branch, len(gitlabBranches)) + for i, gb := range gitlabBranches { + branches[i] = MapGitLabBranchToCommon(gb) + } + return branches +} + +// MapGitLabTreeEntryToCommon converts a GitLabTreeEntry to a common TreeEntry type +func MapGitLabTreeEntryToCommon(gitlabEntry types.GitLabTreeEntry) types.TreeEntry { + return types.TreeEntry{ + Name: gitlabEntry.Name, + Path: gitlabEntry.Path, + Type: gitlabEntry.Type, + Mode: gitlabEntry.Mode, + SHA: gitlabEntry.ID, + } +} + +// MapGitLabTreeEntriesToCommon converts multiple GitLab tree entries to common format +func MapGitLabTreeEntriesToCommon(gitlabEntries []types.GitLabTreeEntry) []types.TreeEntry { + entries := make([]types.TreeEntry, len(gitlabEntries)) + for i, ge := range gitlabEntries { + entries[i] = MapGitLabTreeEntryToCommon(ge) + } + return entries +} + +// MapGitLabFileContentToCommon converts GitLab file content to common format +func MapGitLabFileContentToCommon(gitlabFile *GitLabFileContent) types.FileContent { + return types.FileContent{ + Name: gitlabFile.FileName, + Path: gitlabFile.FilePath, + Content: gitlabFile.Content, + Encoding: gitlabFile.Encoding, + Size: gitlabFile.Size, + SHA: gitlabFile.BlobID, + } +} diff --git a/components/backend/gitlab/parser.go b/components/backend/gitlab/parser.go new file mode 100644 index 000000000..edb7122b0 --- /dev/null +++ b/components/backend/gitlab/parser.go @@ -0,0 +1,140 @@ +package gitlab + +import ( + "fmt" + "net/url" + "regexp" + "strings" + + "ambient-code-backend/types" +) + +// ParseGitLabURL parses a GitLab repository URL and returns structured information +func ParseGitLabURL(repoURL string) (*types.ParsedGitLabRepo, error) { + if repoURL == "" { + return nil, fmt.Errorf("repository URL cannot be empty") + } + + // Normalize the URL first + normalized, err := NormalizeGitLabURL(repoURL) + if err != nil { + return nil, err + } + + // Parse the normalized URL + parsed, err := url.Parse(normalized) + if err != nil { + return nil, fmt.Errorf("invalid URL format: %w", err) + } + + // Extract host + host := parsed.Host + if host == "" { + return nil, fmt.Errorf("unable to extract host from URL: %s", repoURL) + } + + // Extract owner and repo from path + // Path format: /owner/repo or /owner/repo.git + path := strings.TrimPrefix(parsed.Path, "/") + path = strings.TrimSuffix(path, ".git") + + parts := strings.Split(path, "/") + if len(parts) < 2 { + return nil, fmt.Errorf("invalid GitLab URL format, expected /owner/repo: %s", repoURL) + } + + owner := parts[0] + repo := parts[1] + + if owner == "" || repo == "" { + return nil, fmt.Errorf("owner and repository name are required") + } + + // Detect if self-hosted or GitLab.com + apiURL := ConstructAPIURL(host) + + // Create project ID (URL-encoded path for GitLab API) + projectID := url.PathEscape(fmt.Sprintf("%s/%s", owner, repo)) + + return &types.ParsedGitLabRepo{ + Host: host, + Owner: owner, + Repo: repo, + APIURL: apiURL, + ProjectID: projectID, + }, nil +} + +// NormalizeGitLabURL converts various GitLab URL formats to a canonical HTTPS format +func NormalizeGitLabURL(repoURL string) (string, error) { + // Trim whitespace + repoURL = strings.TrimSpace(repoURL) + + // Handle SSH format: git@gitlab.com:owner/repo.git + sshPattern := regexp.MustCompile(`^git@([^:]+):(.+)$`) + if matches := sshPattern.FindStringSubmatch(repoURL); matches != nil { + host := matches[1] + path := matches[2] + path = strings.TrimSuffix(path, ".git") + return fmt.Sprintf("https://%s/%s", host, path), nil + } + + // Handle HTTPS URLs + if strings.HasPrefix(repoURL, "https://") || strings.HasPrefix(repoURL, "http://") { + // Upgrade HTTP to HTTPS for security + if strings.HasPrefix(repoURL, "http://") { + repoURL = strings.Replace(repoURL, "http://", "https://", 1) + } + + // Remove .git suffix if present + repoURL = strings.TrimSuffix(repoURL, ".git") + + return repoURL, nil + } + + // If no protocol, assume https:// + if !strings.Contains(repoURL, "://") { + return fmt.Sprintf("https://%s", repoURL), nil + } + + return "", fmt.Errorf("unsupported URL format: %s", repoURL) +} + +// IsGitLabSelfHosted determines if a host is a self-hosted GitLab instance +func IsGitLabSelfHosted(host string) bool { + // GitLab.com is not self-hosted + if host == "gitlab.com" || strings.HasSuffix(host, ".gitlab.com") { + return false + } + + // Everything else containing "gitlab" is assumed to be self-hosted + // This includes domains like gitlab.company.com, gitlab.internal.example.com, etc. + return strings.Contains(strings.ToLower(host), "gitlab") +} + +// ConstructAPIURL builds the GitLab API base URL from a host +func ConstructAPIURL(host string) string { + // For all GitLab instances (both .com and self-hosted), API is at /api/v4 + // Handle ports if present + return fmt.Sprintf("https://%s/api/v4", host) +} + +// ValidateGitLabURL checks if a URL is a valid GitLab repository URL +func ValidateGitLabURL(repoURL string) error { + parsed, err := ParseGitLabURL(repoURL) + if err != nil { + return err + } + + // Basic validation + if parsed.Owner == "" || parsed.Repo == "" { + return fmt.Errorf("invalid repository URL: missing owner or repository name") + } + + // Ensure the host contains "gitlab" + if !strings.Contains(strings.ToLower(parsed.Host), "gitlab") { + return fmt.Errorf("URL does not appear to be a GitLab repository: %s", repoURL) + } + + return nil +} diff --git a/components/backend/gitlab/token.go b/components/backend/gitlab/token.go new file mode 100644 index 000000000..6642bc541 --- /dev/null +++ b/components/backend/gitlab/token.go @@ -0,0 +1,215 @@ +package gitlab + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/url" + "time" + + "ambient-code-backend/types" +) + +// GitLabUser represents a GitLab user from the /user API +type GitLabUser struct { + ID int `json:"id"` + Username string `json:"username"` + Name string `json:"name"` + Email string `json:"email"` +} + +// TokenValidationResult contains the result of token validation +type TokenValidationResult struct { + Valid bool + User *GitLabUser + InstanceURL string + ErrorMessage string + ErrorCode int +} + +// ValidateGitLabToken validates a GitLab Personal Access Token +func ValidateGitLabToken(ctx context.Context, token, instanceURL string) (*TokenValidationResult, error) { + if token == "" { + return nil, fmt.Errorf("token cannot be empty") + } + + if instanceURL == "" { + instanceURL = "https://gitlab.com" + } + + // Construct API URL + apiURL := ConstructAPIURL(ExtractHost(instanceURL)) + client := NewClient(apiURL, token) + + // Call /user API to validate token + user, err := GetCurrentUser(ctx, client) + if err != nil { + // Check if it's a GitLabAPIError + if gitlabErr, ok := err.(*types.GitLabAPIError); ok { + return &TokenValidationResult{ + Valid: false, + ErrorMessage: gitlabErr.Message, + ErrorCode: gitlabErr.StatusCode, + }, nil + } + + return nil, fmt.Errorf("failed to validate token: %w", err) + } + + return &TokenValidationResult{ + Valid: true, + User: user, + InstanceURL: instanceURL, + }, nil +} + +// GetCurrentUser retrieves the current authenticated user from GitLab API +func GetCurrentUser(ctx context.Context, client *Client) (*GitLabUser, error) { + resp, err := client.doRequest(ctx, "GET", "/user", nil) + if err != nil { + return nil, err + } + defer resp.Body.Close() + + if err := CheckResponse(resp); err != nil { + return nil, err + } + + body, err := io.ReadAll(resp.Body) + if err != nil { + return nil, fmt.Errorf("failed to read response body: %w", err) + } + + var user GitLabUser + if err := json.Unmarshal(body, &user); err != nil { + return nil, fmt.Errorf("failed to parse user response: %w", err) + } + + return &user, nil +} + +// ValidateRepositoryAccess checks if the token has access to a specific repository +func ValidateRepositoryAccess(ctx context.Context, client *Client, owner, repo string) error { + // Construct project path + projectPath := fmt.Sprintf("%s/%s", owner, repo) + projectID := EncodeProjectPath(projectPath) + + // Try to get project information + resp, err := client.doRequest(ctx, "GET", fmt.Sprintf("/projects/%s", projectID), nil) + if err != nil { + return fmt.Errorf("failed to access repository: %w", err) + } + defer resp.Body.Close() + + if err := CheckResponse(resp); err != nil { + // Customize error message for repository access + if gitlabErr, ok := err.(*types.GitLabAPIError); ok { + if gitlabErr.StatusCode == 404 { + gitlabErr.Message = fmt.Sprintf("Repository '%s/%s' not found or you don't have access", owner, repo) + gitlabErr.Remediation = "Verify the repository URL and ensure your token has access to this repository" + } + } + return err + } + + return nil +} + +// ValidateTokenAndRepository performs comprehensive validation of token and repository access +func ValidateTokenAndRepository(ctx context.Context, token, repoURL string) (*TokenValidationResult, error) { + // Parse repository URL + parsed, err := ParseGitLabURL(repoURL) + if err != nil { + return nil, fmt.Errorf("invalid repository URL: %w", err) + } + + // Construct instance URL from host + instanceURL := fmt.Sprintf("https://%s", parsed.Host) + + // Validate token + result, err := ValidateGitLabToken(ctx, token, instanceURL) + if err != nil { + return nil, err + } + + if !result.Valid { + return result, nil + } + + // Create client for repository access check + client := NewClient(parsed.APIURL, token) + + // Validate repository access + if err := ValidateRepositoryAccess(ctx, client, parsed.Owner, parsed.Repo); err != nil { + if gitlabErr, ok := err.(*types.GitLabAPIError); ok { + return &TokenValidationResult{ + Valid: false, + ErrorMessage: gitlabErr.Message, + ErrorCode: gitlabErr.StatusCode, + }, nil + } + return nil, err + } + + return result, nil +} + +// ExtractHost extracts the host from a full URL +func ExtractHost(urlStr string) string { + // Remove protocol + host := urlStr + if len(host) > 8 && host[:8] == "https://" { + host = host[8:] + } else if len(host) > 7 && host[:7] == "http://" { + host = host[7:] + } + + // Remove path + if idx := len(host); idx > 0 { + for i, ch := range host { + if ch == '/' { + idx = i + break + } + } + host = host[:idx] + } + + return host +} + +// EncodeProjectPath URL-encodes a GitLab project path for API calls +func EncodeProjectPath(projectPath string) string { + // GitLab API accepts URL-encoded project paths + // e.g., "namespace/project" becomes "namespace%2Fproject" + // Use url.PathEscape for safe, standards-compliant encoding + return url.PathEscape(projectPath) +} + +// TokenInfo contains metadata about a GitLab token +type TokenInfo struct { + UserID int + Username string + InstanceURL string + ValidatedAt time.Time +} + +// GetTokenInfo retrieves information about a validated token +func GetTokenInfo(ctx context.Context, token, instanceURL string) (*TokenInfo, error) { + result, err := ValidateGitLabToken(ctx, token, instanceURL) + if err != nil { + return nil, err + } + + if !result.Valid { + return nil, fmt.Errorf("token is invalid: %s", result.ErrorMessage) + } + + return &TokenInfo{ + UserID: result.User.ID, + Username: result.User.Username, + InstanceURL: instanceURL, + ValidatedAt: time.Now(), + }, nil +} diff --git a/components/backend/go.mod b/components/backend/go.mod index 69050d560..9267fb825 100644 --- a/components/backend/go.mod +++ b/components/backend/go.mod @@ -8,8 +8,10 @@ require ( github.com/gin-contrib/cors v1.7.6 github.com/gin-gonic/gin v1.10.1 github.com/golang-jwt/jwt/v5 v5.3.0 + github.com/google/uuid v1.6.0 github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 github.com/joho/godotenv v1.5.1 + github.com/stretchr/testify v1.11.1 k8s.io/api v0.34.0 k8s.io/apimachinery v0.34.0 k8s.io/client-go v0.34.0 @@ -34,7 +36,6 @@ require ( github.com/goccy/go-json v0.10.5 // indirect github.com/gogo/protobuf v1.3.2 // indirect github.com/google/gnostic-models v0.7.0 // indirect - github.com/google/uuid v1.6.0 // indirect github.com/josharian/intern v1.0.0 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/klauspost/cpuid/v2 v2.2.10 // indirect @@ -46,8 +47,8 @@ require ( github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/pelletier/go-toml/v2 v2.2.4 // indirect github.com/pkg/errors v0.9.1 // indirect + github.com/pmezard/go-difflib v1.0.0 // indirect github.com/spf13/pflag v1.0.6 // indirect - github.com/stretchr/testify v1.11.1 // indirect github.com/twitchyliquid64/golang-asm v0.15.1 // indirect github.com/ugorji/go/codec v1.3.0 // indirect github.com/x448/float16 v0.8.4 // indirect diff --git a/components/backend/handlers/gitlab_auth.go b/components/backend/handlers/gitlab_auth.go new file mode 100644 index 000000000..193d2ab1b --- /dev/null +++ b/components/backend/handlers/gitlab_auth.go @@ -0,0 +1,404 @@ +package handlers + +import ( + "fmt" + "net/http" + "net/url" + "strings" + + "github.com/gin-gonic/gin" + "k8s.io/client-go/kubernetes" + + "ambient-code-backend/gitlab" +) + +// GitLabAuthHandler handles GitLab authentication endpoints +type GitLabAuthHandler struct { + connectionManager *gitlab.ConnectionManager +} + +// NewGitLabAuthHandler creates a new GitLab authentication handler +func NewGitLabAuthHandler(clientset *kubernetes.Clientset, namespace string) *GitLabAuthHandler { + return &GitLabAuthHandler{ + connectionManager: gitlab.NewConnectionManager(clientset, namespace), + } +} + +// ConnectGitLabRequest represents a request to connect a GitLab account +type ConnectGitLabRequest struct { + PersonalAccessToken string `json:"personalAccessToken" binding:"required"` + InstanceURL string `json:"instanceUrl"` +} + +// ConnectGitLabResponse represents the response from connecting a GitLab account +type ConnectGitLabResponse struct { + UserID string `json:"userId"` + GitLabUserID string `json:"gitlabUserId"` + Username string `json:"username"` + InstanceURL string `json:"instanceUrl"` + Connected bool `json:"connected"` + Message string `json:"message"` +} + +// GitLabStatusResponse represents the GitLab connection status +type GitLabStatusResponse struct { + Connected bool `json:"connected"` + Username string `json:"username,omitempty"` + InstanceURL string `json:"instanceUrl,omitempty"` + GitLabUserID string `json:"gitlabUserId,omitempty"` +} + +// validateGitLabInput validates GitLab connection request input +func validateGitLabInput(instanceURL, token string) error { + // Validate instance URL + if instanceURL != "" { + parsedURL, err := url.Parse(instanceURL) + if err != nil { + return fmt.Errorf("invalid instance URL format") + } + + // Require HTTPS for security + if parsedURL.Scheme != "https" { + return fmt.Errorf("instance URL must use HTTPS") + } + + // Validate hostname is not empty + if parsedURL.Host == "" { + return fmt.Errorf("instance URL must have a valid hostname") + } + + // Prevent common injection attempts + if strings.Contains(parsedURL.Host, "@") { + return fmt.Errorf("instance URL hostname cannot contain '@'") + } + } + + // Validate token length (GitLab PATs are 20 chars, but allow for future changes) + // Min: 20 chars, Max: 255 chars (reasonable upper bound) + if len(token) < 20 { + return fmt.Errorf("token must be at least 20 characters") + } + if len(token) > 255 { + return fmt.Errorf("token must not exceed 255 characters") + } + + // Validate token contains only valid characters (alphanumeric and some special chars) + // GitLab tokens use: a-z, A-Z, 0-9, -, _ + for _, char := range token { + if !((char >= 'a' && char <= 'z') || + (char >= 'A' && char <= 'Z') || + (char >= '0' && char <= '9') || + char == '-' || char == '_') { + return fmt.Errorf("token contains invalid characters") + } + } + + return nil +} + +// ConnectGitLab handles POST /projects/:projectName/auth/gitlab/connect +func (h *GitLabAuthHandler) ConnectGitLab(c *gin.Context) { + // Get project from URL parameter + project := c.Param("projectName") + if project == "" { + c.JSON(http.StatusBadRequest, gin.H{ + "error": "Project name is required", + "statusCode": http.StatusBadRequest, + }) + return + } + + var req ConnectGitLabRequest + if err := c.ShouldBindJSON(&req); err != nil { + c.JSON(http.StatusBadRequest, gin.H{ + "error": "Invalid request body", + "statusCode": http.StatusBadRequest, + }) + return + } + + // Default to GitLab.com if no instance URL provided + if req.InstanceURL == "" { + req.InstanceURL = "https://gitlab.com" + } + + // Validate input + if err := validateGitLabInput(req.InstanceURL, req.PersonalAccessToken); err != nil { + c.JSON(http.StatusBadRequest, gin.H{ + "error": fmt.Sprintf("Invalid input: %v", err), + "statusCode": http.StatusBadRequest, + }) + return + } + + // Get user ID from context (set by authentication middleware) + userID, exists := c.Get("userID") + if !exists { + c.JSON(http.StatusUnauthorized, gin.H{ + "error": "User not authenticated", + "statusCode": http.StatusUnauthorized, + }) + return + } + + userIDStr, ok := userID.(string) + if !ok { + c.JSON(http.StatusInternalServerError, gin.H{ + "error": "Invalid user ID format", + "statusCode": http.StatusInternalServerError, + }) + return + } + + // RBAC: Verify user can create/update secrets in this project + reqK8s, _ := GetK8sClientsForRequest(c) + if reqK8s == nil { + c.JSON(http.StatusUnauthorized, gin.H{ + "error": "Invalid or missing token", + "statusCode": http.StatusUnauthorized, + }) + return + } + + ctx := c.Request.Context() + if err := ValidateSecretAccess(ctx, reqK8s, project, "create"); err != nil { + gitlab.LogError("RBAC check failed for user %s in project %s: %v", userIDStr, project, err) + c.JSON(http.StatusForbidden, gin.H{ + "error": "Insufficient permissions to manage GitLab credentials", + "statusCode": http.StatusForbidden, + }) + return + } + + // Store GitLab connection (now project-scoped) + connection, err := h.connectionManager.StoreGitLabConnection(ctx, userIDStr, req.PersonalAccessToken, req.InstanceURL) + if err != nil { + gitlab.LogError("Failed to store GitLab connection for user %s in project %s: %v", userIDStr, project, err) + c.JSON(http.StatusInternalServerError, gin.H{ + "error": err.Error(), + "statusCode": http.StatusInternalServerError, + }) + return + } + + c.JSON(http.StatusOK, ConnectGitLabResponse{ + UserID: connection.UserID, + GitLabUserID: connection.GitLabUserID, + Username: connection.Username, + InstanceURL: connection.InstanceURL, + Connected: true, + Message: "GitLab account connected successfully to project " + project, + }) +} + +// GetGitLabStatus handles GET /projects/:projectName/auth/gitlab/status +func (h *GitLabAuthHandler) GetGitLabStatus(c *gin.Context) { + // Get project from URL parameter + project := c.Param("projectName") + if project == "" { + c.JSON(http.StatusBadRequest, gin.H{ + "error": "Project name is required", + "statusCode": http.StatusBadRequest, + }) + return + } + + // Get user ID from context + userID, exists := c.Get("userID") + if !exists { + c.JSON(http.StatusUnauthorized, gin.H{ + "error": "User not authenticated", + "statusCode": http.StatusUnauthorized, + }) + return + } + + userIDStr, ok := userID.(string) + if !ok { + c.JSON(http.StatusInternalServerError, gin.H{ + "error": "Invalid user ID format", + "statusCode": http.StatusInternalServerError, + }) + return + } + + // RBAC: Verify user can read secrets in this project + reqK8s, _ := GetK8sClientsForRequest(c) + if reqK8s == nil { + c.JSON(http.StatusUnauthorized, gin.H{ + "error": "Invalid or missing token", + "statusCode": http.StatusUnauthorized, + }) + return + } + + ctx := c.Request.Context() + if err := ValidateSecretAccess(ctx, reqK8s, project, "get"); err != nil { + gitlab.LogError("RBAC check failed for user %s in project %s: %v", userIDStr, project, err) + c.JSON(http.StatusForbidden, gin.H{ + "error": "Insufficient permissions to read GitLab credentials", + "statusCode": http.StatusForbidden, + }) + return + } + + // Get connection status (project-scoped) + status, err := h.connectionManager.GetConnectionStatus(ctx, userIDStr) + if err != nil { + gitlab.LogError("Failed to get GitLab status for user %s in project %s: %v", userIDStr, project, err) + c.JSON(http.StatusInternalServerError, gin.H{ + "error": "Failed to retrieve GitLab connection status", + "statusCode": http.StatusInternalServerError, + }) + return + } + + if !status.Connected { + c.JSON(http.StatusOK, GitLabStatusResponse{ + Connected: false, + }) + return + } + + c.JSON(http.StatusOK, GitLabStatusResponse{ + Connected: true, + Username: status.Username, + InstanceURL: status.InstanceURL, + GitLabUserID: status.GitLabUserID, + }) +} + +// DisconnectGitLab handles POST /projects/:projectName/auth/gitlab/disconnect +func (h *GitLabAuthHandler) DisconnectGitLab(c *gin.Context) { + // Get project from URL parameter + project := c.Param("projectName") + if project == "" { + c.JSON(http.StatusBadRequest, gin.H{ + "error": "Project name is required", + "statusCode": http.StatusBadRequest, + }) + return + } + + // Get user ID from context + userID, exists := c.Get("userID") + if !exists { + c.JSON(http.StatusUnauthorized, gin.H{ + "error": "User not authenticated", + "statusCode": http.StatusUnauthorized, + }) + return + } + + userIDStr, ok := userID.(string) + if !ok { + c.JSON(http.StatusInternalServerError, gin.H{ + "error": "Invalid user ID format", + "statusCode": http.StatusInternalServerError, + }) + return + } + + // RBAC: Verify user can update secrets in this project + reqK8s, _ := GetK8sClientsForRequest(c) + if reqK8s == nil { + c.JSON(http.StatusUnauthorized, gin.H{ + "error": "Invalid or missing token", + "statusCode": http.StatusUnauthorized, + }) + return + } + + ctx := c.Request.Context() + if err := ValidateSecretAccess(ctx, reqK8s, project, "update"); err != nil { + gitlab.LogError("RBAC check failed for user %s in project %s: %v", userIDStr, project, err) + c.JSON(http.StatusForbidden, gin.H{ + "error": "Insufficient permissions to manage GitLab credentials", + "statusCode": http.StatusForbidden, + }) + return + } + + // Delete GitLab connection (project-scoped) + if err := h.connectionManager.DeleteGitLabConnection(ctx, userIDStr); err != nil { + gitlab.LogError("Failed to disconnect GitLab for user %s in project %s: %v", userIDStr, project, err) + c.JSON(http.StatusInternalServerError, gin.H{ + "error": "Failed to disconnect GitLab account", + "statusCode": http.StatusInternalServerError, + }) + return + } + + c.JSON(http.StatusOK, gin.H{ + "message": "GitLab account disconnected successfully from project " + project, + "connected": false, + }) +} + +// Global wrapper functions for routes (now project-scoped) + +// ConnectGitLabGlobal is the global handler for POST /projects/:projectName/auth/gitlab/connect +func ConnectGitLabGlobal(c *gin.Context) { + // Get project from URL parameter - this is the namespace where tokens will be stored + project := c.Param("projectName") + if project == "" { + c.JSON(http.StatusBadRequest, gin.H{"error": "Project name is required"}) + return + } + + // Get user-scoped K8s client (RBAC enforcement) + reqK8s, _ := GetK8sClientsForRequest(c) + if reqK8s == nil { + c.JSON(http.StatusUnauthorized, gin.H{"error": "Invalid or missing token"}) + c.Abort() + return + } + + // Create handler with user-scoped client (multi-tenant isolation) + handler := NewGitLabAuthHandler(reqK8s, project) + handler.ConnectGitLab(c) +} + +// GetGitLabStatusGlobal is the global handler for GET /projects/:projectName/auth/gitlab/status +func GetGitLabStatusGlobal(c *gin.Context) { + // Get project from URL parameter + project := c.Param("projectName") + if project == "" { + c.JSON(http.StatusBadRequest, gin.H{"error": "Project name is required"}) + return + } + + // Get user-scoped K8s client (RBAC enforcement) + reqK8s, _ := GetK8sClientsForRequest(c) + if reqK8s == nil { + c.JSON(http.StatusUnauthorized, gin.H{"error": "Invalid or missing token"}) + c.Abort() + return + } + + // Create handler with user-scoped client + handler := NewGitLabAuthHandler(reqK8s, project) + handler.GetGitLabStatus(c) +} + +// DisconnectGitLabGlobal is the global handler for POST /projects/:projectName/auth/gitlab/disconnect +func DisconnectGitLabGlobal(c *gin.Context) { + // Get project from URL parameter + project := c.Param("projectName") + if project == "" { + c.JSON(http.StatusBadRequest, gin.H{"error": "Project name is required"}) + return + } + + // Get user-scoped K8s client (RBAC enforcement) + reqK8s, _ := GetK8sClientsForRequest(c) + if reqK8s == nil { + c.JSON(http.StatusUnauthorized, gin.H{"error": "Invalid or missing token"}) + c.Abort() + return + } + + // Create handler with user-scoped client + handler := NewGitLabAuthHandler(reqK8s, project) + handler.DisconnectGitLab(c) +} diff --git a/components/backend/handlers/helpers.go b/components/backend/handlers/helpers.go index 17d2fcbe4..5db0be832 100644 --- a/components/backend/handlers/helpers.go +++ b/components/backend/handlers/helpers.go @@ -1,12 +1,16 @@ package handlers import ( + "context" "fmt" "log" "math" "time" + authv1 "k8s.io/api/authorization/v1" + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/kubernetes" ) // GetProjectSettingsResource returns the GroupVersionResource for ProjectSettings @@ -43,3 +47,29 @@ func RetryWithBackoff(maxRetries int, initialDelay, maxDelay time.Duration, oper } return fmt.Errorf("operation failed after %d retries: %w", maxRetries, lastErr) } + +// ValidateSecretAccess checks if the user has permission to perform the given verb on secrets +// Returns an error if the user lacks the required permission +func ValidateSecretAccess(ctx context.Context, k8sClient *kubernetes.Clientset, namespace, verb string) error { + ssar := &authv1.SelfSubjectAccessReview{ + Spec: authv1.SelfSubjectAccessReviewSpec{ + ResourceAttributes: &authv1.ResourceAttributes{ + Group: "", // core API group for secrets + Resource: "secrets", + Verb: verb, // "create", "get", "update", "delete" + Namespace: namespace, + }, + }, + } + + res, err := k8sClient.AuthorizationV1().SelfSubjectAccessReviews().Create(ctx, ssar, v1.CreateOptions{}) + if err != nil { + return fmt.Errorf("RBAC check failed: %w", err) + } + + if !res.Status.Allowed { + return fmt.Errorf("user not allowed to %s secrets in namespace %s", verb, namespace) + } + + return nil +} diff --git a/components/backend/handlers/repo.go b/components/backend/handlers/repo.go index 6e94a52f0..2373393fa 100644 --- a/components/backend/handlers/repo.go +++ b/components/backend/handlers/repo.go @@ -15,6 +15,10 @@ import ( v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/dynamic" "k8s.io/client-go/kubernetes" + + "ambient-code-backend/git" + "ambient-code-backend/gitlab" + "ambient-code-backend/types" ) // Dependencies injected from main package @@ -233,7 +237,7 @@ func CreateUserFork(c *gin.Context) { } // GetRepoTree handles GET /projects/:projectName/repo/tree -// Fetch repo tree entries via backend proxy +// Fetch repo tree entries via backend proxy (supports both GitHub and GitLab) func GetRepoTree(c *gin.Context) { project := c.Param("projectName") repo := c.Query("repo") @@ -248,77 +252,129 @@ func GetRepoTree(c *gin.Context) { userID, _ := c.Get("userID") reqK8s, reqDyn := GetK8sClientsForRequestRepo(c) - // Try to get GitHub token (GitHub App or PAT from runner secret) - token, err := GetGitHubTokenRepo(c.Request.Context(), reqK8s, reqDyn, project, userID.(string)) - if err != nil { - c.JSON(http.StatusUnauthorized, gin.H{"error": err.Error()}) - return - } + // Detect provider from repo URL + provider := types.DetectProvider(repo) - owner, repoName, err := parseOwnerRepo(repo) - if err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - } - api := githubAPIBaseURL("github.com") - p := path - if p == "" || p == "/" { - p = "" - } - url := fmt.Sprintf("%s/repos/%s/%s/contents/%s?ref=%s", api, owner, repoName, strings.TrimPrefix(p, "/"), ref) - resp, err := doGitHubRequest(c.Request.Context(), http.MethodGet, url, "Bearer "+token, "", nil) - if err != nil { - c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("GitHub request failed: %v", err)}) - return - } - defer resp.Body.Close() - if resp.StatusCode < 200 || resp.StatusCode >= 300 { - b, _ := io.ReadAll(resp.Body) - c.JSON(resp.StatusCode, gin.H{"error": string(b)}) - return - } - var decoded interface{} - if err := json.NewDecoder(resp.Body).Decode(&decoded); err != nil { - c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("failed to parse GitHub response: %v", err)}) - return - } - entries := []map[string]interface{}{} - if arr, ok := decoded.([]interface{}); ok { - for _, item := range arr { - if m, ok := item.(map[string]interface{}); ok { - name, _ := m["name"].(string) - typ, _ := m["type"].(string) - size, _ := m["size"].(float64) - mapped := "blob" - switch strings.ToLower(typ) { - case "dir": - mapped = "tree" - case "file", "symlink", "submodule": - mapped = "blob" - default: - if strings.TrimSpace(typ) == "" { + switch provider { + case types.ProviderGitLab: + // Handle GitLab repository + token, err := git.GetGitLabToken(c.Request.Context(), reqK8s, project, userID.(string)) + if err != nil { + c.JSON(http.StatusUnauthorized, gin.H{"error": err.Error()}) + return + } + + // Parse GitLab repository URL + parsed, err := gitlab.ParseGitLabURL(repo) + if err != nil { + c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("invalid GitLab URL: %v", err)}) + return + } + + // Create GitLab client and fetch tree + client := gitlab.NewClient(parsed.APIURL, token) + gitlabEntries, err := client.GetAllTreeEntries(c.Request.Context(), parsed.ProjectID, ref, path) + if err != nil { + if gitlabErr, ok := err.(*types.GitLabAPIError); ok { + c.JSON(gitlabErr.StatusCode, gin.H{"error": gitlabErr.Error()}) + return + } + c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("GitLab request failed: %v", err)}) + return + } + + // Map GitLab tree entries to common format + entries := gitlab.MapGitLabTreeEntriesToCommon(gitlabEntries) + c.JSON(http.StatusOK, gin.H{"path": path, "entries": entries}) + + case types.ProviderGitHub: + // Handle GitHub repository (existing logic) + token, err := GetGitHubTokenRepo(c.Request.Context(), reqK8s, reqDyn, project, userID.(string)) + if err != nil { + c.JSON(http.StatusUnauthorized, gin.H{"error": err.Error()}) + return + } + + owner, repoName, err := parseOwnerRepo(repo) + if err != nil { + c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + api := githubAPIBaseURL("github.com") + p := path + if p == "" || p == "/" { + p = "" + } + url := fmt.Sprintf("%s/repos/%s/%s/contents/%s?ref=%s", api, owner, repoName, strings.TrimPrefix(p, "/"), ref) + resp, err := doGitHubRequest(c.Request.Context(), http.MethodGet, url, "Bearer "+token, "", nil) + if err != nil { + c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("GitHub request failed: %v", err)}) + return + } + defer resp.Body.Close() + if resp.StatusCode < 200 || resp.StatusCode >= 300 { + b, _ := io.ReadAll(resp.Body) + c.JSON(resp.StatusCode, gin.H{"error": string(b)}) + return + } + var decoded interface{} + if err := json.NewDecoder(resp.Body).Decode(&decoded); err != nil { + c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("failed to parse GitHub response: %v", err)}) + return + } + entries := []types.TreeEntry{} + if arr, ok := decoded.([]interface{}); ok { + for _, item := range arr { + if m, ok := item.(map[string]interface{}); ok { + name, _ := m["name"].(string) + pathStr, _ := m["path"].(string) + typ, _ := m["type"].(string) + size, _ := m["size"].(float64) + mapped := "blob" + switch strings.ToLower(typ) { + case "dir": + mapped = "tree" + case "file", "symlink", "submodule": mapped = "blob" + default: + if strings.TrimSpace(typ) == "" { + mapped = "blob" + } } + entries = append(entries, types.TreeEntry{ + Name: name, + Path: pathStr, + Type: mapped, + Size: int(size), + }) } - entries = append(entries, map[string]interface{}{"name": name, "type": mapped, "size": int(size)}) } + } else if m, ok := decoded.(map[string]interface{}); ok { + // single file; present as one entry + name, _ := m["name"].(string) + pathStr, _ := m["path"].(string) + typ, _ := m["type"].(string) + size, _ := m["size"].(float64) + mapped := "blob" + if strings.ToLower(typ) == "dir" { + mapped = "tree" + } + entries = append(entries, types.TreeEntry{ + Name: name, + Path: pathStr, + Type: mapped, + Size: int(size), + }) } - } else if m, ok := decoded.(map[string]interface{}); ok { - // single file; present as one entry - name, _ := m["name"].(string) - typ, _ := m["type"].(string) - size, _ := m["size"].(float64) - mapped := "blob" - if strings.ToLower(typ) == "dir" { - mapped = "tree" - } - entries = append(entries, map[string]interface{}{"name": name, "type": mapped, "size": int(size)}) + c.JSON(http.StatusOK, gin.H{"path": path, "entries": entries}) + + default: + c.JSON(http.StatusBadRequest, gin.H{"error": "unsupported repository provider (only GitHub and GitLab are supported)"}) } - c.JSON(http.StatusOK, map[string]interface{}{"path": path, "entries": entries}) } // ListRepoBranches handles GET /projects/:projectName/repo/branches -// List all branches in a repository +// List all branches in a repository (supports both GitHub and GitLab) func ListRepoBranches(c *gin.Context) { project := c.Param("projectName") repo := c.Query("repo") @@ -331,58 +387,94 @@ func ListRepoBranches(c *gin.Context) { userID, _ := c.Get("userID") reqK8s, reqDyn := GetK8sClientsForRequestRepo(c) - // Try to get GitHub token (GitHub App or PAT from runner secret) - token, err := GetGitHubTokenRepo(c.Request.Context(), reqK8s, reqDyn, project, userID.(string)) - if err != nil { - c.JSON(http.StatusUnauthorized, gin.H{"error": err.Error()}) - return - } + // Detect provider from repo URL + provider := types.DetectProvider(repo) - owner, repoName, err := parseOwnerRepo(repo) - if err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - } + switch provider { + case types.ProviderGitLab: + // Handle GitLab repository + token, err := git.GetGitLabToken(c.Request.Context(), reqK8s, project, userID.(string)) + if err != nil { + c.JSON(http.StatusUnauthorized, gin.H{"error": err.Error()}) + return + } - api := githubAPIBaseURL("github.com") - url := fmt.Sprintf("%s/repos/%s/%s/branches", api, owner, repoName) - resp, err := doGitHubRequest(c.Request.Context(), http.MethodGet, url, "Bearer "+token, "", nil) - if err != nil { - c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("GitHub request failed: %v", err)}) - return - } - defer resp.Body.Close() + // Parse GitLab repository URL + parsed, err := gitlab.ParseGitLabURL(repo) + if err != nil { + c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("invalid GitLab URL: %v", err)}) + return + } - if resp.StatusCode < 200 || resp.StatusCode >= 300 { - b, _ := io.ReadAll(resp.Body) - c.JSON(resp.StatusCode, gin.H{"error": string(b)}) - return - } + // Create GitLab client and fetch branches + client := gitlab.NewClient(parsed.APIURL, token) + gitlabBranches, err := client.GetAllBranches(c.Request.Context(), parsed.ProjectID) + if err != nil { + if gitlabErr, ok := err.(*types.GitLabAPIError); ok { + c.JSON(gitlabErr.StatusCode, gin.H{"error": gitlabErr.Error()}) + return + } + c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("GitLab request failed: %v", err)}) + return + } - var branchesResp []map[string]interface{} - if err := json.NewDecoder(resp.Body).Decode(&branchesResp); err != nil { - c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("failed to parse GitHub response: %v", err)}) - return - } + // Map GitLab branches to common format + branches := gitlab.MapGitLabBranchesToCommon(gitlabBranches) + c.JSON(http.StatusOK, gin.H{"branches": branches}) - // Map branches to a simpler format - branches := make([]map[string]interface{}, 0, len(branchesResp)) - for _, b := range branchesResp { - name, _ := b["name"].(string) - if name != "" { - branches = append(branches, map[string]interface{}{ - "name": name, - }) + case types.ProviderGitHub: + // Handle GitHub repository (existing logic) + token, err := GetGitHubTokenRepo(c.Request.Context(), reqK8s, reqDyn, project, userID.(string)) + if err != nil { + c.JSON(http.StatusUnauthorized, gin.H{"error": err.Error()}) + return } - } - c.JSON(http.StatusOK, gin.H{ - "branches": branches, - }) + owner, repoName, err := parseOwnerRepo(repo) + if err != nil { + c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + api := githubAPIBaseURL("github.com") + url := fmt.Sprintf("%s/repos/%s/%s/branches", api, owner, repoName) + resp, err := doGitHubRequest(c.Request.Context(), http.MethodGet, url, "Bearer "+token, "", nil) + if err != nil { + c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("GitHub request failed: %v", err)}) + return + } + defer resp.Body.Close() + + if resp.StatusCode < 200 || resp.StatusCode >= 300 { + b, _ := io.ReadAll(resp.Body) + c.JSON(resp.StatusCode, gin.H{"error": string(b)}) + return + } + + var branchesResp []map[string]interface{} + if err := json.NewDecoder(resp.Body).Decode(&branchesResp); err != nil { + c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("failed to parse GitHub response: %v", err)}) + return + } + + // Map GitHub branches to common format + branches := make([]types.Branch, 0, len(branchesResp)) + for _, b := range branchesResp { + name, _ := b["name"].(string) + if name != "" { + branches = append(branches, types.Branch{Name: name}) + } + } + + c.JSON(http.StatusOK, gin.H{"branches": branches}) + + default: + c.JSON(http.StatusBadRequest, gin.H{"error": "unsupported repository provider (only GitHub and GitLab are supported)"}) + } } // GetRepoBlob handles GET /projects/:projectName/repo/blob -// Fetch blob (text) via backend proxy +// Fetch blob (text) via backend proxy (supports both GitHub and GitLab) func GetRepoBlob(c *gin.Context) { project := c.Param("projectName") repo := c.Query("repo") @@ -397,75 +489,130 @@ func GetRepoBlob(c *gin.Context) { userID, _ := c.Get("userID") reqK8s, reqDyn := GetK8sClientsForRequestRepo(c) - // Try to get GitHub token (GitHub App or PAT from runner secret) - token, err := GetGitHubTokenRepo(c.Request.Context(), reqK8s, reqDyn, project, userID.(string)) - if err != nil { - c.JSON(http.StatusUnauthorized, gin.H{"error": err.Error()}) - return - } + // Detect provider from repo URL + provider := types.DetectProvider(repo) - owner, repoName, err := parseOwnerRepo(repo) - if err != nil { - c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) - return - } - api := githubAPIBaseURL("github.com") - url := fmt.Sprintf("%s/repos/%s/%s/contents/%s?ref=%s", api, owner, repoName, strings.TrimPrefix(path, "/"), ref) - resp, err := doGitHubRequest(c.Request.Context(), http.MethodGet, url, "Bearer "+token, "", nil) - if err != nil { - c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("GitHub request failed: %v", err)}) - return - } - defer resp.Body.Close() - if resp.StatusCode < 200 || resp.StatusCode >= 300 { - b, _ := io.ReadAll(resp.Body) - c.JSON(resp.StatusCode, gin.H{"error": string(b)}) - return - } - // Decode generically first because GitHub returns an array for directories - var decoded interface{} - if err := json.NewDecoder(resp.Body).Decode(&decoded); err != nil { - c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("failed to parse GitHub response: %v", err)}) - return - } - // If the response is an array, the path is a directory. Return entries for convenience. - if arr, ok := decoded.([]interface{}); ok { - entries := []map[string]interface{}{} - for _, item := range arr { - if m, ok := item.(map[string]interface{}); ok { - name, _ := m["name"].(string) - typ, _ := m["type"].(string) - size, _ := m["size"].(float64) - mapped := "blob" - switch strings.ToLower(typ) { - case "dir": - mapped = "tree" - case "file", "symlink", "submodule": - mapped = "blob" - default: - if strings.TrimSpace(typ) == "" { - mapped = "blob" - } - } - entries = append(entries, map[string]interface{}{"name": name, "type": mapped, "size": int(size)}) + switch provider { + case types.ProviderGitLab: + // Handle GitLab repository + token, err := git.GetGitLabToken(c.Request.Context(), reqK8s, project, userID.(string)) + if err != nil { + c.JSON(http.StatusUnauthorized, gin.H{"error": err.Error()}) + return + } + + // Parse GitLab repository URL + parsed, err := gitlab.ParseGitLabURL(repo) + if err != nil { + c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("invalid GitLab URL: %v", err)}) + return + } + + // Create GitLab client and fetch file content + client := gitlab.NewClient(parsed.APIURL, token) + fileContent, err := client.GetFileContents(c.Request.Context(), parsed.ProjectID, path, ref) + if err != nil { + if gitlabErr, ok := err.(*types.GitLabAPIError); ok { + c.JSON(gitlabErr.StatusCode, gin.H{"error": gitlabErr.Error()}) + return } + c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("GitLab request failed: %v", err)}) + return } - c.JSON(http.StatusOK, gin.H{"isDir": true, "path": path, "entries": entries}) - return - } - // Otherwise, treat as a file object - if m, ok := decoded.(map[string]interface{}); ok { - content, _ := m["content"].(string) - encoding, _ := m["encoding"].(string) + + // Decode base64 content if needed + content := fileContent.Content + encoding := fileContent.Encoding if strings.ToLower(encoding) == "base64" { raw := strings.ReplaceAll(content, "\n", "") if data, err := base64.StdEncoding.DecodeString(raw); err == nil { - c.JSON(http.StatusOK, gin.H{"content": string(data), "encoding": "utf-8"}) - return + content = string(data) + encoding = "utf-8" } } + c.JSON(http.StatusOK, gin.H{"content": content, "encoding": encoding}) - return + + case types.ProviderGitHub: + // Handle GitHub repository (existing logic) + token, err := GetGitHubTokenRepo(c.Request.Context(), reqK8s, reqDyn, project, userID.(string)) + if err != nil { + c.JSON(http.StatusUnauthorized, gin.H{"error": err.Error()}) + return + } + + owner, repoName, err := parseOwnerRepo(repo) + if err != nil { + c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + api := githubAPIBaseURL("github.com") + url := fmt.Sprintf("%s/repos/%s/%s/contents/%s?ref=%s", api, owner, repoName, strings.TrimPrefix(path, "/"), ref) + resp, err := doGitHubRequest(c.Request.Context(), http.MethodGet, url, "Bearer "+token, "", nil) + if err != nil { + c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("GitHub request failed: %v", err)}) + return + } + defer resp.Body.Close() + if resp.StatusCode < 200 || resp.StatusCode >= 300 { + b, _ := io.ReadAll(resp.Body) + c.JSON(resp.StatusCode, gin.H{"error": string(b)}) + return + } + // Decode generically first because GitHub returns an array for directories + var decoded interface{} + if err := json.NewDecoder(resp.Body).Decode(&decoded); err != nil { + c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("failed to parse GitHub response: %v", err)}) + return + } + // If the response is an array, the path is a directory. Return entries for convenience. + if arr, ok := decoded.([]interface{}); ok { + entries := []types.TreeEntry{} + for _, item := range arr { + if m, ok := item.(map[string]interface{}); ok { + name, _ := m["name"].(string) + pathStr, _ := m["path"].(string) + typ, _ := m["type"].(string) + size, _ := m["size"].(float64) + mapped := "blob" + switch strings.ToLower(typ) { + case "dir": + mapped = "tree" + case "file", "symlink", "submodule": + mapped = "blob" + default: + if strings.TrimSpace(typ) == "" { + mapped = "blob" + } + } + entries = append(entries, types.TreeEntry{ + Name: name, + Path: pathStr, + Type: mapped, + Size: int(size), + }) + } + } + c.JSON(http.StatusOK, gin.H{"isDir": true, "path": path, "entries": entries}) + return + } + // Otherwise, treat as a file object + if m, ok := decoded.(map[string]interface{}); ok { + content, _ := m["content"].(string) + encoding, _ := m["encoding"].(string) + if strings.ToLower(encoding) == "base64" { + raw := strings.ReplaceAll(content, "\n", "") + if data, err := base64.StdEncoding.DecodeString(raw); err == nil { + c.JSON(http.StatusOK, gin.H{"content": string(data), "encoding": "utf-8"}) + return + } + } + c.JSON(http.StatusOK, gin.H{"content": content, "encoding": encoding}) + return + } + + default: + c.JSON(http.StatusBadRequest, gin.H{"error": "unsupported repository provider (only GitHub and GitLab are supported)"}) } // Fallback unexpected structure c.JSON(http.StatusBadGateway, gin.H{"error": "unexpected GitHub response structure"}) diff --git a/components/backend/handlers/repo_seed.go b/components/backend/handlers/repo_seed.go new file mode 100644 index 000000000..bd36edcfe --- /dev/null +++ b/components/backend/handlers/repo_seed.go @@ -0,0 +1,433 @@ +package handlers + +import ( + "context" + "fmt" + "log" + "net/http" + "os" + "os/exec" + "path/filepath" + "strings" + "time" + + "github.com/gin-gonic/gin" + + "ambient-code-backend/git" + "ambient-code-backend/types" +) + +// SeedingStatus represents the status of repository seeding +type SeedingStatus struct { + Required bool `json:"required"` + MissingDirs []string `json:"missingDirs,omitempty"` + MissingFiles []string `json:"missingFiles,omitempty"` + InProgress bool `json:"inProgress"` + LastSeeded *string `json:"lastSeeded,omitempty"` + Error string `json:"error,omitempty"` + CompletedAt *string `json:"completedAt,omitempty"` + RepositoryURL string `json:"repositoryUrl"` +} + +// SeedRequest represents a request to seed a repository +type SeedRequest struct { + RepositoryURL string `json:"repositoryUrl" binding:"required"` + Branch string `json:"branch"` + Force bool `json:"force"` // Force re-seed even if structure exists +} + +// SeedResponse represents the response from a seeding operation +type SeedResponse struct { + Success bool `json:"success"` + Message string `json:"message"` + SeededDirs []string `json:"seededDirs,omitempty"` + SeededFiles []string `json:"seededFiles,omitempty"` + CommitSHA string `json:"commitSha,omitempty"` + Error string `json:"error,omitempty"` + RepositoryURL string `json:"repositoryUrl"` +} + +// RequiredClaudeStructure defines the required .claude/ directory structure +var RequiredClaudeStructure = map[string][]string{ + ".claude": {}, + ".claude/commands": { + "README.md", + }, +} + +// ClaudeTemplates contains default template content for .claude/ files +var ClaudeTemplates = map[string]string{ + ".claude/README.md": "# Claude Code Configuration\n\n" + + "This directory contains configuration for Claude Code integration.\n\n" + + "## Structure\n\n" + + "- `commands/` - Custom slash commands for this project\n" + + "- `settings.local.json` - Local Claude Code settings (not committed)\n\n" + + "## Documentation\n\n" + + "For more information, see the [Claude Code documentation](https://docs.claude.com/claude-code).\n", + + ".claude/commands/README.md": "# Custom Commands\n\n" + + "Add custom slash commands for your project here.\n\n" + + "Each command is a markdown file that defines:\n" + + "- Command name (from filename)\n" + + "- Command description\n" + + "- Prompt template\n\n" + + "## Example\n\n" + + "Create `analyze.md`:\n\n" + + "```markdown\n" + + "Analyze the codebase and provide insights about:\n" + + "- Architecture patterns\n" + + "- Code quality issues\n" + + "- Potential improvements\n" + + "```\n\n" + + "Then use with `/analyze` in Claude Code.\n", + ".claude/settings.local.json": `{ + "permissions": { + "allow": [], + "deny": [], + "ask": [] + } +} +`, + ".claude/.gitignore": `settings.local.json +*.log +`, +} + +// DetectMissingStructure checks if a repository is missing required .claude/ structure +func DetectMissingStructure(ctx context.Context, repoPath string) (*SeedingStatus, error) { + status := &SeedingStatus{ + Required: false, + MissingDirs: []string{}, + MissingFiles: []string{}, + InProgress: false, + RepositoryURL: "", + } + + // Check each required directory + for dir, files := range RequiredClaudeStructure { + dirPath := filepath.Join(repoPath, dir) + if _, err := os.Stat(dirPath); os.IsNotExist(err) { + status.Required = true + status.MissingDirs = append(status.MissingDirs, dir) + } else { + // Directory exists, check required files + for _, file := range files { + filePath := filepath.Join(dirPath, file) + if _, err := os.Stat(filePath); os.IsNotExist(err) { + status.Required = true + status.MissingFiles = append(status.MissingFiles, filepath.Join(dir, file)) + } + } + } + } + + return status, nil +} + +// SeedRepository creates the .claude/ directory structure in a repository +func SeedRepository(ctx context.Context, repoPath, repoURL, branch, userEmail, userName string) (*SeedResponse, error) { + response := &SeedResponse{ + Success: false, + SeededDirs: []string{}, + SeededFiles: []string{}, + RepositoryURL: repoURL, + } + + // Create required directories + for dir := range RequiredClaudeStructure { + dirPath := filepath.Join(repoPath, dir) + if err := os.MkdirAll(dirPath, 0755); err != nil { + response.Error = fmt.Sprintf("Failed to create directory %s: %v", dir, err) + return response, err + } + response.SeededDirs = append(response.SeededDirs, dir) + } + + // Copy template files + for templatePath, content := range ClaudeTemplates { + filePath := filepath.Join(repoPath, templatePath) + + // Check if file already exists + if _, err := os.Stat(filePath); err == nil { + // File exists, skip + continue + } + + // Create parent directory if needed + parentDir := filepath.Dir(filePath) + if err := os.MkdirAll(parentDir, 0755); err != nil { + response.Error = fmt.Sprintf("Failed to create parent directory for %s: %v", templatePath, err) + return response, err + } + + // Write template content + if err := os.WriteFile(filePath, []byte(content), 0644); err != nil { + response.Error = fmt.Sprintf("Failed to write template file %s: %v", templatePath, err) + return response, err + } + response.SeededFiles = append(response.SeededFiles, templatePath) + } + + // Commit changes + commitMsg := "chore: initialize .claude/ directory structure\n\nAdd Claude Code configuration for AI-assisted development.\n\n🤖 Seeded by vTeam Ambient Code Platform" + + // Configure git user if provided + if userEmail != "" && userName != "" { + gitConfig := exec.CommandContext(ctx, "git", "-C", repoPath, "config", "user.email", userEmail) + if err := gitConfig.Run(); err != nil { + response.Error = fmt.Sprintf("Failed to configure git user email: %v", err) + return response, err + } + + gitConfig = exec.CommandContext(ctx, "git", "-C", repoPath, "config", "user.name", userName) + if err := gitConfig.Run(); err != nil { + response.Error = fmt.Sprintf("Failed to configure git user name: %v", err) + return response, err + } + } + + // Add files to git + gitAdd := exec.CommandContext(ctx, "git", "-C", repoPath, "add", ".claude/") + if err := gitAdd.Run(); err != nil { + response.Error = fmt.Sprintf("Failed to add files to git: %v", err) + return response, err + } + + // Commit + gitCommit := exec.CommandContext(ctx, "git", "-C", repoPath, "commit", "-m", commitMsg) + if output, err := gitCommit.CombinedOutput(); err != nil { + // Check if error is because there's nothing to commit + if strings.Contains(string(output), "nothing to commit") { + response.Message = "Claude structure already exists, nothing to seed" + response.Success = true + return response, nil + } + response.Error = fmt.Sprintf("Failed to commit changes: %v - %s", err, string(output)) + return response, err + } + + // Get commit SHA + gitRev := exec.CommandContext(ctx, "git", "-C", repoPath, "rev-parse", "HEAD") + if output, err := gitRev.Output(); err == nil { + response.CommitSHA = strings.TrimSpace(string(output)) + } + + response.Success = true + response.Message = fmt.Sprintf("Successfully seeded .claude/ structure with %d directories and %d files", + len(response.SeededDirs), len(response.SeededFiles)) + + return response, nil +} + +// GetRepoSeedStatus handles GET /projects/:project/repo/seed-status +func GetRepoSeedStatus(c *gin.Context) { + project := c.Param("projectName") + repoURL := c.Query("repo") + + if repoURL == "" { + c.JSON(http.StatusBadRequest, gin.H{"error": "repo query parameter required"}) + return + } + + userID, _ := c.Get("userID") + reqK8s, reqDyn := GetK8sClientsForRequestRepo(c) + + // Detect provider + provider := types.DetectProvider(repoURL) + + // Clone repository temporarily to check structure + tmpDir, err := os.MkdirTemp("", "seed-check-*") + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Failed to create temp directory: %v", err)}) + return + } + defer func() { + if err := os.RemoveAll(tmpDir); err != nil { + log.Printf("Warning: failed to cleanup temp directory %s: %v", tmpDir, err) + } + }() + + // Get appropriate token + var token string + switch provider { + case types.ProviderGitLab: + token, err = git.GetGitLabToken(c.Request.Context(), reqK8s, project, userID.(string)) + if err != nil { + c.JSON(http.StatusUnauthorized, gin.H{"error": err.Error()}) + return + } + case types.ProviderGitHub: + token, err = GetGitHubTokenRepo(c.Request.Context(), reqK8s, reqDyn, project, userID.(string)) + if err != nil { + c.JSON(http.StatusUnauthorized, gin.H{"error": err.Error()}) + return + } + default: + c.JSON(http.StatusBadRequest, gin.H{"error": "unsupported repository provider"}) + return + } + + // Clone repository + authURL, err := git.InjectGitToken(repoURL, token) + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Failed to prepare repository URL: %v", err)}) + return + } + + gitClone := exec.CommandContext(c.Request.Context(), "git", "clone", "--depth", "1", authURL, tmpDir) + if output, err := gitClone.CombinedOutput(); err != nil { + c.JSON(http.StatusBadGateway, gin.H{"error": fmt.Sprintf("Failed to clone repository: %v - %s", err, string(output))}) + return + } + + // Detect missing structure + status, err := DetectMissingStructure(c.Request.Context(), tmpDir) + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Failed to detect structure: %v", err)}) + return + } + + status.RepositoryURL = repoURL + c.JSON(http.StatusOK, status) +} + +// SeedRepositoryEndpoint handles POST /projects/:project/repo/seed +func SeedRepositoryEndpoint(c *gin.Context) { + project := c.Param("projectName") + + var req SeedRequest + if err := c.ShouldBindJSON(&req); err != nil { + c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("Invalid request: %v", err)}) + return + } + + if req.Branch == "" { + req.Branch = "main" + } + + userID, _ := c.Get("userID") + reqK8s, reqDyn := GetK8sClientsForRequestRepo(c) + + // Detect provider + provider := types.DetectProvider(req.RepositoryURL) + + // Get appropriate token + var token string + var err error + switch provider { + case types.ProviderGitLab: + token, err = git.GetGitLabToken(c.Request.Context(), reqK8s, project, userID.(string)) + if err != nil { + c.JSON(http.StatusUnauthorized, gin.H{ + "error": err.Error(), + "remediation": "Connect your GitLab account via /auth/gitlab/connect", + }) + return + } + case types.ProviderGitHub: + token, err = GetGitHubTokenRepo(c.Request.Context(), reqK8s, reqDyn, project, userID.(string)) + if err != nil { + c.JSON(http.StatusUnauthorized, gin.H{ + "error": err.Error(), + "remediation": "Ensure GitHub App is installed or configure GIT_TOKEN in project runner secret", + }) + return + } + default: + c.JSON(http.StatusBadRequest, gin.H{"error": "unsupported repository provider"}) + return + } + + // Clone repository + tmpDir, err := os.MkdirTemp("", "repo-seed-*") + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Failed to create temp directory: %v", err)}) + return + } + defer func() { + if err := os.RemoveAll(tmpDir); err != nil { + log.Printf("Warning: failed to cleanup temp directory %s: %v", tmpDir, err) + } + }() + + authURL, err := git.InjectGitToken(req.RepositoryURL, token) + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Failed to prepare repository URL: %v", err)}) + return + } + + gitClone := exec.CommandContext(c.Request.Context(), "git", "clone", "--branch", req.Branch, authURL, tmpDir) + if output, err := gitClone.CombinedOutput(); err != nil { + c.JSON(http.StatusBadGateway, gin.H{ + "error": fmt.Sprintf("Failed to clone repository: %v", err), + "details": string(output), + "remediation": "Verify repository URL and branch name, ensure token has read/write access", + }) + return + } + + // Check if seeding is needed + status, err := DetectMissingStructure(c.Request.Context(), tmpDir) + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("Failed to detect structure: %v", err)}) + return + } + + if !status.Required && !req.Force { + c.JSON(http.StatusOK, SeedResponse{ + Success: true, + Message: "Repository already has .claude/ structure, no seeding needed", + RepositoryURL: req.RepositoryURL, + }) + return + } + + // Get user info for git commits (use a default if not available) + userEmail := "ambient-bot@vteam.ambient-code" + userName := "vTeam Ambient Bot" + + // Seed repository + response, err := SeedRepository(c.Request.Context(), tmpDir, req.RepositoryURL, req.Branch, userEmail, userName) + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{ + "error": response.Error, + "remediation": "Check repository permissions and try again", + }) + return + } + + // Push changes back to remote + gitPush := exec.CommandContext(c.Request.Context(), "git", "-C", tmpDir, "push", "origin", req.Branch) + if output, err := gitPush.CombinedOutput(); err != nil { + // Check for permission errors + outputStr := string(output) + if strings.Contains(outputStr, "403") || strings.Contains(outputStr, "Permission denied") { + remediation := "Ensure your token has write access to the repository" + if provider == types.ProviderGitLab { + remediation = "Ensure your GitLab PAT has 'write_repository' scope" + } + c.JSON(http.StatusForbidden, gin.H{ + "error": "Failed to push changes: permission denied", + "details": outputStr, + "remediation": remediation, + }) + return + } + + c.JSON(http.StatusBadGateway, gin.H{ + "error": fmt.Sprintf("Failed to push changes: %v", err), + "details": outputStr, + "remediation": "Check repository permissions and network connectivity", + }) + return + } + + // Add timestamp + now := time.Now().Format(time.RFC3339) + response.Success = true + if response.Message == "" { + response.Message = fmt.Sprintf("Successfully seeded and pushed .claude/ structure at %s", now) + } + + c.JSON(http.StatusOK, response) +} diff --git a/components/backend/handlers/repository.go b/components/backend/handlers/repository.go new file mode 100644 index 000000000..5da10a494 --- /dev/null +++ b/components/backend/handlers/repository.go @@ -0,0 +1,154 @@ +package handlers + +import ( + "context" + "fmt" + + "ambient-code-backend/gitlab" + "ambient-code-backend/types" +) + +// DetectRepositoryProvider determines the Git provider from a repository URL +func DetectRepositoryProvider(repoURL string) types.ProviderType { + return types.DetectProvider(repoURL) +} + +// ValidateGitLabRepository validates a GitLab repository URL and token access +func ValidateGitLabRepository(ctx context.Context, repoURL, token string) error { + if token == "" { + return fmt.Errorf("GitLab token is required for repository validation") + } + + // Validate URL format + if err := gitlab.ValidateGitLabURL(repoURL); err != nil { + return fmt.Errorf("invalid GitLab repository URL: %w", err) + } + + // Validate token and repository access + result, err := gitlab.ValidateTokenAndRepository(ctx, token, repoURL) + if err != nil { + return fmt.Errorf("failed to validate GitLab repository: %w", err) + } + + if !result.Valid { + return fmt.Errorf("GitLab validation failed: %s", result.ErrorMessage) + } + + return nil +} + +// NormalizeRepositoryURL normalizes a repository URL based on its provider +func NormalizeRepositoryURL(repoURL string, provider types.ProviderType) (string, error) { + switch provider { + case types.ProviderGitLab: + return gitlab.NormalizeGitLabURL(repoURL) + case types.ProviderGitHub: + // GitHub normalization would go here (if implemented) + return repoURL, nil + default: + return repoURL, fmt.Errorf("unsupported provider: %s", provider) + } +} + +// GetRepositoryInfo retrieves information about a repository +func GetRepositoryInfo(repoURL string) (*RepositoryInfo, error) { + provider := DetectRepositoryProvider(repoURL) + + info := &RepositoryInfo{ + URL: repoURL, + Provider: provider, + } + + switch provider { + case types.ProviderGitLab: + parsed, err := gitlab.ParseGitLabURL(repoURL) + if err != nil { + return nil, fmt.Errorf("failed to parse GitLab URL: %w", err) + } + info.Owner = parsed.Owner + info.Repo = parsed.Repo + info.Host = parsed.Host + info.APIURL = parsed.APIURL + info.IsGitLabSelfHosted = gitlab.IsGitLabSelfHosted(parsed.Host) + + case types.ProviderGitHub: + // GitHub parsing would go here (if needed) + info.Host = "github.com" + + default: + return nil, fmt.Errorf("unsupported provider: %s", provider) + } + + return info, nil +} + +// RepositoryInfo contains parsed information about a repository +type RepositoryInfo struct { + URL string `json:"url"` + Provider types.ProviderType `json:"provider"` + Owner string `json:"owner,omitempty"` + Repo string `json:"repo,omitempty"` + Host string `json:"host,omitempty"` + APIURL string `json:"apiUrl,omitempty"` + IsGitLabSelfHosted bool `json:"isGitlabSelfHosted,omitempty"` +} + +// ValidateProjectRepository validates a repository for use in a project +func ValidateProjectRepository(ctx context.Context, repoURL string, userID string) (*RepositoryInfo, error) { + // Get repository info + info, err := GetRepositoryInfo(repoURL) + if err != nil { + return nil, err + } + + // For GitLab repositories, validate access if we have a token + if info.Provider == types.ProviderGitLab { + // Try to get GitLab token for this user + // Use the handlers package K8sClient and Namespace globals + connMgr := gitlab.NewConnectionManager(K8sClient, Namespace) + _, token, err := connMgr.GetGitLabConnectionWithToken(ctx, userID) + if err != nil { + // If no token found, just return info without validation + // The user will need to connect GitLab account first + gitlab.LogWarning("No GitLab token found for user %s, skipping repository validation", userID) + return info, nil + } + + // Validate repository access with the token + if err := ValidateGitLabRepository(ctx, repoURL, token); err != nil { + return nil, fmt.Errorf("repository validation failed: %w", err) + } + + gitlab.LogInfo("GitLab repository %s validated successfully for user %s", repoURL, userID) + } + + return info, nil +} + +// EnrichProjectSettingsWithProviders adds provider information to repositories in ProjectSettings +func EnrichProjectSettingsWithProviders(repositories []map[string]interface{}) []map[string]interface{} { + enriched := make([]map[string]interface{}, len(repositories)) + + for i, repo := range repositories { + enrichedRepo := make(map[string]interface{}) + + // Copy existing fields + for k, v := range repo { + enrichedRepo[k] = v + } + + // Add provider if not already present + if _, hasProvider := repo["provider"]; !hasProvider { + if url, hasURL := repo["url"].(string); hasURL { + provider := DetectRepositoryProvider(url) + if provider != "" { + enrichedRepo["provider"] = string(provider) + } + } + } + + enriched[i] = enrichedRepo + } + + return enriched +} diff --git a/components/backend/k8s/configmap.go b/components/backend/k8s/configmap.go new file mode 100644 index 000000000..f5d791113 --- /dev/null +++ b/components/backend/k8s/configmap.go @@ -0,0 +1,161 @@ +package k8s + +import ( + "context" + "encoding/json" + "fmt" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes" + + "ambient-code-backend/types" +) + +const ( + // GitLabConnectionsConfigMapName is the name of the ConfigMap storing GitLab connection metadata + GitLabConnectionsConfigMapName = "gitlab-connections" +) + +// StoreGitLabConnection stores GitLab connection metadata in a ConfigMap +func StoreGitLabConnection(ctx context.Context, clientset *kubernetes.Clientset, namespace string, connection *types.GitLabConnection) error { + configMapsClient := clientset.CoreV1().ConfigMaps(namespace) + + // Serialize connection to JSON + connectionJSON, err := json.Marshal(connection) + if err != nil { + return fmt.Errorf("failed to serialize connection: %w", err) + } + + // Get existing ConfigMap or create new one + configMap, err := configMapsClient.Get(ctx, GitLabConnectionsConfigMapName, metav1.GetOptions{}) + if errors.IsNotFound(err) { + // Create new ConfigMap + configMap = &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: GitLabConnectionsConfigMapName, + Namespace: namespace, + }, + Data: map[string]string{ + connection.UserID: string(connectionJSON), + }, + } + + _, err = configMapsClient.Create(ctx, configMap, metav1.CreateOptions{}) + if err != nil { + return fmt.Errorf("failed to create GitLab connections ConfigMap: %w", err) + } + + return nil + } else if err != nil { + return fmt.Errorf("failed to get GitLab connections ConfigMap: %w", err) + } + + // Update existing ConfigMap + if configMap.Data == nil { + configMap.Data = make(map[string]string) + } + + configMap.Data[connection.UserID] = string(connectionJSON) + + _, err = configMapsClient.Update(ctx, configMap, metav1.UpdateOptions{}) + if err != nil { + return fmt.Errorf("failed to update GitLab connections ConfigMap: %w", err) + } + + return nil +} + +// GetGitLabConnection retrieves GitLab connection metadata from a ConfigMap +func GetGitLabConnection(ctx context.Context, clientset *kubernetes.Clientset, namespace, userID string) (*types.GitLabConnection, error) { + configMapsClient := clientset.CoreV1().ConfigMaps(namespace) + + configMap, err := configMapsClient.Get(ctx, GitLabConnectionsConfigMapName, metav1.GetOptions{}) + if err != nil { + if errors.IsNotFound(err) { + return nil, fmt.Errorf("GitLab connections ConfigMap not found") + } + return nil, fmt.Errorf("failed to get GitLab connections ConfigMap: %w", err) + } + + connectionJSON, exists := configMap.Data[userID] + if !exists { + return nil, fmt.Errorf("no GitLab connection found for user %s", userID) + } + + var connection types.GitLabConnection + if err := json.Unmarshal([]byte(connectionJSON), &connection); err != nil { + return nil, fmt.Errorf("failed to parse connection data: %w", err) + } + + return &connection, nil +} + +// DeleteGitLabConnection removes GitLab connection metadata from a ConfigMap +func DeleteGitLabConnection(ctx context.Context, clientset *kubernetes.Clientset, namespace, userID string) error { + configMapsClient := clientset.CoreV1().ConfigMaps(namespace) + + configMap, err := configMapsClient.Get(ctx, GitLabConnectionsConfigMapName, metav1.GetOptions{}) + if err != nil { + if errors.IsNotFound(err) { + return nil // Already doesn't exist + } + return fmt.Errorf("failed to get GitLab connections ConfigMap: %w", err) + } + + if configMap.Data == nil { + return nil // No data to delete + } + + delete(configMap.Data, userID) + + _, err = configMapsClient.Update(ctx, configMap, metav1.UpdateOptions{}) + if err != nil { + return fmt.Errorf("failed to update GitLab connections ConfigMap: %w", err) + } + + return nil +} + +// HasGitLabConnection checks if a user has a GitLab connection stored +func HasGitLabConnection(ctx context.Context, clientset *kubernetes.Clientset, namespace, userID string) (bool, error) { + configMapsClient := clientset.CoreV1().ConfigMaps(namespace) + + configMap, err := configMapsClient.Get(ctx, GitLabConnectionsConfigMapName, metav1.GetOptions{}) + if err != nil { + if errors.IsNotFound(err) { + return false, nil + } + return false, fmt.Errorf("failed to get GitLab connections ConfigMap: %w", err) + } + + _, exists := configMap.Data[userID] + return exists, nil +} + +// ListGitLabConnections retrieves all GitLab connections from a ConfigMap +func ListGitLabConnections(ctx context.Context, clientset *kubernetes.Clientset, namespace string) ([]*types.GitLabConnection, error) { + configMapsClient := clientset.CoreV1().ConfigMaps(namespace) + + configMap, err := configMapsClient.Get(ctx, GitLabConnectionsConfigMapName, metav1.GetOptions{}) + if err != nil { + if errors.IsNotFound(err) { + return []*types.GitLabConnection{}, nil + } + return nil, fmt.Errorf("failed to get GitLab connections ConfigMap: %w", err) + } + + connections := make([]*types.GitLabConnection, 0, len(configMap.Data)) + + for _, connectionJSON := range configMap.Data { + var connection types.GitLabConnection + if err := json.Unmarshal([]byte(connectionJSON), &connection); err != nil { + // Skip invalid entries + continue + } + connections = append(connections, &connection) + } + + return connections, nil +} diff --git a/components/backend/k8s/secrets.go b/components/backend/k8s/secrets.go new file mode 100644 index 000000000..80105915a --- /dev/null +++ b/components/backend/k8s/secrets.go @@ -0,0 +1,171 @@ +package k8s + +import ( + "context" + "fmt" + "time" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes" +) + +const ( + // GitLabTokensSecretName is the name of the secret storing GitLab PATs + GitLabTokensSecretName = "gitlab-user-tokens" +) + +// StoreGitLabToken stores a GitLab Personal Access Token in Kubernetes Secrets +// Uses optimistic concurrency control with retry to handle concurrent updates +func StoreGitLabToken(ctx context.Context, clientset *kubernetes.Clientset, namespace, userID, token string) error { + secretsClient := clientset.CoreV1().Secrets(namespace) + + // Retry up to 3 times with exponential backoff + const maxRetries = 3 + var lastErr error + + for attempt := 0; attempt < maxRetries; attempt++ { + // Get existing secret or create new one + secret, err := secretsClient.Get(ctx, GitLabTokensSecretName, metav1.GetOptions{}) + if errors.IsNotFound(err) { + // Create new secret + secret = &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{ + Name: GitLabTokensSecretName, + Namespace: namespace, + }, + Type: corev1.SecretTypeOpaque, + StringData: map[string]string{ + userID: token, + }, + } + + _, err = secretsClient.Create(ctx, secret, metav1.CreateOptions{}) + if err != nil && !errors.IsAlreadyExists(err) { + return fmt.Errorf("failed to create GitLab tokens secret: %w", err) + } + if err == nil { + return nil + } + // If AlreadyExists, retry the Get-Update loop + lastErr = err + time.Sleep(time.Millisecond * 100 * time.Duration(attempt+1)) + continue + } else if err != nil { + return fmt.Errorf("failed to get GitLab tokens secret: %w", err) + } + + // Update existing secret + // Make a deep copy to avoid modifying the original + secretCopy := secret.DeepCopy() + + // Update the data in the copy + if secretCopy.Data == nil { + secretCopy.Data = make(map[string][]byte) + } + secretCopy.Data[userID] = []byte(token) + + // Attempt update with current ResourceVersion (optimistic concurrency) + _, err = secretsClient.Update(ctx, secretCopy, metav1.UpdateOptions{}) + if err == nil { + return nil + } + + // If conflict, retry + if errors.IsConflict(err) { + lastErr = err + // Exponential backoff: 100ms, 200ms, 400ms + time.Sleep(time.Millisecond * 100 * time.Duration(attempt+1)) + continue + } + + // Other errors are not retryable + return fmt.Errorf("failed to update GitLab tokens secret: %w", err) + } + + return fmt.Errorf("failed to update GitLab tokens secret after %d retries: %w", maxRetries, lastErr) +} + +// GetGitLabToken retrieves a GitLab Personal Access Token from Kubernetes Secrets +func GetGitLabToken(ctx context.Context, clientset *kubernetes.Clientset, namespace, userID string) (string, error) { + secretsClient := clientset.CoreV1().Secrets(namespace) + + secret, err := secretsClient.Get(ctx, GitLabTokensSecretName, metav1.GetOptions{}) + if err != nil { + if errors.IsNotFound(err) { + return "", fmt.Errorf("GitLab tokens secret not found") + } + return "", fmt.Errorf("failed to get GitLab tokens secret: %w", err) + } + + tokenBytes, exists := secret.Data[userID] + if !exists { + return "", fmt.Errorf("no GitLab token found for user %s", userID) + } + + return string(tokenBytes), nil +} + +// DeleteGitLabToken removes a GitLab Personal Access Token from Kubernetes Secrets +// Uses optimistic concurrency control with retry to handle concurrent updates +func DeleteGitLabToken(ctx context.Context, clientset *kubernetes.Clientset, namespace, userID string) error { + secretsClient := clientset.CoreV1().Secrets(namespace) + + // Retry up to 3 times with exponential backoff + const maxRetries = 3 + var lastErr error + + for attempt := 0; attempt < maxRetries; attempt++ { + secret, err := secretsClient.Get(ctx, GitLabTokensSecretName, metav1.GetOptions{}) + if err != nil { + if errors.IsNotFound(err) { + return nil // Already doesn't exist + } + return fmt.Errorf("failed to get GitLab tokens secret: %w", err) + } + + if secret.Data == nil || secret.Data[userID] == nil { + return nil // No data to delete + } + + // Make a deep copy to avoid modifying the original + secretCopy := secret.DeepCopy() + delete(secretCopy.Data, userID) + + // Attempt update with current ResourceVersion (optimistic concurrency) + _, err = secretsClient.Update(ctx, secretCopy, metav1.UpdateOptions{}) + if err == nil { + return nil + } + + // If conflict, retry + if errors.IsConflict(err) { + lastErr = err + // Exponential backoff: 100ms, 200ms, 400ms + time.Sleep(time.Millisecond * 100 * time.Duration(attempt+1)) + continue + } + + // Other errors are not retryable + return fmt.Errorf("failed to update GitLab tokens secret: %w", err) + } + + return fmt.Errorf("failed to delete GitLab token after %d retries: %w", maxRetries, lastErr) +} + +// HasGitLabToken checks if a user has a GitLab token stored +func HasGitLabToken(ctx context.Context, clientset *kubernetes.Clientset, namespace, userID string) (bool, error) { + secretsClient := clientset.CoreV1().Secrets(namespace) + + secret, err := secretsClient.Get(ctx, GitLabTokensSecretName, metav1.GetOptions{}) + if err != nil { + if errors.IsNotFound(err) { + return false, nil + } + return false, fmt.Errorf("failed to get GitLab tokens secret: %w", err) + } + + _, exists := secret.Data[userID] + return exists, nil +} diff --git a/components/backend/main.go b/components/backend/main.go index b2b343c04..e4803193a 100644 --- a/components/backend/main.go +++ b/components/backend/main.go @@ -64,6 +64,9 @@ func main() { return github.GetInstallation(ctx, userID) } git.GitHubTokenManager = github.Manager + git.GetBackendNamespace = func() string { + return server.Namespace + } // Initialize content handlers handlers.StateBaseDir = server.StateBaseDir diff --git a/components/backend/routes.go b/components/backend/routes.go index b4a28b1ff..54fb86189 100644 --- a/components/backend/routes.go +++ b/components/backend/routes.go @@ -43,6 +43,8 @@ func registerRoutes(r *gin.Engine) { projectGroup.GET("/repo/tree", handlers.GetRepoTree) projectGroup.GET("/repo/blob", handlers.GetRepoBlob) projectGroup.GET("/repo/branches", handlers.ListRepoBranches) + projectGroup.GET("/repo/seed-status", handlers.GetRepoSeedStatus) + projectGroup.POST("/repo/seed", handlers.SeedRepositoryEndpoint) projectGroup.GET("/agentic-sessions", handlers.ListSessions) projectGroup.POST("/agentic-sessions", handlers.CreateSession) @@ -95,6 +97,11 @@ func registerRoutes(r *gin.Engine) { projectGroup.PUT("/runner-secrets", handlers.UpdateRunnerSecrets) projectGroup.GET("/integration-secrets", handlers.ListIntegrationSecrets) projectGroup.PUT("/integration-secrets", handlers.UpdateIntegrationSecrets) + + // GitLab authentication endpoints (project-scoped) + projectGroup.POST("/auth/gitlab/connect", handlers.ConnectGitLabGlobal) + projectGroup.GET("/auth/gitlab/status", handlers.GetGitLabStatusGlobal) + projectGroup.POST("/auth/gitlab/disconnect", handlers.DisconnectGitLabGlobal) } api.POST("/auth/github/install", handlers.LinkGitHubInstallationGlobal) diff --git a/components/backend/tests/integration/gitlab/README.md b/components/backend/tests/integration/gitlab/README.md new file mode 100644 index 000000000..26ebae971 --- /dev/null +++ b/components/backend/tests/integration/gitlab/README.md @@ -0,0 +1,314 @@ +# GitLab Integration Tests + +This directory contains end-to-end integration tests for the GitLab integration functionality. + +## Overview + +The integration tests validate the complete GitLab workflow: +1. Token validation and storage +2. Connection management +3. Repository configuration and validation +4. Git operations (clone, push) +5. Error handling +6. Token security (redaction) +7. Self-hosted GitLab support + +## Prerequisites + +### Required Environment Variables + +**For GitLab.com Tests**: +```bash +export INTEGRATION_TESTS=true +export GITLAB_TEST_TOKEN="glpat-your-token-here" +export GITLAB_TEST_REPO_URL="https://gitlab.com/yourusername/test-repo.git" +``` + +**For Self-Hosted GitLab Tests** (optional): +```bash +export GITLAB_SELFHOSTED_TOKEN="glpat-your-selfhosted-token" +export GITLAB_SELFHOSTED_URL="https://gitlab.company.com" +export GITLAB_SELFHOSTED_REPO_URL="https://gitlab.company.com/group/project.git" +``` + +### GitLab Setup + +1. **Create Test Repository**: + - Public or private repository on GitLab.com + - You must have Developer+ access (to test push operations) + +2. **Create Personal Access Token**: + - Required scopes: `api`, `read_api`, `read_user`, `write_repository` + - See: [GitLab PAT Setup Guide](../../../docs/gitlab-token-setup.md) + +## Running Tests + +### Run All Integration Tests + +```bash +cd components/backend + +# Set environment variables +export INTEGRATION_TESTS=true +export GITLAB_TEST_TOKEN="glpat-..." +export GITLAB_TEST_REPO_URL="https://gitlab.com/user/repo.git" + +# Run tests +go test -v ./tests/integration/gitlab/... +``` + +### Run Specific Test + +```bash +# Run only end-to-end test +go test -v ./tests/integration/gitlab -run TestGitLabIntegrationEnd2End + +# Run only self-hosted tests +go test -v ./tests/integration/gitlab -run TestGitLabSelfHostedIntegration + +# Run only provider detection tests (no GitLab access needed) +go test -v ./tests/integration/gitlab -run TestGitLabProviderDetection +``` + +### Run with Verbose Output + +```bash +go test -v -count=1 ./tests/integration/gitlab/... +``` + +### Skip Integration Tests + +Integration tests are automatically skipped unless `INTEGRATION_TESTS=true` is set: + +```bash +# This will skip integration tests +go test ./tests/integration/gitlab/... +``` + +## Test Coverage + +### TestGitLabIntegrationEnd2End + +**Phases**: +1. **Phase 1: Connect GitLab Account** + - Token validation + - Token storage in Kubernetes Secret + - Connection metadata storage + +2. **Phase 2: Repository Configuration** + - Provider detection + - URL normalization + - Repository validation + - Repository info extraction + +3. **Phase 3: Git Operations** + - Token retrieval for git operations + - Token injection into URLs + - Branch URL construction + +4. **Phase 4: Error Handling** + - Invalid token detection + - Push error parsing and user-friendly messages + +5. **Phase 5: Token Security** + - Token redaction in logs + - URL redaction + +6. **Phase 6: Cleanup** + - Token deletion + - Connection deletion + +### TestGitLabSelfHostedIntegration + +Tests self-hosted GitLab functionality: +- Instance validation +- Self-hosted detection +- API URL construction for custom domains + +### TestGitLabProviderDetection + +Tests provider detection for various URL formats (no GitLab access required): +- GitLab.com HTTPS and SSH URLs +- Self-hosted HTTPS and SSH URLs +- GitHub URLs (to verify no false positives) + +### TestGitLabURLNormalization + +Tests URL normalization (no GitLab access required): +- HTTPS URLs with/without .git suffix +- SSH to HTTPS conversion +- Self-hosted URL handling + +### Benchmarks + +- `BenchmarkGitLabTokenValidation`: Token validation performance +- `BenchmarkProviderDetection`: Provider detection performance + +## Expected Results + +### Success Criteria + +All tests should pass with valid GitLab credentials: + +``` +=== RUN TestGitLabIntegrationEnd2End +=== RUN TestGitLabIntegrationEnd2End/Phase_1:_Connect_GitLab_Account +=== RUN TestGitLabIntegrationEnd2End/Phase_1:_Connect_GitLab_Account/Validate_GitLab_Token +=== RUN TestGitLabIntegrationEnd2End/Phase_1:_Connect_GitLab_Account/Store_GitLab_Token_in_Kubernetes_Secret +=== RUN TestGitLabIntegrationEnd2End/Phase_1:_Connect_GitLab_Account/Store_GitLab_Connection_Metadata +=== RUN TestGitLabIntegrationEnd2End/Phase_2:_Repository_Configuration +... (more tests) +--- PASS: TestGitLabIntegrationEnd2End (2.34s) +PASS +``` + +### Performance Expectations + +- Token validation: < 200ms (per SC-002 from spec) +- Provider detection: < 1ms +- URL normalization: < 1ms + +Run benchmarks to verify: +```bash +go test -bench=. ./tests/integration/gitlab/ +``` + +## Troubleshooting + +### "Skipping integration test" + +**Cause**: `INTEGRATION_TESTS` environment variable not set + +**Solution**: +```bash +export INTEGRATION_TESTS=true +``` + +### "GITLAB_TEST_TOKEN and GITLAB_TEST_REPO_URL must be set" + +**Cause**: Required environment variables missing + +**Solution**: +```bash +export GITLAB_TEST_TOKEN="glpat-your-token-here" +export GITLAB_TEST_REPO_URL="https://gitlab.com/user/repo.git" +``` + +### "Token validation should succeed" fails + +**Possible Causes**: +1. Token expired +2. Token invalid or revoked +3. Token missing required scopes +4. Network connectivity issues + +**Debug**: +```bash +# Test token manually +curl -H "Authorization: Bearer $GITLAB_TEST_TOKEN" \ + https://gitlab.com/api/v4/user +``` + +### "Repository validation should succeed" fails + +**Possible Causes**: +1. Repository URL incorrect +2. Repository doesn't exist +3. You don't have access to repository +4. Token lacks `write_repository` scope + +**Debug**: +```bash +# Test repository access manually +curl -H "Authorization: Bearer $GITLAB_TEST_TOKEN" \ + "https://gitlab.com/api/v4/projects/$(echo $GITLAB_TEST_REPO_URL | sed 's|https://gitlab.com/||' | sed 's|.git$||' | sed 's|/|%2F|')" +``` + +## CI/CD Integration + +### GitHub Actions Example + +```yaml +name: Integration Tests + +on: [push, pull_request] + +jobs: + gitlab-integration: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + - name: Set up Go + uses: actions/setup-go@v4 + with: + go-version: '1.24' + + - name: Run GitLab Integration Tests + env: + INTEGRATION_TESTS: true + GITLAB_TEST_TOKEN: ${{ secrets.GITLAB_TEST_TOKEN }} + GITLAB_TEST_REPO_URL: ${{ secrets.GITLAB_TEST_REPO_URL }} + run: | + cd components/backend + go test -v ./tests/integration/gitlab/... +``` + +### GitLab CI Example + +```yaml +integration-tests: + stage: test + image: golang:1.24 + variables: + INTEGRATION_TESTS: "true" + GITLAB_TEST_TOKEN: $GITLAB_TEST_TOKEN + GITLAB_TEST_REPO_URL: $GITLAB_TEST_REPO_URL + script: + - cd components/backend + - go test -v ./tests/integration/gitlab/... +``` + +## Security Notes + +### Token Safety + +- **Never commit tokens to git** +- Use environment variables or CI/CD secrets +- Rotate test tokens regularly +- Use separate token for testing (not production) + +### Test Repository + +- Use a dedicated test repository +- Don't use production repositories +- Can be public or private (tests work for both) +- Should be a repository you control + +## Additional Tests + +For comprehensive testing, also run: + +### Unit Tests + +```bash +cd components/backend +go test ./gitlab/... -v +go test ./types/... -v +go test ./handlers/... -v +``` + +### Regression Tests + +Verify GitHub functionality still works: + +```bash +cd components/backend +go test ./tests/integration/github/... -v +``` + +## References + +- [GitLab Integration Guide](../../../docs/gitlab-integration.md) +- [GitLab API Documentation](https://docs.gitlab.com/ee/api/) +- [Testing Best Practices](https://go.dev/doc/tutorial/add-a-test) diff --git a/components/backend/tests/integration/gitlab/gitlab_integration_test.go b/components/backend/tests/integration/gitlab/gitlab_integration_test.go new file mode 100644 index 000000000..3a26fec26 --- /dev/null +++ b/components/backend/tests/integration/gitlab/gitlab_integration_test.go @@ -0,0 +1,396 @@ +package gitlab_test + +import ( + "context" + "os" + "testing" + "time" + + "ambient-code-backend/git" + "ambient-code-backend/gitlab" + "ambient-code-backend/handlers" + "ambient-code-backend/k8s" + "ambient-code-backend/types" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/fake" +) + +// TestGitLabIntegrationEnd2End tests the complete GitLab integration workflow +// This test validates the full user journey from connecting GitLab to pushing code +func TestGitLabIntegrationEnd2End(t *testing.T) { + // Skip if not in integration test mode + if os.Getenv("INTEGRATION_TESTS") != "true" { + t.Skip("Skipping integration test. Set INTEGRATION_TESTS=true to run") + } + + // Require GitLab credentials from environment + gitlabToken := os.Getenv("GITLAB_TEST_TOKEN") + gitlabURL := os.Getenv("GITLAB_TEST_REPO_URL") + + if gitlabToken == "" || gitlabURL == "" { + t.Skip("Skipping GitLab integration test: GITLAB_TEST_TOKEN and GITLAB_TEST_REPO_URL must be set") + } + + ctx := context.Background() + testNamespace := "vteam-backend-test" + testUserID := "test-user-123" + + // Create fake Kubernetes client + clientset := fake.NewSimpleClientset() + + t.Run("Phase 1: Connect GitLab Account", func(t *testing.T) { + // Test token validation + t.Run("Validate GitLab Token", func(t *testing.T) { + result, err := gitlab.ValidateGitLabToken(ctx, gitlabToken, "https://gitlab.com") + require.NoError(t, err, "Token validation should succeed") + assert.True(t, result.Valid, "Token should be valid") + assert.NotEmpty(t, result.Username, "Username should be populated") + assert.NotEmpty(t, result.GitLabUserID, "GitLab user ID should be populated") + }) + + // Test token storage + t.Run("Store GitLab Token in Kubernetes Secret", func(t *testing.T) { + err := k8s.StoreGitLabToken(ctx, clientset, testNamespace, testUserID, gitlabToken) + require.NoError(t, err, "Token storage should succeed") + + // Verify token stored + secret, err := clientset.CoreV1().Secrets(testNamespace).Get(ctx, "gitlab-user-tokens", metav1.GetOptions{}) + require.NoError(t, err, "Secret should be created") + assert.Contains(t, secret.Data, testUserID, "Secret should contain user's token") + + // Verify token can be retrieved + retrievedToken, err := k8s.GetGitLabToken(ctx, clientset, testNamespace, testUserID) + require.NoError(t, err, "Token retrieval should succeed") + assert.Equal(t, gitlabToken, retrievedToken, "Retrieved token should match stored token") + }) + + // Test connection management + t.Run("Store GitLab Connection Metadata", func(t *testing.T) { + connMgr := gitlab.NewConnectionManager(clientset, testNamespace) + + connection, err := connMgr.StoreGitLabConnection(ctx, testUserID, gitlabToken, "https://gitlab.com") + require.NoError(t, err, "Connection storage should succeed") + assert.Equal(t, testUserID, connection.UserID) + assert.NotEmpty(t, connection.Username) + assert.Equal(t, "https://gitlab.com", connection.InstanceURL) + + // Verify connection can be retrieved + retrievedConn, err := connMgr.GetGitLabConnection(ctx, testUserID) + require.NoError(t, err, "Connection retrieval should succeed") + assert.Equal(t, connection.Username, retrievedConn.Username) + assert.Equal(t, connection.GitLabUserID, retrievedConn.GitLabUserID) + }) + }) + + t.Run("Phase 2: Repository Configuration", func(t *testing.T) { + // Test provider detection + t.Run("Detect GitLab Provider from URL", func(t *testing.T) { + provider := types.DetectProvider(gitlabURL) + assert.Equal(t, types.ProviderGitLab, provider, "Provider should be detected as GitLab") + }) + + // Test URL normalization + t.Run("Normalize GitLab URL", func(t *testing.T) { + normalized, err := gitlab.NormalizeGitLabURL(gitlabURL) + require.NoError(t, err, "URL normalization should succeed") + assert.Contains(t, normalized, "https://", "Normalized URL should use HTTPS") + assert.Contains(t, normalized, ".git", "Normalized URL should have .git suffix") + }) + + // Test repository validation + t.Run("Validate GitLab Repository Access", func(t *testing.T) { + err := handlers.ValidateGitLabRepository(ctx, gitlabURL, gitlabToken) + require.NoError(t, err, "Repository validation should succeed") + }) + + // Test repository info extraction + t.Run("Extract Repository Information", func(t *testing.T) { + info, err := handlers.GetRepositoryInfo(gitlabURL) + require.NoError(t, err, "Repository info extraction should succeed") + assert.Equal(t, types.ProviderGitLab, info.Provider) + assert.NotEmpty(t, info.Owner, "Owner should be extracted") + assert.NotEmpty(t, info.Repo, "Repo name should be extracted") + assert.Equal(t, "https://gitlab.com/api/v4", info.APIURL, "API URL should be constructed correctly") + }) + }) + + t.Run("Phase 3: Git Operations", func(t *testing.T) { + // Test token retrieval for git operations + t.Run("Retrieve GitLab Token for Git Operations", func(t *testing.T) { + token, err := git.GetGitLabToken(ctx, clientset, "test-project", testUserID) + require.NoError(t, err, "Token retrieval should succeed") + assert.Equal(t, gitlabToken, token, "Retrieved token should match") + }) + + // Test token injection + t.Run("Inject Token into GitLab URL", func(t *testing.T) { + authenticatedURL, err := git.InjectGitLabToken(gitlabURL, gitlabToken) + require.NoError(t, err, "Token injection should succeed") + assert.Contains(t, authenticatedURL, "oauth2:", "URL should contain oauth2 authentication") + assert.NotContains(t, authenticatedURL, gitlabToken, "Raw token should not be visible in URL") + }) + + // Test branch URL construction + t.Run("Construct GitLab Branch URL", func(t *testing.T) { + branchURL, err := git.ConstructGitLabBranchURL(gitlabURL, "main") + require.NoError(t, err, "Branch URL construction should succeed") + assert.Contains(t, branchURL, "/-/tree/main", "Branch URL should have GitLab tree format") + }) + }) + + t.Run("Phase 4: Error Handling", func(t *testing.T) { + // Test invalid token detection + t.Run("Detect Invalid Token", func(t *testing.T) { + invalidToken := "glpat-invalid-token-123" + _, err := gitlab.ValidateGitLabToken(ctx, invalidToken, "https://gitlab.com") + assert.Error(t, err, "Invalid token should fail validation") + }) + + // Test push error detection + t.Run("Parse GitLab Push Errors", func(t *testing.T) { + // Test 403 Forbidden + err := git.DetectPushError(gitlabURL, "remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled", "") + assert.Error(t, err) + assert.Contains(t, err.Error(), "Insufficient permissions", "Should detect permission error") + assert.Contains(t, err.Error(), "write_repository", "Should mention required scope") + + // Test 401 Unauthorized + err = git.DetectPushError(gitlabURL, "fatal: Authentication failed", "") + assert.Error(t, err) + assert.Contains(t, err.Error(), "Authentication failed", "Should detect auth error") + }) + }) + + t.Run("Phase 5: Token Security", func(t *testing.T) { + // Test token redaction in logs + t.Run("Redact Tokens in Log Messages", func(t *testing.T) { + logMsg := "Cloning https://oauth2:" + gitlabToken + "@gitlab.com/owner/repo.git" + redacted := gitlab.RedactToken(logMsg) + assert.NotContains(t, redacted, gitlabToken, "Token should be redacted") + assert.Contains(t, redacted, gitlab.TokenRedactionPlaceholder, "Should contain redaction placeholder") + }) + + // Test URL redaction + t.Run("Redact URLs with Tokens", func(t *testing.T) { + urlWithToken := "https://oauth2:" + gitlabToken + "@gitlab.com/owner/repo.git" + redactedURL := gitlab.RedactURL(urlWithToken) + assert.NotContains(t, redactedURL, gitlabToken, "Token should be redacted from URL") + assert.Contains(t, redactedURL, gitlab.TokenRedactionPlaceholder, "Should contain redaction placeholder") + }) + }) + + t.Run("Phase 6: Cleanup", func(t *testing.T) { + // Test token deletion + t.Run("Delete GitLab Token", func(t *testing.T) { + err := k8s.DeleteGitLabToken(ctx, clientset, testNamespace, testUserID) + require.NoError(t, err, "Token deletion should succeed") + + // Verify token deleted + _, err = k8s.GetGitLabToken(ctx, clientset, testNamespace, testUserID) + assert.Error(t, err, "Token should not exist after deletion") + }) + + // Test connection deletion + t.Run("Delete GitLab Connection", func(t *testing.T) { + connMgr := gitlab.NewConnectionManager(clientset, testNamespace) + err := connMgr.DeleteGitLabConnection(ctx, testUserID) + require.NoError(t, err, "Connection deletion should succeed") + + // Verify connection deleted + conn, err := connMgr.GetGitLabConnection(ctx, testUserID) + assert.Error(t, err, "Connection should not exist after deletion") + assert.Nil(t, conn, "Connection should be nil") + }) + }) +} + +// TestGitLabSelfHostedIntegration tests self-hosted GitLab instance integration +func TestGitLabSelfHostedIntegration(t *testing.T) { + if os.Getenv("INTEGRATION_TESTS") != "true" { + t.Skip("Skipping integration test. Set INTEGRATION_TESTS=true to run") + } + + // Require self-hosted GitLab credentials + token := os.Getenv("GITLAB_SELFHOSTED_TOKEN") + instanceURL := os.Getenv("GITLAB_SELFHOSTED_URL") + repoURL := os.Getenv("GITLAB_SELFHOSTED_REPO_URL") + + if token == "" || instanceURL == "" || repoURL == "" { + t.Skip("Skipping self-hosted GitLab test: GITLAB_SELFHOSTED_TOKEN, GITLAB_SELFHOSTED_URL, and GITLAB_SELFHOSTED_REPO_URL must be set") + } + + ctx := context.Background() + + t.Run("Validate Self-Hosted Instance", func(t *testing.T) { + result, err := gitlab.ValidateGitLabToken(ctx, token, instanceURL) + require.NoError(t, err, "Self-hosted token validation should succeed") + assert.True(t, result.Valid, "Token should be valid") + }) + + t.Run("Detect Self-Hosted Instance", func(t *testing.T) { + parsed, err := gitlab.ParseGitLabURL(repoURL) + require.NoError(t, err, "URL parsing should succeed") + + isSeflHosted := gitlab.IsGitLabSelfHosted(parsed.Host) + assert.True(t, isSeflHosted, "Should detect as self-hosted instance") + }) + + t.Run("Construct Self-Hosted API URL", func(t *testing.T) { + parsed, err := gitlab.ParseGitLabURL(repoURL) + require.NoError(t, err) + + apiURL := gitlab.ConstructAPIURL(parsed.Host) + assert.Contains(t, apiURL, parsed.Host, "API URL should contain instance host") + assert.Contains(t, apiURL, "/api/v4", "API URL should have /api/v4 path") + }) +} + +// TestGitLabProviderDetection tests provider detection for various URL formats +func TestGitLabProviderDetection(t *testing.T) { + testCases := []struct { + name string + url string + expected types.ProviderType + }{ + { + name: "GitLab.com HTTPS", + url: "https://gitlab.com/owner/repo.git", + expected: types.ProviderGitLab, + }, + { + name: "GitLab.com HTTPS without .git", + url: "https://gitlab.com/owner/repo", + expected: types.ProviderGitLab, + }, + { + name: "GitLab.com SSH", + url: "git@gitlab.com:owner/repo.git", + expected: types.ProviderGitLab, + }, + { + name: "Self-hosted GitLab HTTPS", + url: "https://gitlab.company.com/group/project.git", + expected: types.ProviderGitLab, + }, + { + name: "Self-hosted GitLab SSH", + url: "git@gitlab.company.com:group/project.git", + expected: types.ProviderGitLab, + }, + { + name: "GitHub.com HTTPS", + url: "https://github.com/owner/repo.git", + expected: types.ProviderGitHub, + }, + { + name: "GitHub.com SSH", + url: "git@github.com:owner/repo.git", + expected: types.ProviderGitHub, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + detected := types.DetectProvider(tc.url) + assert.Equal(t, tc.expected, detected, "Provider detection failed for %s", tc.url) + }) + } +} + +// TestGitLabURLNormalization tests URL normalization for various formats +func TestGitLabURLNormalization(t *testing.T) { + testCases := []struct { + name string + input string + expected string + shouldError bool + }{ + { + name: "HTTPS with .git", + input: "https://gitlab.com/owner/repo.git", + expected: "https://gitlab.com/owner/repo.git", + }, + { + name: "HTTPS without .git", + input: "https://gitlab.com/owner/repo", + expected: "https://gitlab.com/owner/repo.git", + }, + { + name: "SSH format", + input: "git@gitlab.com:owner/repo.git", + expected: "https://gitlab.com/owner/repo.git", + }, + { + name: "Self-hosted HTTPS", + input: "https://gitlab.company.com/group/project", + expected: "https://gitlab.company.com/group/project.git", + }, + { + name: "Self-hosted SSH", + input: "git@gitlab.company.com:group/project.git", + expected: "https://gitlab.company.com/group/project.git", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + normalized, err := gitlab.NormalizeGitLabURL(tc.input) + + if tc.shouldError { + assert.Error(t, err) + } else { + require.NoError(t, err) + assert.Equal(t, tc.expected, normalized) + } + }) + } +} + +// TestGitLabClientLogging tests that API calls are properly logged with request IDs +func TestGitLabClientLogging(t *testing.T) { + // Skip if no real GitLab access + token := os.Getenv("GITLAB_TEST_TOKEN") + if token == "" { + t.Skip("Skipping: GITLAB_TEST_TOKEN not set") + } + + ctx := context.Background() + client := gitlab.NewClient("https://gitlab.com/api/v4", token) + + // Make a simple API request + resp, err := client.GetCurrentUser(ctx) + require.NoError(t, err, "API call should succeed") + assert.NotNil(t, resp, "Response should not be nil") + + // Note: Actual log verification would require capturing log output + // This test validates the happy path executes without errors +} + +// BenchmarkGitLabTokenValidation benchmarks token validation performance +func BenchmarkGitLabTokenValidation(b *testing.B) { + token := os.Getenv("GITLAB_TEST_TOKEN") + if token == "" { + b.Skip("Skipping: GITLAB_TEST_TOKEN not set") + } + + ctx := context.Background() + + b.ResetTimer() + for i := 0; i < b.N; i++ { + _, _ = gitlab.ValidateGitLabToken(ctx, token, "https://gitlab.com") + } +} + +// BenchmarkProviderDetection benchmarks provider detection performance +func BenchmarkProviderDetection(b *testing.B) { + url := "https://gitlab.com/owner/repo.git" + + b.ResetTimer() + for i := 0; i < b.N; i++ { + _ = types.DetectProvider(url) + } +} diff --git a/components/backend/tests/regression/backward_compat_test.go b/components/backend/tests/regression/backward_compat_test.go new file mode 100644 index 000000000..1e0bd6485 --- /dev/null +++ b/components/backend/tests/regression/backward_compat_test.go @@ -0,0 +1,150 @@ +package regression_test + +import ( + "testing" + + "ambient-code-backend/types" + + "github.com/stretchr/testify/assert" +) + +// TestBackwardCompatibility_ProviderDetection verifies provider detection +// doesn't break GitHub URLs or return incorrect values for existing repos +func TestBackwardCompatibility_ProviderDetection(t *testing.T) { + testCases := []struct { + name string + url string + expected types.ProviderType + }{ + // GitHub URLs should still be detected correctly + { + name: "GitHub HTTPS", + url: "https://github.com/owner/repo.git", + expected: types.ProviderGitHub, + }, + { + name: "GitHub HTTPS without .git", + url: "https://github.com/owner/repo", + expected: types.ProviderGitHub, + }, + { + name: "GitHub SSH", + url: "git@github.com:owner/repo.git", + expected: types.ProviderGitHub, + }, + { + name: "GitHub Enterprise", + url: "https://github.company.com/owner/repo.git", + expected: types.ProviderGitHub, + }, + + // New GitLab URLs should be detected + { + name: "GitLab HTTPS", + url: "https://gitlab.com/owner/repo.git", + expected: types.ProviderGitLab, + }, + { + name: "GitLab SSH", + url: "git@gitlab.com:owner/repo.git", + expected: types.ProviderGitLab, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + detected := types.DetectProvider(tc.url) + assert.Equal(t, tc.expected, detected, + "Provider detection changed for %s - this breaks backward compatibility!", tc.url) + }) + } +} + +// TestBackwardCompatibility_ProviderEnumValues ensures provider type values +// haven't changed (would break existing database/CRD records) +func TestBackwardCompatibility_ProviderEnumValues(t *testing.T) { + // These values must never change - they're stored in Kubernetes CRDs + assert.Equal(t, types.ProviderType("github"), types.ProviderGitHub, + "GitHub provider enum value changed - breaks existing CRDs!") + assert.Equal(t, types.ProviderType("gitlab"), types.ProviderGitLab, + "GitLab provider enum value changed - breaks existing CRDs!") +} + +// TestBackwardCompatibility_EmptyProvider ensures empty provider field +// doesn't cause errors (existing ProjectSettings may not have provider) +func TestBackwardCompatibility_EmptyProvider(t *testing.T) { + // Simulate existing ProjectSettings without provider field + emptyProvider := types.ProviderType("") + + // Should not panic or error + assert.NotPanics(t, func() { + _ = string(emptyProvider) + }, "Empty provider should not cause panic") + + // Empty provider should not equal valid providers + assert.NotEqual(t, types.ProviderGitHub, emptyProvider) + assert.NotEqual(t, types.ProviderGitLab, emptyProvider) +} + +// TestBackwardCompatibility_GitHubOperationsUnchanged verifies that +// GitHub-specific functionality still works exactly as before +func TestBackwardCompatibility_GitHubOperationsUnchanged(t *testing.T) { + // Test that GitHub URLs are not affected by GitLab code + githubURLs := []string{ + "https://github.com/user/repo.git", + "git@github.com:user/repo.git", + "https://github.company.com/user/repo.git", + } + + for _, url := range githubURLs { + provider := types.DetectProvider(url) + assert.Equal(t, types.ProviderGitHub, provider, + "GitHub detection broken for URL: %s", url) + } +} + +// TestBackwardCompatibility_NoGitLabFalsePositives ensures GitLab detection +// doesn't incorrectly identify GitHub URLs as GitLab +func TestBackwardCompatibility_NoGitLabFalsePositives(t *testing.T) { + notGitLabURLs := []string{ + "https://github.com/gitlab/repo.git", // Contains "gitlab" but is GitHub + "https://github.com/user/gitlab-cli.git", // Project named gitlab + "git@github.com:company/gitlab-docs.git", // Repo contains gitlab + } + + for _, url := range notGitLabURLs { + provider := types.DetectProvider(url) + assert.NotEqual(t, types.ProviderGitLab, provider, + "False positive GitLab detection for GitHub URL: %s", url) + assert.Equal(t, types.ProviderGitHub, provider, + "Should detect as GitHub: %s", url) + } +} + +// TestBackwardCompatibility_ExistingProjectSettings verifies that existing +// ProjectSettings CRs (without provider field) still work +func TestBackwardCompatibility_ExistingProjectSettings(t *testing.T) { + // Simulate existing ProjectSettings YAML without provider field + // This represents real CRs created before GitLab support was added + + type Repository struct { + URL string `json:"url"` + Branch string `json:"branch,omitempty"` + Provider types.ProviderType `json:"provider,omitempty"` + } + + // Existing repo without provider field (should auto-detect) + existingRepo := Repository{ + URL: "https://github.com/user/repo.git", + Branch: "main", + // Provider field not set (empty string) + } + + // Should be able to detect provider from URL even if field is empty + if existingRepo.Provider == "" { + existingRepo.Provider = types.DetectProvider(existingRepo.URL) + } + + assert.Equal(t, types.ProviderGitHub, existingRepo.Provider, + "Auto-detection should work for existing repos without provider field") +} diff --git a/components/backend/types/common.go b/components/backend/types/common.go index 9fa4ea51c..1d86bb2a1 100644 --- a/components/backend/types/common.go +++ b/components/backend/types/common.go @@ -4,8 +4,9 @@ package types // Common types used across the application type GitRepository struct { - URL string `json:"url"` - Branch *string `json:"branch,omitempty"` + URL string `json:"url"` + Branch *string `json:"branch,omitempty"` + Provider ProviderType `json:"provider,omitempty"` // Optional: auto-detected if not specified } type UserContext struct { @@ -41,7 +42,45 @@ type Paths struct { Inbox string `json:"inbox,omitempty"` } -// BoolPtr returns a pointer to the given bool value. +// Common repository browsing types (used by both GitHub and GitLab) + +// Branch represents a Git branch (common format for UI) +type Branch struct { + Name string `json:"name"` + Protected bool `json:"protected"` + Default bool `json:"default,omitempty"` + Commit CommitInfo `json:"commit,omitempty"` +} + +// CommitInfo represents basic commit information +type CommitInfo struct { + SHA string `json:"sha"` + Message string `json:"message,omitempty"` + Author string `json:"author,omitempty"` + Timestamp string `json:"timestamp,omitempty"` +} + +// TreeEntry represents a file or directory in a repository +type TreeEntry struct { + Name string `json:"name"` + Path string `json:"path"` + Type string `json:"type"` // "blob" (file) or "tree" (directory) + Mode string `json:"mode,omitempty"` + SHA string `json:"sha,omitempty"` + Size int `json:"size,omitempty"` +} + +// FileContent represents file contents from a repository +type FileContent struct { + Name string `json:"name"` + Path string `json:"path"` + Content string `json:"content"` + Encoding string `json:"encoding"` // "base64" or "utf-8" + Size int `json:"size"` + SHA string `json:"sha,omitempty"` +} + +// Helper functions for pointer types func BoolPtr(b bool) *bool { return &b } diff --git a/components/backend/types/errors.go b/components/backend/types/errors.go new file mode 100644 index 000000000..39a3df7ae --- /dev/null +++ b/components/backend/types/errors.go @@ -0,0 +1,94 @@ +package types + +import ( + "fmt" + "strings" +) + +// GetProviderSpecificGuidance returns remediation guidance for provider-specific errors +func GetProviderSpecificGuidance(provider ProviderType, errorType string) string { + switch provider { + case ProviderGitHub: + switch errorType { + case "auth": + return "Ensure your GitHub App is installed and has access to the repository, or configure a GitHub PAT in the project runner secret" + case "permissions": + return "Ensure the GitHub App or PAT has write access to the repository" + case "not_found": + return "Verify the repository URL is correct and you have access to it on GitHub" + default: + return "Check your GitHub repository configuration and try again" + } + case ProviderGitLab: + switch errorType { + case "auth": + return "Connect your GitLab account with a valid Personal Access Token via /auth/gitlab/connect" + case "permissions": + return "Ensure your GitLab PAT has 'api', 'read_repository', and 'write_repository' scopes" + case "not_found": + return "Verify the repository URL is correct and you have access to it on GitLab" + default: + return "Check your GitLab repository configuration and try again" + } + default: + return "Check your repository configuration and try again" + } +} + +// FormatMixedProviderError formats an error message for mixed-provider scenarios +func FormatMixedProviderError(results []ProviderResult) string { + failedProviders := []string{} + successfulProviders := []string{} + + for _, result := range results { + if result.Success { + successfulProviders = append(successfulProviders, string(result.Provider)) + } else { + failedProviders = append(failedProviders, string(result.Provider)) + } + } + + if len(failedProviders) == 0 { + return "All repository operations completed successfully" + } else if len(failedProviders) == len(results) { + return "All repository operations failed. Check your provider configurations and credentials" + } else { + return fmt.Sprintf("Some repository operations failed (%s). Successful: %s", + strings.Join(failedProviders, ", "), + strings.Join(successfulProviders, ", ")) + } +} + +// CreateProviderResult creates a ProviderResult from an operation outcome +func CreateProviderResult(provider ProviderType, repoURL string, err error) ProviderResult { + result := ProviderResult{ + Provider: provider, + RepoURL: repoURL, + } + + if err != nil { + result.Success = false + result.Error = err.Error() + } else { + result.Success = true + } + + return result +} + +// AggregateProviderResults creates a MixedProviderSessionResult from multiple provider results +func AggregateProviderResults(results []ProviderResult) MixedProviderSessionResult { + allSuccess := true + for _, result := range results { + if !result.Success { + allSuccess = false + break + } + } + + return MixedProviderSessionResult{ + OverallSuccess: allSuccess, + Results: results, + Message: FormatMixedProviderError(results), + } +} diff --git a/components/backend/types/gitlab.go b/components/backend/types/gitlab.go new file mode 100644 index 000000000..78dfe3830 --- /dev/null +++ b/components/backend/types/gitlab.go @@ -0,0 +1,68 @@ +package types + +import "time" + +// GitLabConnection represents a user's connection to GitLab (GitLab.com or self-hosted) +type GitLabConnection struct { + UserID string `json:"userId"` // vTeam user identifier + GitLabUserID string `json:"gitlabUserId"` // GitLab user ID (from /user API) + InstanceURL string `json:"instanceUrl"` // e.g., "https://gitlab.com" or "https://gitlab.company.com" + Username string `json:"username"` // GitLab username + UpdatedAt time.Time `json:"updatedAt"` // Last connection update +} + +// GitLabRepository extends GitRepository for GitLab-specific attributes +// Internal parsed representation (not persisted to CRD) +type ParsedGitLabRepo struct { + Host string // "gitlab.com" or "gitlab.example.com" + Owner string // Repository owner/namespace + Repo string // Repository name + APIURL string // Constructed API base URL (e.g., "https://gitlab.com/api/v4") + ProjectID string // URL-encoded project path (owner%2Frepo) for API calls +} + +// GitLabAPIError represents structured error type for GitLab API failures +type GitLabAPIError struct { + StatusCode int `json:"statusCode"` // HTTP status code + Message string `json:"message"` // User-friendly error message + Remediation string `json:"remediation"` // Actionable guidance for user + RawError string `json:"rawError"` // Original error from GitLab API + RequestID string `json:"requestId"` // GitLab request ID for debugging + Metadata map[string]interface{} `json:"metadata"` // Additional context +} + +// Error implements the error interface +func (e *GitLabAPIError) Error() string { + if e.Remediation != "" { + return e.Message + ". " + e.Remediation + } + return e.Message +} + +// GitLabBranch represents a Git branch in a GitLab repository +type GitLabBranch struct { + Name string `json:"name"` + Commit GitLabCommit `json:"commit"` + Protected bool `json:"protected"` + Default bool `json:"default"` +} + +// GitLabCommit represents commit information +type GitLabCommit struct { + ID string `json:"id"` // SHA + ShortID string `json:"short_id"` // Short SHA + Title string `json:"title"` // Commit title + Message string `json:"message"` // Full commit message + AuthorName string `json:"author_name"` // Author name + AuthorEmail string `json:"author_email"` + CommittedDate time.Time `json:"committed_date"` +} + +// GitLabTreeEntry represents a file or directory entry in a GitLab repository tree +type GitLabTreeEntry struct { + ID string `json:"id"` // Object SHA + Name string `json:"name"` // File/directory name + Type string `json:"type"` // "blob" or "tree" + Path string `json:"path"` // Full path from repository root + Mode string `json:"mode"` // File mode (e.g., "100644") +} diff --git a/components/backend/types/provider.go b/components/backend/types/provider.go new file mode 100644 index 000000000..be2dd7d23 --- /dev/null +++ b/components/backend/types/provider.go @@ -0,0 +1,75 @@ +package types + +import ( + "net/url" + "strings" +) + +// ProviderType distinguishes between Git hosting providers +type ProviderType string + +const ( + // ProviderGitHub represents GitHub repositories + ProviderGitHub ProviderType = "github" + // ProviderGitLab represents GitLab repositories + ProviderGitLab ProviderType = "gitlab" +) + +// DetectProvider determines the Git provider from a repository URL +// Uses precise hostname matching to prevent false positives +func DetectProvider(repoURL string) ProviderType { + if repoURL == "" { + return "" + } + + // Normalize SSH URLs (git@host:path) to https://host/path for parsing + normalizedURL := repoURL + if strings.HasPrefix(repoURL, "git@") { + // Convert git@github.com:owner/repo.git to https://github.com/owner/repo.git + normalizedURL = strings.Replace(repoURL, ":", "/", 1) + normalizedURL = strings.Replace(normalizedURL, "git@", "https://", 1) + } + + // Parse the URL to extract hostname + parsedURL, err := url.Parse(normalizedURL) + if err != nil { + // Fallback to basic string matching if URL parsing fails + lowerURL := strings.ToLower(repoURL) + if strings.Contains(lowerURL, "github.com") { + return ProviderGitHub + } + if strings.Contains(lowerURL, "gitlab.com") { + return ProviderGitLab + } + return "" + } + + hostname := strings.ToLower(parsedURL.Hostname()) + if hostname == "" { + return "" + } + + // Check for GitHub (github.com or *.github.com for enterprise) + if hostname == "github.com" || strings.HasSuffix(hostname, ".github.com") { + return ProviderGitHub + } + + // Check for GitLab (gitlab.com or any hostname containing "gitlab" for self-hosted) + // GitLab self-hosted instances typically use gitlab.company.com or gitlab-ce.company.com + if hostname == "gitlab.com" || strings.Contains(hostname, "gitlab") { + return ProviderGitLab + } + + // Unknown provider + return "" +} + +// String returns the string representation of the provider type +func (p ProviderType) String() string { + return string(p) +} + +// IsValid checks if the provider type is valid +func (p ProviderType) IsValid() bool { + return p == ProviderGitHub || p == ProviderGitLab +} diff --git a/components/backend/types/session.go b/components/backend/types/session.go index 144140752..9126ed390 100644 --- a/components/backend/types/session.go +++ b/components/backend/types/session.go @@ -107,3 +107,20 @@ type WorkflowSelection struct { Branch string `json:"branch,omitempty"` Path string `json:"path,omitempty"` } + +// Mixed Provider Support Types + +// ProviderResult contains the result of operations for a specific provider +type ProviderResult struct { + Provider ProviderType `json:"provider"` + Success bool `json:"success"` + Error string `json:"error,omitempty"` + RepoURL string `json:"repoUrl"` +} + +// MixedProviderSessionResult contains results from multiple providers +type MixedProviderSessionResult struct { + OverallSuccess bool `json:"overallSuccess"` + Results []ProviderResult `json:"results"` + Message string `json:"message"` +} diff --git a/components/frontend/src/components/workspace-sections/settings-section.tsx b/components/frontend/src/components/workspace-sections/settings-section.tsx index c58c84236..607da5ef8 100644 --- a/components/frontend/src/components/workspace-sections/settings-section.tsx +++ b/components/frontend/src/components/workspace-sections/settings-section.tsx @@ -35,10 +35,14 @@ export function SettingsSection({ projectName }: SettingsSectionProps) { const [jiraEmail, setJiraEmail] = useState(""); const [jiraToken, setJiraToken] = useState(""); const [showJiraToken, setShowJiraToken] = useState(false); + const [gitlabToken, setGitlabToken] = useState(""); + const [gitlabInstanceUrl, setGitlabInstanceUrl] = useState(""); + const [showGitlabToken, setShowGitlabToken] = useState(false); const [anthropicExpanded, setAnthropicExpanded] = useState(false); const [githubExpanded, setGithubExpanded] = useState(false); const [jiraExpanded, setJiraExpanded] = useState(false); - const FIXED_KEYS = useMemo(() => ["ANTHROPIC_API_KEY","GIT_USER_NAME","GIT_USER_EMAIL","GITHUB_TOKEN","JIRA_URL","JIRA_PROJECT","JIRA_EMAIL","JIRA_API_TOKEN"] as const, []); + const [gitlabExpanded, setGitlabExpanded] = useState(false); + const FIXED_KEYS = useMemo(() => ["ANTHROPIC_API_KEY","GIT_USER_NAME","GIT_USER_EMAIL","GITHUB_TOKEN","JIRA_URL","JIRA_PROJECT","JIRA_EMAIL","JIRA_API_TOKEN","GITLAB_TOKEN","GITLAB_INSTANCE_URL"] as const, []); // React Query hooks const { data: project, isLoading: projectLoading } = useProject(projectName); @@ -69,6 +73,8 @@ export function SettingsSection({ projectName }: SettingsSectionProps) { setJiraProject(byKey["JIRA_PROJECT"] || ""); setJiraEmail(byKey["JIRA_EMAIL"] || ""); setJiraToken(byKey["JIRA_API_TOKEN"] || ""); + setGitlabToken(byKey["GITLAB_TOKEN"] || ""); + setGitlabInstanceUrl(byKey["GITLAB_INSTANCE_URL"] || ""); setSecrets(allSecrets.filter(s => !FIXED_KEYS.includes(s.key as typeof FIXED_KEYS[number]))); } }, [runnerSecrets, integrationSecrets, FIXED_KEYS]); @@ -139,6 +145,8 @@ export function SettingsSection({ projectName }: SettingsSectionProps) { if (jiraProject) integrationData["JIRA_PROJECT"] = jiraProject; if (jiraEmail) integrationData["JIRA_EMAIL"] = jiraEmail; if (jiraToken) integrationData["JIRA_API_TOKEN"] = jiraToken; + if (gitlabToken) integrationData["GITLAB_TOKEN"] = gitlabToken; + if (gitlabInstanceUrl) integrationData["GITLAB_INSTANCE_URL"] = gitlabInstanceUrl; for (const { key, value } of secrets) { if (!key) continue; if (FIXED_KEYS.includes(key as typeof FIXED_KEYS[number])) continue; @@ -412,6 +420,54 @@ export function SettingsSection({ projectName }: SettingsSectionProps) { )} + {/* GitLab Integration Section */} +
+ + {gitlabExpanded && ( +
+
Configure GitLab credentials for repository operations (clone, commit, push)
+
+ +
GitLab personal access token (glpat-...) with api, read_api, read_user, write_repository scopes
+
+ setGitlabToken(e.target.value)} + className="flex-1" + /> + +
+
+
+ +
GitLab instance URL (leave empty for gitlab.com, or use https://gitlab.company.com for self-hosted)
+ setGitlabInstanceUrl(e.target.value)} + /> +
+
+ )} +
+ {/* Custom Environment Variables Section */}
diff --git a/components/manifests/base/crds/projectsettings-crd.yaml b/components/manifests/base/crds/projectsettings-crd.yaml index 28ed18d03..f14fc1219 100644 --- a/components/manifests/base/crds/projectsettings-crd.yaml +++ b/components/manifests/base/crds/projectsettings-crd.yaml @@ -42,6 +42,26 @@ spec: runnerSecretsName: type: string description: "Name of the Kubernetes Secret in this namespace that stores runner configuration key/value pairs" + repositories: + type: array + description: "Git repositories configured for this project" + items: + type: object + required: + - url + properties: + url: + type: string + description: "Repository URL (HTTPS or SSH format)" + branch: + type: string + description: "Optional branch override (defaults to repository's default branch)" + provider: + type: string + enum: + - "github" + - "gitlab" + description: "Git hosting provider (auto-detected from URL if not specified)" status: type: object properties: diff --git a/components/runners/claude-code-runner/wrapper.py b/components/runners/claude-code-runner/wrapper.py index 3e87328c4..118fe7efd 100644 --- a/components/runners/claude-code-runner/wrapper.py +++ b/components/runners/claude-code-runner/wrapper.py @@ -70,7 +70,6 @@ async def run(self): except Exception as _: logging.debug("CR status update (Running) skipped") - # Append token to websocket URL if available (to pass SA token to backend) try: if self.shell and getattr(self.shell, 'transport', None): @@ -86,7 +85,7 @@ async def run(self): result = None while True: result = await self._run_claude_agent_sdk(prompt) - + # Check if restart was requested (workflow changed) if self._restart_requested: self._restart_requested = False @@ -94,7 +93,7 @@ async def run(self): logging.info("Restarting Claude SDK due to workflow change") # Loop will call _run_claude_agent_sdk again with updated env vars continue - + # Normal exit - no restart requested break @@ -224,7 +223,7 @@ async def _run_claude_agent_sdk(self, prompt: str): cwd_path = self.context.workspace_path add_dirs = [] derived_name = None # Track workflow name for system prompt - + # Check for active workflow first active_workflow_url = (os.getenv('ACTIVE_WORKFLOW_GIT_URL') or '').strip() if active_workflow_url: @@ -233,18 +232,17 @@ async def _run_claude_agent_sdk(self, prompt: str): owner, repo, _ = self._parse_owner_repo(active_workflow_url) derived_name = repo or '' if not derived_name: - from urllib.parse import urlparse as _urlparse - p = _urlparse(active_workflow_url) + p = urlparse(active_workflow_url) parts = [p for p in (p.path or '').split('/') if p] if parts: derived_name = parts[-1] derived_name = (derived_name or '').removesuffix('.git').strip() - + if derived_name: workflow_path = str(Path(self.context.workspace_path) / "workflows" / derived_name) - # NOTE: Don't append ACTIVE_WORKFLOW_PATH here - we already extracted + # NOTE: Don't append ACTIVE_WORKFLOW_PATH here - we already extracted # the subdirectory during clone, so workflow_path is the final location - + if Path(workflow_path).exists(): cwd_path = workflow_path logging.info(f"Using workflow as CWD: {derived_name}") @@ -256,7 +254,7 @@ async def _run_claude_agent_sdk(self, prompt: str): except Exception as e: logging.warning(f"Failed to derive workflow name: {e}, using default") cwd_path = str(Path(self.context.workspace_path) / "workflows" / "default") - + # Add all repos as additional directories so they're accessible to Claude for r in repos_cfg: name = (r.get('name') or '').strip() @@ -265,7 +263,7 @@ async def _run_claude_agent_sdk(self, prompt: str): if repo_path not in add_dirs: add_dirs.append(repo_path) logging.info(f"Added repo as additional directory: {name}") - + # Add artifacts directory artifacts_path = str(Path(self.context.workspace_path) / "artifacts") if artifacts_path not in add_dirs: @@ -294,7 +292,7 @@ async def _run_claude_agent_sdk(self, prompt: str): p = str(Path(self.context.workspace_path) / name) if p != cwd_path: add_dirs.append(p) - + # Add artifacts directory for repos mode too artifacts_path = str(Path(self.context.workspace_path) / "artifacts") if artifacts_path not in add_dirs: @@ -351,7 +349,7 @@ async def _run_claude_agent_sdk(self, prompt: str): options = ClaudeAgentOptions( cwd=cwd_path, permission_mode="acceptEdits", - allowed_tools= allowed_tools, + allowed_tools=allowed_tools, mcp_servers=mcp_servers, setting_sources=["project"], system_prompt=system_prompt_config @@ -536,7 +534,7 @@ async def process_one_prompt(text: str): logging.info("No initial prompt - Claude will greet based on system prompt") else: logging.info("Skipping prompts - SDK resuming with full context") - + # Mark that first run is complete self._first_run = False @@ -665,7 +663,6 @@ async def _setup_vertex_credentials(self) -> dict: async def _prepare_workspace(self): """Clone input repo/branch into workspace and configure git remotes.""" - token = os.getenv("GITHUB_TOKEN") or await self._fetch_github_token() workspace = Path(self.context.workspace_path) workspace.mkdir(parents=True, exist_ok=True) @@ -691,6 +688,9 @@ async def _prepare_workspace(self): continue repo_dir = workspace / name + # Fetch appropriate token for this repo's URL + token = await self._fetch_token_for_url(url) + # Check if repo already exists repo_exists = repo_dir.exists() and (repo_dir / ".git").exists() @@ -747,6 +747,9 @@ async def _prepare_workspace(self): input_branch = os.getenv("INPUT_BRANCH", "").strip() or "main" output_repo = os.getenv("OUTPUT_REPO_URL", "").strip() + # Fetch appropriate token for this repo's URL + token = await self._fetch_token_for_url(input_repo) + workspace_has_git = (workspace / ".git").exists() logging.info(f"Single-repo setup: workspace_has_git={workspace_has_git}, reusing={reusing_workspace}") @@ -857,62 +860,63 @@ async def _initialize_workflow_if_set(self): active_workflow_url = (os.getenv('ACTIVE_WORKFLOW_GIT_URL') or '').strip() if not active_workflow_url: return # No workflow to initialize - + active_workflow_branch = (os.getenv('ACTIVE_WORKFLOW_BRANCH') or 'main').strip() active_workflow_path = (os.getenv('ACTIVE_WORKFLOW_PATH') or '').strip() - + # Derive workflow name from URL try: owner, repo, _ = self._parse_owner_repo(active_workflow_url) derived_name = repo or '' if not derived_name: - from urllib.parse import urlparse as _urlparse - p = _urlparse(active_workflow_url) + p = urlparse(active_workflow_url) parts = [p for p in (p.path or '').split('/') if p] if parts: derived_name = parts[-1] derived_name = (derived_name or '').removesuffix('.git').strip() - + if not derived_name: logging.warning("Could not derive workflow name from URL, skipping initialization") return - + workflow_dir = Path(self.context.workspace_path) / "workflows" / derived_name - + # Only clone if workflow directory doesn't exist if workflow_dir.exists(): logging.info(f"Workflow {derived_name} already exists, skipping initialization") return - + logging.info(f"Initializing workflow {derived_name} from CR spec on startup") # Clone the workflow but don't request restart (we haven't started yet) await self._clone_workflow_repository(active_workflow_url, active_workflow_branch, active_workflow_path, derived_name) - + except Exception as e: logging.error(f"Failed to initialize workflow on startup: {e}") # Don't fail the session if workflow init fails - continue without it - + async def _clone_workflow_repository(self, git_url: str, branch: str, path: str, workflow_name: str): """Clone workflow repository without requesting restart (used during initialization).""" workspace = Path(self.context.workspace_path) - token = os.getenv("GITHUB_TOKEN") or await self._fetch_github_token() - + workflow_dir = workspace / "workflows" / workflow_name temp_clone_dir = workspace / "workflows" / f"{workflow_name}-clone-temp" - + # Check if workflow already exists if workflow_dir.exists(): await self._send_log(f"✓ Workflow {workflow_name} already loaded") logging.info(f"Workflow {workflow_name} already exists at {workflow_dir}") return - + + # Fetch appropriate token based on repo URL + token = await self._fetch_token_for_url(git_url) + # Clone to temporary directory first await self._send_log(f"📥 Cloning workflow {workflow_name}...") logging.info(f"Cloning workflow from {git_url} (branch: {branch})") clone_url = self._url_with_token(git_url, token) if token else git_url await self._run_cmd(["git", "clone", "--branch", branch, "--single-branch", clone_url, str(temp_clone_dir)], cwd=str(workspace)) logging.info(f"Successfully cloned workflow to temp directory") - + # Extract subdirectory if path is specified if path and path.strip(): subdir_path = temp_clone_dir / path.strip() @@ -931,7 +935,7 @@ async def _clone_workflow_repository(self, git_url: str, branch: str, path: str, # No path specified, use entire repo temp_clone_dir.rename(workflow_dir) logging.info(f"Using entire repository as workflow") - + await self._send_log(f"✅ Workflow {workflow_name} ready") logging.info(f"Workflow {workflow_name} setup complete at {workflow_dir}") @@ -944,31 +948,30 @@ async def _handle_workflow_selection(self, git_url: str, branch: str = "main", p derived_name = repo or '' if not derived_name: # Fallback: last path segment without .git - from urllib.parse import urlparse as _urlparse - p = _urlparse(git_url) + p = urlparse(git_url) parts = [p for p in (p.path or '').split('/') if p] if parts: derived_name = parts[-1] derived_name = (derived_name or '').removesuffix('.git').strip() except Exception: derived_name = 'workflow' - + if not derived_name: await self._send_log("❌ Could not derive workflow name from URL") return - + # Clone the workflow repository await self._clone_workflow_repository(git_url, branch, path, derived_name) - + # Set environment variables for the restart os.environ['ACTIVE_WORKFLOW_GIT_URL'] = git_url os.environ['ACTIVE_WORKFLOW_BRANCH'] = branch if path and path.strip(): os.environ['ACTIVE_WORKFLOW_PATH'] = path - + # Request restart to switch Claude's working directory self._restart_requested = True - + except Exception as e: logging.error(f"Failed to setup workflow: {e}") await self._send_log(f"❌ Workflow setup failed: {e}") @@ -978,21 +981,22 @@ async def _handle_repo_added(self, payload): repo_url = str(payload.get('url') or '').strip() repo_branch = str(payload.get('branch') or '').strip() or 'main' repo_name = str(payload.get('name') or '').strip() - + if not repo_url or not repo_name: logging.warning("Invalid repo_added payload") return - + workspace = Path(self.context.workspace_path) repo_dir = workspace / repo_name - + if repo_dir.exists(): await self._send_log(f"Repository {repo_name} already exists") return - - token = os.getenv("GITHUB_TOKEN") or await self._fetch_github_token() + + # Fetch appropriate token based on repo URL + token = await self._fetch_token_for_url(repo_url) clone_url = self._url_with_token(repo_url, token) if token else repo_url - + await self._send_log(f"📥 Cloning {repo_name}...") await self._run_cmd(["git", "clone", "--branch", repo_branch, "--single-branch", clone_url, str(repo_dir)], cwd=str(workspace)) @@ -1003,40 +1007,40 @@ async def _handle_repo_added(self, payload): await self._run_cmd(["git", "config", "user.email", user_email], cwd=str(repo_dir)) await self._send_log(f"✅ Repository {repo_name} added") - + # Update REPOS_JSON env var repos_cfg = self._get_repos_config() repos_cfg.append({'name': repo_name, 'input': {'url': repo_url, 'branch': repo_branch}}) os.environ['REPOS_JSON'] = _json.dumps(repos_cfg) - + # Request restart to update additional directories self._restart_requested = True async def _handle_repo_removed(self, payload): """Remove repository and request restart.""" repo_name = str(payload.get('name') or '').strip() - + if not repo_name: logging.warning("Invalid repo_removed payload") return - + workspace = Path(self.context.workspace_path) repo_dir = workspace / repo_name - + if not repo_dir.exists(): await self._send_log(f"Repository {repo_name} not found") return - + await self._send_log(f"🗑️ Removing {repo_name}...") shutil.rmtree(repo_dir) - + # Update REPOS_JSON env var repos_cfg = self._get_repos_config() repos_cfg = [r for r in repos_cfg if r.get('name') != repo_name] os.environ['REPOS_JSON'] = _json.dumps(repos_cfg) - + await self._send_log(f"✅ Repository {repo_name} removed") - + # Request restart to update additional directories self._restart_requested = True @@ -1342,13 +1346,12 @@ async def _update_cr_annotation(self, key: str, value: str): # Transform status URL to patch endpoint try: - from urllib.parse import urlparse as _up, urlunparse as _uu - p = _up(status_url) + p = urlparse(status_url) # Remove /status suffix to get base resource URL new_path = p.path.rstrip("/") if new_path.endswith("/status"): new_path = new_path[:-7] - url = _uu((p.scheme, p.netloc, new_path, '', '', '')) + url = urlunparse((p.scheme, p.netloc, new_path, '', '', '')) # JSON merge patch to update annotations patch = _json.dumps({ @@ -1368,6 +1371,7 @@ async def _update_cr_annotation(self, key: str, value: str): req.add_header('Authorization', f'Bearer {token}') loop = asyncio.get_event_loop() + def _do(): try: with _urllib_request.urlopen(req, timeout=10) as resp: @@ -1511,9 +1515,19 @@ def _url_with_token(self, url: str, token: str) -> str: netloc = parsed.netloc if "@" in netloc: netloc = netloc.split("@", 1)[1] - auth = f"x-access-token:{token}@" + + # Use appropriate auth format based on provider + hostname = parsed.hostname or "" + if 'gitlab' in hostname.lower(): + # GitLab uses oauth2 token format + auth = f"oauth2:{token}@" + else: + # GitHub and others use x-access-token format + auth = f"x-access-token:{token}@" + new_netloc = auth + netloc - return urlunparse((parsed.scheme, new_netloc, parsed.path, parsed.params, parsed.query, parsed.fragment)) + return urlunparse((parsed.scheme, new_netloc, parsed.path, + parsed.params, parsed.query, parsed.fragment)) except Exception: return url @@ -1540,8 +1554,7 @@ async def _get_sdk_session_id(self, session_name: str) -> str: try: # Transform status URL to point to parent session - from urllib.parse import urlparse as _up, urlunparse as _uu - p = _up(status_url) + p = urlparse(status_url) path_parts = [pt for pt in p.path.split('/') if pt] if 'projects' in path_parts and 'agentic-sessions' in path_parts: @@ -1549,7 +1562,7 @@ async def _get_sdk_session_id(self, session_name: str) -> str: project = path_parts[proj_idx + 1] if len(path_parts) > proj_idx + 1 else '' # Point to parent session's status new_path = f"/api/projects/{project}/agentic-sessions/{session_name}" - url = _uu((p.scheme, p.netloc, new_path, '', '', '')) + url = urlunparse((p.scheme, p.netloc, new_path, '', '', '')) logging.info(f"Fetching SDK session ID from: {url}") else: logging.error("Could not parse project path from status URL") @@ -1564,6 +1577,7 @@ async def _get_sdk_session_id(self, session_name: str) -> str: req.add_header('Authorization', f'Bearer {bot}') loop = asyncio.get_event_loop() + def _do_req(): try: with _urllib_request.urlopen(req, timeout=15) as resp: @@ -1601,6 +1615,41 @@ def _do_req(): logging.error(f"Failed to parse SDK session ID: {e}") return "" + async def _fetch_token_for_url(self, url: str) -> str: + """Fetch appropriate token based on repository URL. + + Detects the provider (GitHub, GitLab) from the hostname and returns + the corresponding token from environment or API. + """ + try: + parsed = urlparse(url) + hostname = parsed.hostname or "" + + # Check if it's a GitLab instance (gitlab.com or self-hosted) + if 'gitlab' in hostname.lower(): + token = os.getenv("GITLAB_TOKEN", "").strip() + if token: + logging.info(f"Using GITLAB_TOKEN for {hostname}") + return token + else: + logging.warning( + f"No GITLAB_TOKEN found for GitLab URL: {url}") + return "" + + # Default to GitHub for github.com or unknown hosts + token = os.getenv( + "GITHUB_TOKEN") or await self._fetch_github_token() + if token: + logging.info(f"Using GitHub token for {hostname}") + return token + + except Exception as e: + logging.warning( + f"Failed to parse URL {url}: {e}, falling back to GitHub token" + ) + return os.getenv( + "GITHUB_TOKEN") or await self._fetch_github_token() + async def _fetch_github_token(self) -> str: # Try cached value from env first (GITHUB_TOKEN from ambient-non-vertex-integrations) cached = os.getenv("GITHUB_TOKEN", "").strip() @@ -1615,14 +1664,13 @@ async def _fetch_github_token(self) -> str: return "" try: - from urllib.parse import urlparse as _up, urlunparse as _uu - p = _up(status_url) + p = urlparse(status_url) new_path = p.path.rstrip("/") if new_path.endswith("/status"): new_path = new_path[:-7] + "/github/token" else: new_path = new_path + "/github/token" - url = _uu((p.scheme, p.netloc, new_path, '', '', '')) + url = urlunparse((p.scheme, p.netloc, new_path, '', '', '')) logging.info(f"Fetching GitHub token from: {url}") except Exception as e: logging.error(f"Failed to construct token URL: {e}") @@ -1637,6 +1685,7 @@ async def _fetch_github_token(self) -> str: logging.warning("No BOT_TOKEN available for token fetch") loop = asyncio.get_event_loop() + def _do_req(): try: with _urllib_request.urlopen(req, timeout=10) as resp: @@ -1677,7 +1726,6 @@ async def _send_partial_output(self, output_chunk: str, *, stream_id: str, index partial=partial, ) - async def _check_pr_intent(self, output: str): """Check if output indicates PR creation intent.""" pr_indicators = [ @@ -1709,21 +1757,21 @@ async def handle_message(self, message: dict): def _build_workspace_context_prompt(self, repos_cfg, workflow_name, artifacts_path, ambient_config): """Generate comprehensive system prompt describing workspace layout.""" - + prompt = "You are Claude Code working in a structured development workspace.\n\n" - + # Current working directory if workflow_name: prompt += "## Current Workflow\n" prompt += f"Working directory: workflows/{workflow_name}/\n" prompt += "This directory contains workflow logic and automation scripts.\n\n" - + # Artifacts directory prompt += "## Shared Artifacts Directory\n" prompt += f"Location: {artifacts_path}\n" prompt += "Purpose: Create all output artifacts (documents, specs, reports) here.\n" prompt += "This directory persists across workflows and has its own git remote.\n\n" - + # Available repos if repos_cfg: prompt += "## Available Code Repositories\n" @@ -1732,16 +1780,16 @@ def _build_workspace_context_prompt(self, repos_cfg, workflow_name, artifacts_pa prompt += f"- {name}/\n" prompt += "\nThese repositories contain source code you can read or modify.\n" prompt += "Each has its own git configuration and remote.\n\n" - + # Workflow-specific instructions if ambient_config.get("systemPrompt"): prompt += f"## Workflow Instructions\n{ambient_config['systemPrompt']}\n\n" - + prompt += "## Navigation\n" prompt += "All directories are accessible via relative or absolute paths.\n" - + return prompt - + def _get_repos_config(self) -> list[dict]: """Read repos mapping from REPOS_JSON env if present.""" try: @@ -1766,8 +1814,7 @@ def _get_repos_config(self) -> list[dict]: derived = repo or '' if not derived: # Fallback: last path segment without .git - from urllib.parse import urlparse as _urlparse - p = _urlparse(url) + p = urlparse(url) parts = [p for p in (p.path or '').split('/') if p] if parts: derived = parts[-1] @@ -1895,16 +1942,16 @@ def _load_ambient_config(self, cwd_path: str) -> dict: """ try: config_path = Path(cwd_path) / ".ambient" / "ambient.json" - + if not config_path.exists(): logging.info(f"No ambient.json found at {config_path}, using defaults") return {} - + with open(config_path, 'r') as f: config = _json.load(f) logging.info(f"Loaded ambient.json: name={config.get('name')}, artifactsDir={config.get('artifactsDir')}") return config - + except _json.JSONDecodeError as e: logging.error(f"Failed to parse ambient.json: {e}") return {} diff --git a/docs/api/gitlab-endpoints.md b/docs/api/gitlab-endpoints.md new file mode 100644 index 000000000..0f326f4b9 --- /dev/null +++ b/docs/api/gitlab-endpoints.md @@ -0,0 +1,600 @@ +# GitLab Integration API Endpoints + +This document describes the GitLab integration API endpoints available in vTeam. + +## Base URL + +``` +http://vteam-backend:8080/api +``` + +For production deployments, replace with your actual backend URL. + +## Authentication + +All endpoints require authentication via Bearer token in the Authorization header: + +```http +Authorization: Bearer +``` + +--- + +## Endpoints + +### 1. Connect GitLab Account + +Connect a GitLab account to vTeam by providing a Personal Access Token. + +**Endpoint**: `POST /auth/gitlab/connect` + +**Request Headers**: +```http +Content-Type: application/json +Authorization: Bearer +``` + +**Request Body**: +```json +{ + "personalAccessToken": "glpat-xxxxxxxxxxxxxxxxxxxx", + "instanceUrl": "https://gitlab.com" +} +``` + +**Request Parameters**: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `personalAccessToken` | string | Yes | GitLab Personal Access Token (PAT) | +| `instanceUrl` | string | No | GitLab instance URL. Defaults to "https://gitlab.com" if not provided | + +**Example Request**: +```bash +curl -X POST http://vteam-backend:8080/api/auth/gitlab/connect \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{ + "personalAccessToken": "glpat-xyz123abc456", + "instanceUrl": "https://gitlab.com" + }' +``` + +**Success Response** (`200 OK`): +```json +{ + "userId": "user-123", + "gitlabUserId": "456789", + "username": "johndoe", + "instanceUrl": "https://gitlab.com", + "connected": true, + "message": "GitLab account connected successfully" +} +``` + +**Error Responses**: + +**400 Bad Request** - Invalid request body: +```json +{ + "error": "Invalid request body", + "statusCode": 400 +} +``` + +**401 Unauthorized** - Not authenticated: +```json +{ + "error": "User not authenticated", + "statusCode": 401 +} +``` + +**500 Internal Server Error** - Token validation failed: +```json +{ + "error": "GitLab token validation failed: 401 Unauthorized", + "statusCode": 500 +} +``` + +**Notes**: +- Token is validated by calling GitLab API before storing +- Token is stored securely in Kubernetes Secrets +- Connection metadata stored in ConfigMap +- For self-hosted GitLab, `instanceUrl` must include protocol (https://) + +--- + +### 2. Get GitLab Connection Status + +Check if user has a GitLab account connected and retrieve connection details. + +**Endpoint**: `GET /auth/gitlab/status` + +**Request Headers**: +```http +Authorization: Bearer +``` + +**Example Request**: +```bash +curl -X GET http://vteam-backend:8080/api/auth/gitlab/status \ + -H "Authorization: Bearer " +``` + +**Success Response (Connected)** (`200 OK`): +```json +{ + "connected": true, + "username": "johndoe", + "instanceUrl": "https://gitlab.com", + "gitlabUserId": "456789" +} +``` + +**Success Response (Not Connected)** (`200 OK`): +```json +{ + "connected": false +} +``` + +**Error Responses**: + +**401 Unauthorized** - Not authenticated: +```json +{ + "error": "User not authenticated", + "statusCode": 401 +} +``` + +**500 Internal Server Error** - Failed to retrieve status: +```json +{ + "error": "Failed to retrieve GitLab connection status", + "statusCode": 500 +} +``` + +--- + +### 3. Disconnect GitLab Account + +Disconnect GitLab account from vTeam and remove stored credentials. + +**Endpoint**: `POST /auth/gitlab/disconnect` + +**Request Headers**: +```http +Authorization: Bearer +``` + +**Example Request**: +```bash +curl -X POST http://vteam-backend:8080/api/auth/gitlab/disconnect \ + -H "Authorization: Bearer " +``` + +**Success Response** (`200 OK`): +```json +{ + "message": "GitLab account disconnected successfully", + "connected": false +} +``` + +**Error Responses**: + +**401 Unauthorized** - Not authenticated: +```json +{ + "error": "User not authenticated", + "statusCode": 401 +} +``` + +**500 Internal Server Error** - Disconnect failed: +```json +{ + "error": "Failed to disconnect GitLab account", + "statusCode": 500 +} +``` + +**Notes**: +- Removes GitLab PAT from Kubernetes Secrets +- Removes connection metadata from ConfigMap +- Does not affect your GitLab account itself +- AgenticSessions using GitLab will fail after disconnection + +--- + +## Data Models + +### ConnectGitLabRequest + +Request body for connecting GitLab account. + +```typescript +interface ConnectGitLabRequest { + personalAccessToken: string; // Required + instanceUrl?: string; // Optional, defaults to "https://gitlab.com" +} +``` + +**Validation Rules**: +- `personalAccessToken`: Must be non-empty string +- `instanceUrl`: Must be valid HTTPS URL if provided + +**Example**: +```json +{ + "personalAccessToken": "glpat-xyz123abc456", + "instanceUrl": "https://gitlab.company.com" +} +``` + +--- + +### ConnectGitLabResponse + +Response from connecting GitLab account. + +```typescript +interface ConnectGitLabResponse { + userId: string; + gitlabUserId: string; + username: string; + instanceUrl: string; + connected: boolean; + message: string; +} +``` + +**Fields**: +- `userId`: vTeam user ID +- `gitlabUserId`: GitLab user ID (from GitLab API) +- `username`: GitLab username +- `instanceUrl`: GitLab instance URL (GitLab.com or self-hosted) +- `connected`: Always `true` on success +- `message`: Success message + +**Example**: +```json +{ + "userId": "user-abc123", + "gitlabUserId": "789456", + "username": "johndoe", + "instanceUrl": "https://gitlab.com", + "connected": true, + "message": "GitLab account connected successfully" +} +``` + +--- + +### GitLabStatusResponse + +Response from checking GitLab connection status. + +```typescript +interface GitLabStatusResponse { + connected: boolean; + username?: string; // Only present if connected + instanceUrl?: string; // Only present if connected + gitlabUserId?: string; // Only present if connected +} +``` + +**Fields**: +- `connected`: Whether GitLab account is connected +- `username`: GitLab username (omitted if not connected) +- `instanceUrl`: GitLab instance URL (omitted if not connected) +- `gitlabUserId`: GitLab user ID (omitted if not connected) + +**Example (Connected)**: +```json +{ + "connected": true, + "username": "johndoe", + "instanceUrl": "https://gitlab.com", + "gitlabUserId": "789456" +} +``` + +**Example (Not Connected)**: +```json +{ + "connected": false +} +``` + +--- + +## Error Handling + +### Error Response Format + +All error responses follow this format: + +```json +{ + "error": "Error message describing what went wrong", + "statusCode": 400 +} +``` + +### Common Error Codes + +| Status Code | Meaning | Common Causes | +|-------------|---------|---------------| +| 400 | Bad Request | Invalid request body, missing required fields | +| 401 | Unauthorized | vTeam authentication token missing or invalid | +| 500 | Internal Server Error | GitLab token validation failed, database error, network error | + +### GitLab-Specific Errors + +When GitLab token validation fails, error messages include specific remediation: + +**Invalid Token**: +```json +{ + "error": "GitLab token validation failed: 401 Unauthorized. Please ensure your token is valid and not expired", + "statusCode": 500 +} +``` + +**Insufficient Permissions**: +```json +{ + "error": "GitLab token validation failed: 403 Forbidden. Ensure your token has 'api', 'read_api', 'read_user', and 'write_repository' scopes", + "statusCode": 500 +} +``` + +**Network Error**: +```json +{ + "error": "Failed to connect to GitLab instance. Please check network connectivity and instance URL", + "statusCode": 500 +} +``` + +--- + +## Usage Examples + +### Complete Workflow + +#### 1. Check if Connected + +```bash +curl -X GET http://vteam-backend:8080/api/auth/gitlab/status \ + -H "Authorization: Bearer $VTEAM_TOKEN" +``` + +Response: +```json +{"connected": false} +``` + +#### 2. Connect Account + +```bash +curl -X POST http://vteam-backend:8080/api/auth/gitlab/connect \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $VTEAM_TOKEN" \ + -d '{ + "personalAccessToken": "'"$GITLAB_TOKEN"'", + "instanceUrl": "https://gitlab.com" + }' +``` + +Response: +```json +{ + "userId": "user-123", + "gitlabUserId": "456789", + "username": "johndoe", + "instanceUrl": "https://gitlab.com", + "connected": true, + "message": "GitLab account connected successfully" +} +``` + +#### 3. Verify Connection + +```bash +curl -X GET http://vteam-backend:8080/api/auth/gitlab/status \ + -H "Authorization: Bearer $VTEAM_TOKEN" +``` + +Response: +```json +{ + "connected": true, + "username": "johndoe", + "instanceUrl": "https://gitlab.com", + "gitlabUserId": "456789" +} +``` + +#### 4. Disconnect (if needed) + +```bash +curl -X POST http://vteam-backend:8080/api/auth/gitlab/disconnect \ + -H "Authorization: Bearer $VTEAM_TOKEN" +``` + +Response: +```json +{ + "message": "GitLab account disconnected successfully", + "connected": false +} +``` + +--- + +### Self-Hosted GitLab Example + +```bash +# Connect to self-hosted instance +curl -X POST http://vteam-backend:8080/api/auth/gitlab/connect \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $VTEAM_TOKEN" \ + -d '{ + "personalAccessToken": "glpat-selfhosted-token", + "instanceUrl": "https://gitlab.company.com" + }' +``` + +Response indicates self-hosted instance: +```json +{ + "userId": "user-123", + "gitlabUserId": "12345", + "username": "jdoe", + "instanceUrl": "https://gitlab.company.com", + "connected": true, + "message": "GitLab account connected successfully" +} +``` + +--- + +## Security Considerations + +### Token Storage + +- GitLab PATs stored in Kubernetes Secret: `gitlab-user-tokens` +- Stored in backend namespace (not user's project namespace) +- Encrypted at rest by Kubernetes +- Never exposed in API responses +- Automatically redacted in logs + +### Token Scopes + +Required GitLab token scopes: +- `api` - Full API access +- `read_api` - Read API access +- `read_user` - Read user information +- `write_repository` - Push to repositories + +### Best Practices + +1. **Use HTTPS**: Always use HTTPS for API calls in production +2. **Rotate Tokens**: Encourage users to rotate GitLab tokens every 90 days +3. **Minimum Scopes**: Only request required scopes +4. **Token Expiration**: Set expiration dates on GitLab tokens +5. **Secure vTeam Tokens**: Protect vTeam authentication tokens + +--- + +## Rate Limiting + +### GitLab.com Limits + +- 300 requests per minute per user +- 10,000 requests per hour per user + +### Self-Hosted Limits + +Limits configured by GitLab administrator (may differ from GitLab.com). + +### vTeam Behavior + +- No rate limit enforcement on vTeam side +- GitLab API errors (429 Too Many Requests) passed through to user +- Error messages include wait time recommendation + +--- + +## Testing + +### Unit Tests + +```bash +cd components/backend +go test ./handlers/... -run TestGitLab -v +``` + +### Integration Tests + +```bash +export INTEGRATION_TESTS=true +export GITLAB_TEST_TOKEN="glpat-xxx" +export GITLAB_TEST_REPO_URL="https://gitlab.com/user/repo.git" + +go test ./tests/integration/gitlab/... -v +``` + +### Manual Testing with cURL + +See examples throughout this document. + +--- + +## Troubleshooting + +### "Invalid request body" + +**Cause**: JSON malformed or missing required fields + +**Solution**: +- Verify JSON is valid +- Ensure `personalAccessToken` field is present +- Check Content-Type header is `application/json` + +### "User not authenticated" + +**Cause**: vTeam authentication token missing or invalid + +**Solution**: +- Include `Authorization: Bearer ` header +- Verify vTeam token is valid +- Check token hasn't expired + +### "GitLab token validation failed: 401" + +**Cause**: GitLab token is invalid or expired + +**Solution**: +- Create new GitLab Personal Access Token +- Verify token is copied correctly (no extra spaces) +- Check token hasn't been revoked in GitLab + +### "GitLab token validation failed: 403" + +**Cause**: Token lacks required scopes + +**Solution**: +- Recreate token with all required scopes: + - `api` + - `read_api` + - `read_user` + - `write_repository` + +--- + +## Related Documentation + +- [GitLab Integration Guide](../gitlab-integration.md) +- [GitLab Token Setup](../gitlab-token-setup.md) +- [Self-Hosted GitLab Configuration](../gitlab-self-hosted.md) +- [GitLab Testing Procedures](../gitlab-testing-procedures.md) + +--- + +## Changelog + +### v1.1.0 (2025-11-05) + +- ✨ Initial GitLab integration API release +- Added `/auth/gitlab/connect` endpoint +- Added `/auth/gitlab/status` endpoint +- Added `/auth/gitlab/disconnect` endpoint +- Support for GitLab.com and self-hosted instances +- Personal Access Token authentication diff --git a/docs/gitlab-integration-test-plan.md b/docs/gitlab-integration-test-plan.md new file mode 100644 index 000000000..b92eefaac --- /dev/null +++ b/docs/gitlab-integration-test-plan.md @@ -0,0 +1,745 @@ +# GitLab Integration Test Plan + +**Feature**: GitLab Support for vTeam +**Branch**: `feature/gitlab-support` +**Date**: 2025-11-05 +**Status**: Ready for Testing + +## Overview + +This test plan validates the GitLab integration implemented in vTeam, covering User Story 1 (Configure GitLab Repository) and User Story 3 (Execute AgenticSession with GitLab). + +## Prerequisites + +### Test Environment Setup + +1. **GitLab.com Account** + - Active GitLab.com account + - At least one test repository with write access + +2. **Self-Hosted GitLab (Optional)** + - Self-hosted GitLab instance (for advanced testing) + - Test repository with write access + +3. **GitLab Personal Access Token (PAT)** + - Token with required scopes: + - `api` (full API access) + - `read_api` (read API) + - `read_user` (read user info) + - `write_repository` (push to repositories) + - Create at: https://gitlab.com/-/profile/personal_access_tokens + +4. **vTeam Environment** + - vTeam backend running with Kubernetes access + - Backend namespace: `vteam-backend` + - kubectl access to backend namespace + - Valid user authentication token + +### Test Data + +**GitLab.com Test Repository**: +``` +URL: https://gitlab.com//.git +Example: https://gitlab.com/testuser/vteam-test-repo.git +``` + +**Self-Hosted Test Repository** (if applicable): +``` +URL: https://gitlab.example.com//.git +Example: https://gitlab.example.com/dev/integration-test.git +``` + +--- + +## Test Cases + +### TC-001: GitLab Connection - Connect with Valid Token (GitLab.com) + +**User Story**: US1 - Configure vTeam Project with GitLab Repository +**Priority**: P1 (Critical) + +**Setup**: +1. Ensure user has no existing GitLab connection +2. Prepare valid GitLab PAT with required scopes + +**Steps**: +1. Send POST request to `/auth/gitlab/connect`: + ```json + { + "personalAccessToken": "", + "instanceUrl": "" + } + ``` + +**Expected Results**: +- HTTP 200 OK response +- Response body contains: + ```json + { + "userId": "", + "gitlabUserId": "", + "username": "", + "instanceUrl": "https://gitlab.com", + "connected": true, + "message": "GitLab account connected successfully" + } + ``` +- Kubernetes Secret `gitlab-user-tokens` created in `vteam-backend` namespace +- Secret contains entry with key=``, value=`` +- ConfigMap `gitlab-connections` created in `vteam-backend` namespace +- ConfigMap contains JSON entry with connection metadata + +**Validation**: +```bash +# Check secret +kubectl get secret gitlab-user-tokens -n vteam-backend -o json | \ + jq '.data[""]' | base64 -d + +# Check configmap +kubectl get configmap gitlab-connections -n vteam-backend -o json | \ + jq '.data[""]' +``` + +--- + +### TC-002: GitLab Connection - Connect with Invalid Token + +**User Story**: US1 +**Priority**: P1 + +**Setup**: +1. Prepare invalid GitLab PAT (expired or malformed) + +**Steps**: +1. Send POST request to `/auth/gitlab/connect`: + ```json + { + "personalAccessToken": "invalid-token-12345", + "instanceUrl": "" + } + ``` + +**Expected Results**: +- HTTP 500 Internal Server Error response +- Response body contains error message indicating token validation failed +- Error message includes: "GitLab token validation failed" or "401 Unauthorized" +- No Secret or ConfigMap created + +--- + +### TC-003: GitLab Connection - Connect with Self-Hosted Instance + +**User Story**: US1 +**Priority**: P2 + +**Setup**: +1. Prepare valid PAT for self-hosted GitLab instance +2. Note the instance URL (e.g., `https://gitlab.example.com`) + +**Steps**: +1. Send POST request to `/auth/gitlab/connect`: + ```json + { + "personalAccessToken": "", + "instanceUrl": "https://gitlab.example.com" + } + ``` + +**Expected Results**: +- HTTP 200 OK response +- Response includes `"instanceUrl": "https://gitlab.example.com"` +- ConfigMap stores instanceURL correctly +- Self-hosted detection: `isGitlabSelfHosted: true` in repository metadata + +--- + +### TC-004: GitLab Connection - Get Status (Connected) + +**User Story**: US1 +**Priority**: P1 + +**Setup**: +1. Complete TC-001 successfully (user connected) + +**Steps**: +1. Send GET request to `/auth/gitlab/status` + +**Expected Results**: +- HTTP 200 OK response +- Response body: + ```json + { + "connected": true, + "username": "", + "instanceUrl": "https://gitlab.com", + "gitlabUserId": "" + } + ``` + +--- + +### TC-005: GitLab Connection - Get Status (Not Connected) + +**User Story**: US1 +**Priority**: P1 + +**Setup**: +1. Ensure user has no GitLab connection + +**Steps**: +1. Send GET request to `/auth/gitlab/status` + +**Expected Results**: +- HTTP 200 OK response +- Response body: + ```json + { + "connected": false + } + ``` + +--- + +### TC-006: GitLab Connection - Disconnect + +**User Story**: US1 +**Priority**: P1 + +**Setup**: +1. Complete TC-001 successfully (user connected) + +**Steps**: +1. Send POST request to `/auth/gitlab/disconnect` + +**Expected Results**: +- HTTP 200 OK response +- Response body: + ```json + { + "message": "GitLab account disconnected successfully", + "connected": false + } + ``` +- Token removed from `gitlab-user-tokens` Secret +- Connection metadata removed from `gitlab-connections` ConfigMap + +**Validation**: +```bash +# Verify token removed +kubectl get secret gitlab-user-tokens -n vteam-backend -o json | \ + jq '.data[""]' # Should return null + +# Verify connection removed +kubectl get configmap gitlab-connections -n vteam-backend -o json | \ + jq '.data[""]' # Should return null +``` + +--- + +### TC-007: Repository Provider Detection - GitLab HTTPS URL + +**User Story**: US1 +**Priority**: P1 + +**Steps**: +1. Test provider detection with various GitLab HTTPS URLs: + - `https://gitlab.com/owner/repo.git` + - `https://gitlab.com/owner/repo` + - `https://gitlab.example.com/group/project.git` + +**Expected Results**: +- Provider detected as `gitlab` +- URL normalized to HTTPS format with `.git` suffix +- Self-hosted instances correctly identified + +--- + +### TC-008: Repository Provider Detection - GitLab SSH URL + +**User Story**: US1 +**Priority**: P2 + +**Steps**: +1. Test provider detection with GitLab SSH URLs: + - `git@gitlab.com:owner/repo.git` + - `git@gitlab.example.com:group/project.git` + +**Expected Results**: +- Provider detected as `gitlab` +- URL normalized to HTTPS format + +--- + +### TC-009: Repository Configuration - Add GitLab Repository to Project + +**User Story**: US1 +**Priority**: P1 + +**Setup**: +1. Complete TC-001 (GitLab connected) +2. Create or use existing vTeam project + +**Steps**: +1. Update ProjectSettings CR with GitLab repository: + ```yaml + spec: + repositories: + - url: "https://gitlab.com/testuser/vteam-test-repo.git" + branch: "main" + ``` + +**Expected Results**: +- ProjectSettings CR updated successfully +- Provider automatically detected as `gitlab` +- Repository validation succeeds (if connected) +- Provider field populated in repository entry + +--- + +### TC-010: Repository Validation - Valid Repository with Valid Token + +**User Story**: US1 +**Priority**: P1 + +**Setup**: +1. Complete TC-001 (GitLab connected) +2. Use repository URL user has access to + +**Steps**: +1. Call `ValidateProjectRepository` or configure project with GitLab repo + +**Expected Results**: +- Validation succeeds +- Repository metadata returned: + - `provider: "gitlab"` + - `owner: ""` + - `repo: ""` + - `host: "gitlab.com"` or self-hosted host + - `apiUrl: "https://gitlab.com/api/v4"` or self-hosted API URL + +--- + +### TC-011: Repository Validation - Repository User Lacks Access + +**User Story**: US1 +**Priority**: P1 + +**Setup**: +1. Complete TC-001 (GitLab connected) +2. Use private repository URL user does NOT have access to + +**Steps**: +1. Call `ValidateProjectRepository` with inaccessible repository + +**Expected Results**: +- Validation fails +- Error message: "repository validation failed" or "404 Not Found" +- User-friendly error message explaining lack of access + +--- + +### TC-012: AgenticSession - Clone GitLab Repository + +**User Story**: US3 - Execute AgenticSession with GitLab Repository +**Priority**: P1 + +**Setup**: +1. Complete TC-001 (GitLab connected) +2. Complete TC-009 (Project configured with GitLab repo) +3. GitLab repository must exist and be accessible + +**Steps**: +1. Create AgenticSession with GitLab repository +2. Monitor session logs for clone operation + +**Expected Results**: +- Session pod starts successfully +- Clone operation uses oauth2:TOKEN@ authentication format +- Repository cloned successfully to session workspace +- Logs show: "Cloning GitLab repository: " + +**Validation**: +```bash +# Check session logs +kubectl logs -n | grep -i gitlab +``` + +--- + +### TC-013: AgenticSession - Commit and Push to GitLab Repository + +**User Story**: US3 +**Priority**: P1 + +**Setup**: +1. Complete TC-012 (Repository cloned) +2. AgenticSession makes file changes + +**Steps**: +1. Wait for session to complete task +2. Verify commit created +3. Verify push to GitLab succeeds + +**Expected Results**: +- Commit created locally with session changes +- Push to GitLab succeeds using oauth2:TOKEN@ authentication +- Changes visible in GitLab web UI +- Completion notification includes GitLab branch URL: + - Format: `https://gitlab.com///-/tree/` + +**Validation**: +```bash +# Check GitLab UI or API for pushed branch +curl -H "Authorization: Bearer " \ + "https://gitlab.com/api/v4/projects//repository/branches/" +``` + +--- + +### TC-014: AgenticSession - Push Error (Insufficient Permissions) + +**User Story**: US3 +**Priority**: P1 + +**Setup**: +1. Complete TC-001 with token that has read-only access (no `write_repository` scope) +2. Complete TC-009 (Project configured) + +**Steps**: +1. Create AgenticSession that attempts to push changes + +**Expected Results**: +- Clone succeeds +- Commit succeeds +- Push fails with user-friendly error message +- Error message includes: + - "GitLab push failed: Insufficient permissions" + - "Ensure your GitLab token has 'write_repository' scope" + - "You can update your token by reconnecting your GitLab account" + +--- + +### TC-015: AgenticSession - Push Error (Invalid Token) + +**User Story**: US3 +**Priority**: P2 + +**Setup**: +1. Complete TC-001 (GitLab connected) +2. Manually invalidate token or wait for expiration +3. Complete TC-009 (Project configured) + +**Steps**: +1. Create AgenticSession that attempts to push changes + +**Expected Results**: +- Clone may fail or succeed (depending on timing) +- Push fails with authentication error +- Error message includes: + - "GitLab push failed: Authentication failed" + - "Your GitLab token may be invalid or expired" + - "Please reconnect your GitLab account" + +--- + +### TC-016: AgenticSession - Self-Hosted GitLab Instance + +**User Story**: US3 +**Priority**: P2 + +**Setup**: +1. Complete TC-003 (Self-hosted GitLab connected) +2. Configure project with self-hosted GitLab repository + +**Steps**: +1. Create AgenticSession with self-hosted GitLab repo + +**Expected Results**: +- Clone uses correct self-hosted instance URL +- Authentication works with self-hosted API +- Push succeeds to self-hosted instance +- Completion notification uses self-hosted URL: + - Format: `https://gitlab.example.com///-/tree/` + +--- + +### TC-017: Error Handling - User-Friendly Messages + +**User Story**: US1, US3 +**Priority**: P1 + +**Test Scenarios**: + +| Scenario | Expected Error Message | +|----------|------------------------| +| Invalid token | "GitLab token validation failed" with 401 details | +| Insufficient permissions | "Ensure your GitLab token has 'write_repository' scope" | +| Repository not found | "Repository not found. Verify the repository URL" | +| Rate limit exceeded | "Rate limit exceeded. Please wait a few minutes" | +| Network error | "Unable to connect to gitlab.com. Check network connectivity" | + +**Validation**: +- All error messages are user-friendly (no raw stack traces) +- Error messages include remediation guidance +- Tokens are redacted in all log output + +--- + +### TC-018: Token Security - Redaction in Logs + +**User Story**: US1, US3 +**Priority**: P1 (Security Critical) + +**Steps**: +1. Complete TC-001 (GitLab connected) +2. Trigger various operations (validate, clone, push) +3. Review all logs (backend, session pod) + +**Expected Results**: +- Tokens never appear in plaintext in logs +- Token patterns redacted: + - `glpat-...` → `glpat-***` + - `Bearer ` → `Bearer ***` + - `oauth2:TOKEN@` → `oauth2:***@` +- Git URLs in logs show redacted tokens + +**Validation**: +```bash +# Search backend logs for tokens +kubectl logs -n vteam-backend | grep -i "glpat-" # Should find no matches +kubectl logs -n vteam-backend | grep "oauth2:" | grep -v "***" # Should find no matches + +# Search session logs +kubectl logs -n | grep -i "token" | grep -v "***" # Should find no matches +``` + +--- + +### TC-019: Mixed Providers - GitHub and GitLab in Same Project + +**User Story**: US4 (if implemented) +**Priority**: P3 + +**Setup**: +1. Connect both GitHub and GitLab accounts +2. Configure project with both providers: + ```yaml + spec: + repositories: + - url: "https://github.com/user/repo.git" + provider: "github" + - url: "https://gitlab.com/user/repo.git" + provider: "gitlab" + ``` + +**Steps**: +1. Create AgenticSession that uses both repositories + +**Expected Results**: +- Both repositories cloned successfully +- Correct authentication used for each provider +- GitHub uses x-access-token, GitLab uses oauth2 +- Both pushes succeed independently + +--- + +## Regression Testing + +### RT-001: GitHub Functionality Unaffected + +**Priority**: P1 (Critical) + +**Steps**: +1. Run existing GitHub integration tests +2. Verify GitHub App connections still work +3. Verify GitHub repository operations (clone, push) still work +4. Verify GitHub error handling unchanged + +**Expected Results**: +- Zero GitHub functionality regression +- All existing GitHub tests pass +- GitHub-only projects work identically to before + +--- + +### RT-002: Backward Compatibility - Existing Projects + +**Priority**: P1 + +**Steps**: +1. Load existing ProjectSettings CRs (created before GitLab support) +2. Verify they continue to work + +**Expected Results**: +- Existing projects without `provider` field work correctly +- Provider auto-detected for existing repositories +- No migration required for existing projects + +--- + +## Performance Testing + +### PT-001: Token Validation Performance + +**Target**: < 200ms per validation (per SC-002) + +**Steps**: +1. Measure time for GitLab token validation +2. Test with GitLab.com and self-hosted instances + +**Expected Results**: +- Validation completes in < 200ms (95th percentile) +- Timeout set to 15 seconds (matches GitHub) + +--- + +### PT-002: Repository Browsing Performance + +**Target**: < 3s for browsing operations (per SC-002) + +**Steps**: +1. List branches in large repository (100+ branches) +2. Browse large directory tree (1000+ files) + +**Expected Results**: +- Operations complete in < 3s (95th percentile) +- Pagination works correctly for large result sets + +--- + +## Security Testing + +### ST-001: Token Storage Security + +**Steps**: +1. Verify tokens stored in Kubernetes Secrets +2. Verify tokens encrypted at rest (K8s default) +3. Verify tokens never logged in plaintext +4. Verify tokens never exposed in API responses + +**Expected Results**: +- All security requirements met +- No token leakage vectors found + +--- + +### ST-002: Input Validation + +**Steps**: +1. Test with malicious repository URLs: + - Path traversal: `https://gitlab.com/../../etc/passwd` + - Script injection: `https://gitlab.com/` +2. Test with malformed tokens +3. Test with excessively long inputs + +**Expected Results**: +- All malicious inputs rejected +- No code injection possible +- No path traversal vulnerabilities + +--- + +## Manual Testing Checklist + +### Setup Phase +- [ ] Deploy vTeam backend with GitLab support +- [ ] Verify backend namespace exists (`vteam-backend`) +- [ ] Create GitLab.com test account and repository +- [ ] Generate GitLab PAT with required scopes +- [ ] (Optional) Set up self-hosted GitLab instance + +### User Story 1: Configure GitLab +- [ ] TC-001: Connect with valid token (GitLab.com) +- [ ] TC-002: Connect with invalid token +- [ ] TC-003: Connect with self-hosted instance +- [ ] TC-004: Get status (connected) +- [ ] TC-005: Get status (not connected) +- [ ] TC-006: Disconnect +- [ ] TC-007: Provider detection (HTTPS URLs) +- [ ] TC-008: Provider detection (SSH URLs) +- [ ] TC-009: Add GitLab repository to project +- [ ] TC-010: Validate repository with access +- [ ] TC-011: Validate repository without access + +### User Story 3: AgenticSession +- [ ] TC-012: Clone GitLab repository +- [ ] TC-013: Commit and push to GitLab +- [ ] TC-014: Push error (insufficient permissions) +- [ ] TC-015: Push error (invalid token) +- [ ] TC-016: Self-hosted GitLab instance + +### Error Handling & Security +- [ ] TC-017: User-friendly error messages +- [ ] TC-018: Token redaction in logs +- [ ] ST-001: Token storage security +- [ ] ST-002: Input validation + +### Regression & Performance +- [ ] RT-001: GitHub functionality unaffected +- [ ] RT-002: Backward compatibility +- [ ] PT-001: Token validation performance +- [ ] PT-002: Repository browsing performance (if implemented) + +--- + +## Test Results Template + +```markdown +### Test Execution Report + +**Date**: YYYY-MM-DD +**Tester**: +**Environment**: + +| Test Case | Status | Notes | +|-----------|--------|-------| +| TC-001 | ✅ Pass | | +| TC-002 | ✅ Pass | | +| ... | | | + +**Summary**: +- Total Tests: X +- Passed: Y +- Failed: Z +- Blocked: W + +**Issues Found**: +1. [Issue description] +2. ... + +**Recommendations**: +1. [Recommendation] +2. ... +``` + +--- + +## Known Limitations + +1. **User Story 2 (Repository Browsing)**: Not yet implemented +2. **User Story 4 (Mixed Providers)**: Basic support implemented, advanced scenarios untested +3. **User Story 5 (Repository Seeding)**: Not yet implemented +4. **GitLab Groups**: Nested group paths may need additional testing +5. **GitLab Subgroups**: URL parsing for subgroups (e.g., `group/subgroup/project`) needs validation + +--- + +## References + +- **Specification**: `specs/001-gitlab-support/spec.md` +- **Task List**: `specs/001-gitlab-support/tasks.md` +- **Implementation Files**: + - `components/backend/gitlab/` + - `components/backend/handlers/gitlab_auth.go` + - `components/backend/handlers/repository.go` + - `components/backend/git/operations.go` + +--- + +## Support + +For issues or questions during testing: +- Review backend logs: `kubectl logs -l app=vteam-backend -n vteam-backend` +- Review session logs: `kubectl logs -n ` +- Check GitLab API responses using curl with PAT +- Verify Kubernetes resources: Secrets and ConfigMaps in `vteam-backend` namespace diff --git a/docs/gitlab-integration.md b/docs/gitlab-integration.md new file mode 100644 index 000000000..517a1a6e4 --- /dev/null +++ b/docs/gitlab-integration.md @@ -0,0 +1,716 @@ +# GitLab Integration for vTeam + +vTeam now supports GitLab repositories alongside GitHub, enabling you to use your GitLab projects with AgenticSessions. This guide covers everything you need to know about using GitLab with vTeam. + +## Overview + +**What's Supported:** +- ✅ GitLab.com (public SaaS) +- ✅ Self-hosted GitLab instances (Community & Enterprise editions) +- ✅ Personal Access Token (PAT) authentication +- ✅ HTTPS and SSH URL formats +- ✅ Public and private repositories +- ✅ Clone, commit, and push operations +- ✅ Multi-repository projects (mix GitHub and GitLab) + +**Requirements:** +- GitLab Personal Access Token with appropriate scopes +- Repository with write access (for AgenticSessions) +- vTeam backend v1.1.0 or higher + +--- + +## Quick Start + +### 1. Create GitLab Personal Access Token + +1. **Log in to GitLab**: https://gitlab.com (or your self-hosted instance) + +2. **Navigate to Access Tokens**: + - Click your profile icon (top right) + - Select "Preferences" → "Access Tokens" + - Or visit: https://gitlab.com/-/profile/personal_access_tokens + +3. **Create Token**: + - **Token name**: `vTeam Integration` + - **Expiration**: Set 90+ days from now + - **Select scopes**: + - ✅ `api` - Full API access (required) + - ✅ `read_api` - Read API access + - ✅ `read_user` - Read user information + - ✅ `write_repository` - Push to repositories + +4. **Copy Token**: Save the token starting with `glpat-...` securely + +**Detailed instructions**: See [GitLab PAT Setup Guide](./gitlab-token-setup.md) + +--- + +### 2. Connect GitLab Account to vTeam + +**Via vTeam UI** (if available): +1. Navigate to Settings → Integrations +2. Click "Connect GitLab" +3. Paste your Personal Access Token +4. (Optional) For self-hosted: Enter instance URL (e.g., `https://gitlab.company.com`) +5. Click "Connect" + +**Via API**: +```bash +curl -X POST http://vteam-backend:8080/api/auth/gitlab/connect \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{ + "personalAccessToken": "glpat-your-token-here", + "instanceUrl": "" + }' +``` + +**For self-hosted GitLab**, include the instance URL: +```json +{ + "personalAccessToken": "glpat-your-token-here", + "instanceUrl": "https://gitlab.company.com" +} +``` + +**Success Response**: +```json +{ + "userId": "user-123", + "gitlabUserId": "456789", + "username": "yourname", + "instanceUrl": "https://gitlab.com", + "connected": true, + "message": "GitLab account connected successfully" +} +``` + +--- + +### 3. Configure Project with GitLab Repository + +**Option A: Via vTeam UI** +1. Open your vTeam project +2. Navigate to Project Settings +3. Under "Repositories", click "Add Repository" +4. Enter GitLab repository URL: + - HTTPS: `https://gitlab.com/owner/repo.git` + - SSH: `git@gitlab.com:owner/repo.git` +5. Enter default branch (e.g., `main`) +6. Save settings + +**Option B: Via Kubernetes** + +Edit your ProjectSettings custom resource: +```bash +kubectl edit projectsettings -n +``` + +Add GitLab repository to spec: +```yaml +apiVersion: ambient-code.io/v1 +kind: ProjectSettings +metadata: + name: projectsettings + namespace: my-project +spec: + repositories: + - url: "https://gitlab.com/myteam/myrepo.git" + branch: "main" + provider: "gitlab" # Auto-detected, optional +``` + +**Multiple Repositories** (GitHub + GitLab): +```yaml +spec: + repositories: + - url: "https://github.com/myteam/frontend.git" + branch: "main" + - url: "https://gitlab.com/myteam/backend.git" + branch: "develop" +``` + +--- + +### 4. Create AgenticSession with GitLab Repository + +Once your GitLab account is connected and repository configured, create sessions normally: + +**Example AgenticSession CR**: +```yaml +apiVersion: ambient-code.io/v1alpha1 +kind: AgenticSession +metadata: + name: add-feature-x + namespace: my-project +spec: + description: "Add feature X to the backend service" + outputRepo: + url: "https://gitlab.com/myteam/backend.git" + branch: "feature/add-feature-x" +``` + +**What Happens**: +1. Session pod starts with your task description +2. Repository clones using your GitLab PAT (automatic authentication) +3. Claude Code agent makes changes +4. Changes committed to local repository +5. Branch pushed to GitLab with your commits +6. Completion notification includes GitLab branch link + +**Completion Notification**: +``` +AgenticSession completed successfully! + +View changes in GitLab: +https://gitlab.com/myteam/backend/-/tree/feature/add-feature-x +``` + +--- + +## Supported URL Formats + +vTeam automatically detects and normalizes GitLab URLs: + +### HTTPS URLs (Recommended) +``` +✅ https://gitlab.com/owner/repo.git +✅ https://gitlab.com/owner/repo +✅ https://gitlab.company.com/group/project.git +✅ https://gitlab.company.com/group/subgroup/project.git +``` + +### SSH URLs +``` +✅ git@gitlab.com:owner/repo.git +✅ git@gitlab.company.com:group/project.git +``` + +**Provider Auto-Detection**: +- URLs containing `gitlab.com` → Detected as GitLab.com +- URLs containing other hosts with `gitlab` pattern → Detected as self-hosted GitLab +- Provider field is optional in ProjectSettings (auto-detected from URL) + +--- + +## Repository Access Validation + +vTeam validates your access to GitLab repositories before allowing operations: + +**Validation Checks**: +1. ✅ Token is valid and not expired +2. ✅ User has access to repository +3. ✅ Token has sufficient permissions (read + write) +4. ✅ Repository exists and is accessible + +**When Validation Occurs**: +- When connecting GitLab account (token validation) +- When configuring project repository (repository access check) +- Before starting AgenticSession (pre-flight validation) + +**Common Validation Errors**: + +| Error | Cause | Solution | +|-------|-------|----------| +| Invalid token | Token expired or revoked | Reconnect with new PAT | +| Insufficient permissions | Token lacks `write_repository` | Recreate token with required scopes | +| Repository not found | Private repo or no access | Verify URL and repository permissions | +| Rate limit exceeded | Too many API calls | Wait a few minutes, then retry | + +--- + +## Managing GitLab Connection + +### Check Connection Status + +**Via API**: +```bash +curl -X GET http://vteam-backend:8080/api/auth/gitlab/status \ + -H "Authorization: Bearer " +``` + +**Response (Connected)**: +```json +{ + "connected": true, + "username": "yourname", + "instanceUrl": "https://gitlab.com", + "gitlabUserId": "456789" +} +``` + +**Response (Not Connected)**: +```json +{ + "connected": false +} +``` + +### Disconnect GitLab Account + +**Via API**: +```bash +curl -X POST http://vteam-backend:8080/api/auth/gitlab/disconnect \ + -H "Authorization: Bearer " +``` + +This removes: +- Your GitLab PAT from vTeam secrets +- Connection metadata +- Access to GitLab repositories (AgenticSessions will fail) + +**Note**: Your repositories and GitLab account are not affected. + +### Update GitLab Token + +To update your token (when expired or scopes changed): + +1. **Disconnect** current account +2. **Create new token** in GitLab with updated scopes +3. **Reconnect** with new token + +--- + +## Self-Hosted GitLab + +vTeam fully supports self-hosted GitLab instances (Community and Enterprise editions). + +### Requirements + +- GitLab instance accessible from vTeam backend pods +- Personal Access Token from your GitLab instance +- Network connectivity to GitLab API (default: port 443) + +### Configuration + +When connecting, provide your instance URL: + +```bash +curl -X POST http://vteam-backend:8080/api/auth/gitlab/connect \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{ + "personalAccessToken": "glpat-xxx", + "instanceUrl": "https://gitlab.company.com" + }' +``` + +**Instance URL Format**: +- ✅ Include `https://` protocol +- ✅ No trailing slash +- ❌ Don't include `/api/v4` path + +**Examples**: +``` +✅ https://gitlab.company.com +✅ https://git.myorg.io +❌ gitlab.company.com (missing protocol) +❌ https://gitlab.company.com/ (trailing slash) +❌ https://gitlab.company.com/api/v4 (includes API path) +``` + +### API URL Construction + +vTeam automatically constructs the correct API URL: + +| Instance URL | API URL | +|--------------|---------| +| `https://gitlab.com` | `https://gitlab.com/api/v4` | +| `https://gitlab.company.com` | `https://gitlab.company.com/api/v4` | +| `https://git.myorg.io` | `https://git.myorg.io/api/v4` | + +### Troubleshooting Self-Hosted + +**Issue**: Connection fails with "connection refused" + +**Solutions**: +1. Verify instance URL is correct and accessible +2. Check network connectivity from backend pods: + ```bash + kubectl exec -it -n vteam-backend -- curl https://gitlab.company.com + ``` +3. Verify SSL certificate is valid (or configure trust for self-signed) +4. Check firewall rules allow traffic from Kubernetes cluster + +**Issue**: Token validation fails + +**Solutions**: +1. Verify token is from correct GitLab instance (not GitLab.com) +2. Check token hasn't expired +3. Verify admin hasn't revoked token +4. Test token manually: + ```bash + curl -H "Authorization: Bearer glpat-xxx" \ + https://gitlab.company.com/api/v4/user + ``` + +**Detailed guide**: See [Self-Hosted GitLab Configuration](./gitlab-self-hosted.md) + +--- + +## Security & Best Practices + +### Token Security + +**How Tokens Are Stored**: +- ✅ Stored in Kubernetes Secrets (encrypted at rest) +- ✅ Never logged in plaintext +- ✅ Redacted in all log output (`glpat-***`) +- ✅ Not exposed in API responses +- ✅ Injected into git URLs only in memory + +**Token Redaction Examples**: +``` +# In logs, you'll see: +[GitLab] Using token glpat-*** for user john +[GitLab] Cloning https://oauth2:***@gitlab.com/team/repo.git + +# Never: +[GitLab] Using token glpat-abc123xyz456 +``` + +### Token Scopes + +**Minimum Required Scopes**: +- `api` - Full API access +- `read_repository` - Clone repositories +- `write_repository` - Push changes + +**Recommended Scopes**: +- `api` - Covers all operations +- `read_api` - Read-only API access +- `read_user` - User information +- `write_repository` - Push to repos + +**Avoid**: +- `sudo` - Not needed, grants excessive privileges +- `admin_mode` - Not needed + +### Token Rotation + +**Recommendation**: Rotate tokens every 90 days + +**Process**: +1. Create new token in GitLab with same scopes +2. Test new token works (curl to GitLab API) +3. Disconnect vTeam GitLab connection +4. Reconnect with new token +5. Revoke old token in GitLab + +### Repository Permissions + +**Required Permissions**: +- Read access for cloning +- Write access for pushing changes +- For private repos: Must be member with at least Developer role + +**GitLab Role Requirements**: +| Role | Can Clone | Can Push | Recommended For | +|------|-----------|----------|-----------------| +| Guest | ❌ | ❌ | Not supported | +| Reporter | ✅ | ❌ | Read-only use cases | +| Developer | ✅ | ✅ | ✅ AgenticSessions | +| Maintainer | ✅ | ✅ | ✅ AgenticSessions | +| Owner | ✅ | ✅ | ✅ AgenticSessions | + +--- + +## Troubleshooting + +### Connection Issues + +**Problem**: "Invalid token" error when connecting + +**Solutions**: +1. Verify token is copied correctly (no extra spaces) +2. Check token hasn't expired in GitLab +3. For self-hosted: Ensure `instanceUrl` is correct +4. Test token manually: + ```bash + curl -H "Authorization: Bearer glpat-xxx" \ + https://gitlab.com/api/v4/user + ``` + +**Problem**: "Insufficient permissions" error + +**Solutions**: +1. Verify token has all required scopes: + - `api` ✅ + - `read_repository` ✅ + - `write_repository` ✅ +2. Recreate token with correct scopes +3. Reconnect vTeam with new token + +--- + +### Repository Configuration Issues + +**Problem**: Provider not auto-detected + +**Solutions**: +1. Verify URL contains `gitlab.com` or matches GitLab pattern +2. Manually specify provider in ProjectSettings: + ```yaml + spec: + repositories: + - url: "https://gitlab.company.com/team/repo.git" + provider: "gitlab" + ``` + +**Problem**: Repository validation fails + +**Solutions**: +1. Check you're connected to GitLab (`/auth/gitlab/status`) +2. Verify you have access to repository (try cloning manually) +3. For private repos: Ensure you're a member with Developer+ role +4. Check repository URL is correct + +--- + +### AgenticSession Issues + +**Problem**: Session fails to clone repository + +**Solutions**: +1. Verify GitLab account is connected +2. Check token stored in secret: + ```bash + kubectl get secret gitlab-user-tokens -n vteam-backend + ``` +3. Verify repository URL is correct +4. Check session logs: + ```bash + kubectl logs -n + ``` + +**Problem**: Clone succeeds but push fails (403 Forbidden) + +**Solutions**: +1. Token lacks `write_repository` scope → Recreate token +2. You don't have push access → Contact repo owner +3. Branch is protected → Use different branch or update protection rules + +**Error Message**: +``` +GitLab push failed: Insufficient permissions. Ensure your GitLab token +has 'write_repository' scope. You can update your token by reconnecting +your GitLab account with the required permissions. +``` + +**Problem**: Self-hosted GitLab URL not working + +**Solutions**: +1. Verify instance URL format (must include `https://`) +2. Check API URL construction in logs +3. Test connectivity from backend pod +4. Verify SSL certificate is valid + +--- + +## Limits & Quotas + +### GitLab.com Rate Limits + +**API Rate Limits** (GitLab.com): +- 300 requests per minute per user +- 10,000 requests per hour per user + +**How vTeam Handles Rate Limits**: +- Errors returned with clear message +- Recommended wait time provided +- No automatic retry (to avoid making it worse) + +**Error Message**: +``` +GitLab API rate limit exceeded. Please wait a few minutes before +retrying. GitLab.com allows 300 requests per minute. +``` + +### Self-Hosted Rate Limits + +Self-hosted instances may have different limits configured by admins. Check with your GitLab administrator. + +--- + +## Mixing GitHub and GitLab + +vTeam supports projects with both GitHub and GitLab repositories. + +**Example Multi-Provider Project**: +```yaml +spec: + repositories: + - url: "https://github.com/team/frontend.git" + branch: "main" + provider: "github" + + - url: "https://gitlab.com/team/backend.git" + branch: "develop" + provider: "gitlab" + + - url: "https://gitlab.company.com/team/infrastructure.git" + branch: "main" + provider: "gitlab" +``` + +**How It Works**: +- Provider auto-detected from URL +- Correct authentication method used automatically: + - GitHub: Uses GitHub App or GIT_TOKEN + - GitLab: Uses GitLab PAT +- Each repo operates independently +- Errors are provider-specific and clear + +**AgenticSession with Multiple Repos**: +- Session can work with multiple repos in one task +- Each repo cloned with appropriate authentication +- Changes pushed to correct providers + +--- + +## API Reference + +### Connect GitLab Account + +```http +POST /api/auth/gitlab/connect +Content-Type: application/json +Authorization: Bearer + +{ + "personalAccessToken": "glpat-xxx", + "instanceUrl": "https://gitlab.com" # Optional, defaults to gitlab.com +} +``` + +**Response (200 OK)**: +```json +{ + "userId": "user-123", + "gitlabUserId": "456789", + "username": "yourname", + "instanceUrl": "https://gitlab.com", + "connected": true, + "message": "GitLab account connected successfully" +} +``` + +--- + +### Get Connection Status + +```http +GET /api/auth/gitlab/status +Authorization: Bearer +``` + +**Response (200 OK - Connected)**: +```json +{ + "connected": true, + "username": "yourname", + "instanceUrl": "https://gitlab.com", + "gitlabUserId": "456789" +} +``` + +**Response (200 OK - Not Connected)**: +```json +{ + "connected": false +} +``` + +--- + +### Disconnect GitLab Account + +```http +POST /api/auth/gitlab/disconnect +Authorization: Bearer +``` + +**Response (200 OK)**: +```json +{ + "message": "GitLab account disconnected successfully", + "connected": false +} +``` + +--- + +## FAQ + +**Q: Can I use the same token for multiple vTeam users?** +A: No. Each vTeam user should connect their own GitLab account with their own PAT. This ensures: +- Audit trail shows real user +- Correct permissions enforcement +- Individual token rotation + +**Q: What happens if my token expires?** +A: AgenticSessions will fail with "Authentication failed" error. Reconnect with a new token. + +**Q: Can I use SSH URLs?** +A: Yes, vTeam accepts SSH URLs and automatically converts them to HTTPS for authentication. + +**Q: Do I need to configure SSH keys?** +A: No. vTeam uses HTTPS + Personal Access Token authentication exclusively. + +**Q: Can I use Deploy Tokens instead of PATs?** +A: Not currently. Only Personal Access Tokens are supported. + +**Q: Does vTeam support GitLab Groups/Subgroups?** +A: Yes. URLs like `https://gitlab.com/group/subgroup/project.git` work correctly. + +**Q: What if I don't have a GitLab account?** +A: Create one at https://gitlab.com - it's free for public and private repositories. + +**Q: Can I use vTeam with GitLab Enterprise?** +A: Yes. Self-hosted GitLab Enterprise Edition is fully supported. + +**Q: How do I know if my token has the right scopes?** +A: Test it: +```bash +curl -H "Authorization: Bearer glpat-xxx" \ + https://gitlab.com/api/v4/personal_access_tokens/self +``` + +**Q: Will GitHub stop working after I add GitLab?** +A: No. GitHub and GitLab integrations are independent and work side-by-side. + +--- + +## Support & Resources + +**Documentation**: +- [GitLab PAT Setup Guide](./gitlab-token-setup.md) +- [Self-Hosted GitLab Configuration](./gitlab-self-hosted.md) +- [GitLab Testing Procedures](./gitlab-testing-procedures.md) + +**GitLab Resources**: +- [Personal Access Tokens Documentation](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) +- [GitLab API Documentation](https://docs.gitlab.com/ee/api/) +- [GitLab Permissions](https://docs.gitlab.com/ee/user/permissions.html) + +**Troubleshooting**: +- Check backend logs: `kubectl logs -l app=vteam-backend -n vteam-backend` +- Check session logs: `kubectl logs -n ` +- Verify GitLab status: https://status.gitlab.com (for GitLab.com) + +**Getting Help**: +- vTeam GitHub Issues: [Create an issue](https://github.com/natifridman/vTeam/issues) +- vTeam Documentation: [Main README](../README.md) + +--- + +## Changelog + +**v1.1.0** - 2025-11-05 +- ✨ Initial GitLab integration release +- ✅ GitLab.com support +- ✅ Self-hosted GitLab support +- ✅ Personal Access Token authentication +- ✅ AgenticSession clone/commit/push operations +- ✅ Multi-provider support (GitHub + GitLab) diff --git a/docs/gitlab-self-hosted.md b/docs/gitlab-self-hosted.md new file mode 100644 index 000000000..4d9270a53 --- /dev/null +++ b/docs/gitlab-self-hosted.md @@ -0,0 +1,879 @@ +# Self-Hosted GitLab Configuration for vTeam + +This guide covers everything you need to configure vTeam with self-hosted GitLab instances (GitLab Community Edition or GitLab Enterprise Edition). + +## Overview + +vTeam fully supports self-hosted GitLab installations, including: +- ✅ GitLab Community Edition (CE) +- ✅ GitLab Enterprise Edition (EE) +- ✅ Custom domains and ports +- ✅ Self-signed SSL certificates (with configuration) +- ✅ Internal/private networks +- ✅ Air-gapped environments (with limitations) + +--- + +## Prerequisites + +### Network Requirements + +**From vTeam Backend Pods**: +- HTTPS access to GitLab instance (typically port 443) +- DNS resolution of GitLab hostname +- No firewall blocking outbound connections + +**From GitLab Instance**: +- No inbound access required from vTeam +- All communication is outbound from vTeam to GitLab + +### GitLab Requirements + +**Minimum GitLab Version**: 13.0+ +- Recommended: 14.0+ for best API compatibility +- API v4 must be enabled (default) +- Personal Access Tokens enabled (default) + +**Required GitLab Features**: +- API access enabled +- Git over HTTPS enabled +- Personal Access Tokens not disabled by administrator + +### User Permissions + +**GitLab User Requirements**: +- Active user account on GitLab instance +- Ability to create Personal Access Tokens (may be restricted by admin) +- Member of repositories with at least Developer role + +**GitLab Administrator** (for installation setup): +- Access to GitLab admin area (optional, for troubleshooting) +- Can verify token settings and rate limits + +--- + +## Configuration + +### Step 1: Verify GitLab Accessibility + +Before configuring vTeam, verify GitLab is accessible from Kubernetes cluster: + +```bash +# From your local machine (should work if GitLab is public) +curl -I https://gitlab.company.com + +# From vTeam backend pod (critical test) +kubectl exec -it -n vteam-backend -- \ + curl -I https://gitlab.company.com +``` + +**Expected Response**: +``` +HTTP/2 200 +server: nginx +... +``` + +**Common Issues**: + +**Connection refused**: +``` +curl: (7) Failed to connect to gitlab.company.com port 443: Connection refused +``` +- Firewall blocking traffic from Kubernetes +- GitLab not accessible from cluster network +- Wrong hostname or port + +**DNS resolution failed**: +``` +curl: (6) Could not resolve host: gitlab.company.com +``` +- DNS not configured in Kubernetes cluster +- Hostname typo +- Internal DNS not reachable from pods + +**SSL certificate error**: +``` +curl: (60) SSL certificate problem: self signed certificate +``` +- Self-signed certificate (see SSL Configuration section below) + +--- + +### Step 2: Create Personal Access Token + +Follow the [GitLab PAT Setup Guide](./gitlab-token-setup.md) with these self-hosted specific notes: + +**Access Token Page URL**: +``` +https://gitlab.company.com/-/profile/personal_access_tokens +``` +(Replace `gitlab.company.com` with your instance hostname) + +**Required Scopes** (same as GitLab.com): +- ✅ `api` +- ✅ `read_api` +- ✅ `read_user` +- ✅ `write_repository` + +**Expiration Policy**: +- Check with your GitLab administrator +- Some organizations enforce maximum expiration (e.g., 90 days) +- Some require periodic rotation + +--- + +### Step 3: Test GitLab API Access + +Verify token works with your self-hosted instance: + +```bash +# Test user API +curl -H "Authorization: Bearer glpat-your-token" \ + https://gitlab.company.com/api/v4/user +``` + +**Expected Response** (200 OK): +```json +{ + "id": 123, + "username": "yourname", + "name": "Your Name", + "state": "active", + "web_url": "https://gitlab.company.com/yourname" +} +``` + +**Test from Backend Pod** (critical): +```bash +kubectl exec -it -n vteam-backend -- \ + curl -H "Authorization: Bearer glpat-your-token" \ + https://gitlab.company.com/api/v4/user +``` + +If this fails but works from your machine, there's a network/firewall issue. + +--- + +### Step 4: Connect to vTeam + +**Via API** (recommended for initial testing): + +```bash +curl -X POST http://vteam-backend:8080/api/auth/gitlab/connect \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{ + "personalAccessToken": "glpat-your-gitlab-token", + "instanceUrl": "https://gitlab.company.com" + }' +``` + +**Instance URL Format**: +| ✅ Correct | ❌ Incorrect | +|------------|--------------| +| `https://gitlab.company.com` | `gitlab.company.com` (missing protocol) | +| `https://git.internal.corp` | `https://gitlab.company.com/` (trailing slash) | +| `https://gitlab.company.com:8443` | `https://gitlab.company.com/api/v4` (includes path) | + +**Success Response**: +```json +{ + "userId": "user-123", + "gitlabUserId": "456789", + "username": "yourname", + "instanceUrl": "https://gitlab.company.com", + "connected": true, + "message": "GitLab account connected successfully" +} +``` + +**Check Connection**: +```bash +curl -X GET http://vteam-backend:8080/api/auth/gitlab/status \ + -H "Authorization: Bearer " +``` + +Expected: +```json +{ + "connected": true, + "username": "yourname", + "instanceUrl": "https://gitlab.company.com", + "gitlabUserId": "456789" +} +``` + +--- + +## SSL Certificate Configuration + +### Trusted SSL Certificates + +If your GitLab instance uses SSL certificates from a trusted CA (e.g., Let's Encrypt, DigiCert), no additional configuration needed. + +**Verify**: +```bash +curl https://gitlab.company.com +# Should not show certificate errors +``` + +--- + +### Self-Signed SSL Certificates + +If your GitLab uses self-signed certificates, you must configure vTeam backend pods to trust them. + +#### Option 1: Add CA Certificate to Backend Pods + +**Step 1: Get GitLab CA Certificate** + +```bash +# Download GitLab's CA certificate +echo | openssl s_client -showcerts -connect gitlab.company.com:443 2>/dev/null | \ + openssl x509 -outform PEM > gitlab-ca.crt +``` + +**Step 2: Create Kubernetes ConfigMap** + +```bash +kubectl create configmap gitlab-ca-cert \ + --from-file=gitlab-ca.crt=gitlab-ca.crt \ + -n vteam-backend +``` + +**Step 3: Mount CA Certificate in Backend Deployment** + +Edit backend deployment: +```bash +kubectl edit deployment vteam-backend -n vteam-backend +``` + +Add volume and volumeMount: +```yaml +spec: + template: + spec: + containers: + - name: backend + # ... existing config ... + volumeMounts: + - name: gitlab-ca-cert + mountPath: /etc/ssl/certs/gitlab-ca.crt + subPath: gitlab-ca.crt + readOnly: true + volumes: + - name: gitlab-ca-cert + configMap: + name: gitlab-ca-cert +``` + +**Step 4: Update CA Certificates in Pod** + +Add init container to update CA trust store: +```yaml +spec: + template: + spec: + initContainers: + - name: update-ca-certificates + image: alpine:latest + command: + - sh + - -c + - | + cp /etc/ssl/certs/gitlab-ca.crt /usr/local/share/ca-certificates/ + update-ca-certificates + volumeMounts: + - name: gitlab-ca-cert + mountPath: /etc/ssl/certs/gitlab-ca.crt + subPath: gitlab-ca.crt +``` + +**Step 5: Verify** + +```bash +# After pod restarts +kubectl exec -it -n vteam-backend -- \ + curl https://gitlab.company.com +# Should not show certificate errors +``` + +--- + +#### Option 2: Disable SSL Verification (NOT RECOMMENDED) + +**Security Warning**: Only use for testing/development. Never in production. + +Set environment variable in backend deployment: +```yaml +env: +- name: GIT_SSL_NO_VERIFY + value: "true" +``` + +--- + +### Certificate Troubleshooting + +**Problem**: "x509: certificate signed by unknown authority" + +**Cause**: Self-signed certificate not trusted + +**Solutions**: +1. Add CA certificate (Option 1 above) +2. Check certificate chain is complete +3. Verify certificate matches hostname + +**Problem**: "x509: certificate has expired" + +**Cause**: GitLab certificate expired + +**Solutions**: +1. Contact GitLab administrator to renew certificate +2. Cannot be worked around from vTeam side + +**Problem**: "x509: certificate is valid for gitlab.local, not gitlab.company.com" + +**Cause**: Certificate hostname mismatch + +**Solutions**: +1. Use correct hostname in `instanceUrl` +2. GitLab admin must issue certificate for correct hostname +3. Add hostname to certificate SAN (Subject Alternative Names) + +--- + +## Network Configuration + +### Firewall Rules + +**Required Outbound Access** (from vTeam backend pods): +``` +Source: vTeam backend pods (namespace: vteam-backend) +Destination: GitLab instance +Protocol: HTTPS (TCP) +Port: 443 (or custom if GitLab uses different port) +``` + +**Example Firewall Rule**: +``` +ALLOW tcp from 10.0.0.0/8 (Kubernetes pod network) to 192.168.1.100 port 443 +``` + +**No Inbound Access Required**: GitLab doesn't need to reach vTeam. + +--- + +### DNS Configuration + +vTeam backend pods must be able to resolve GitLab hostname. + +**Test DNS Resolution**: +```bash +kubectl exec -it -n vteam-backend -- \ + nslookup gitlab.company.com +``` + +**Expected**: +``` +Server: 10.96.0.10 +Address: 10.96.0.10#53 + +Name: gitlab.company.com +Address: 192.168.1.100 +``` + +**DNS Issues**: + +**Problem**: "server can't find gitlab.company.com: NXDOMAIN" + +**Cause**: Internal DNS not configured + +**Solutions**: +1. Configure CoreDNS to forward internal domains +2. Add custom DNS to backend pods: + ```yaml + spec: + dnsPolicy: "None" + dnsConfig: + nameservers: + - 192.168.1.10 # Internal DNS server + searches: + - company.com + ``` + +--- + +### Proxy Configuration + +If vTeam backend pods require HTTP proxy to reach GitLab: + +**Add Proxy Environment Variables**: +```yaml +env: +- name: HTTP_PROXY + value: "http://proxy.company.com:8080" +- name: HTTPS_PROXY + value: "http://proxy.company.com:8080" +- name: NO_PROXY + value: "localhost,127.0.0.1,.cluster.local" +``` + +**For Authenticated Proxy**: +```yaml +- name: HTTP_PROXY + value: "http://username:password@proxy.company.com:8080" +``` + +**Git Proxy Configuration** (for git operations): +```yaml +- name: GIT_PROXY_COMMAND + value: "http://proxy.company.com:8080" +``` + +--- + +## Custom Ports + +If your GitLab instance runs on a non-standard port: + +**Standard Ports**: +- HTTPS: 443 (default) +- HTTP: 80 (not recommended, insecure) + +**Custom Port Example**: +``` +GitLab URL: https://gitlab.company.com:8443 +``` + +**Configure in vTeam**: +```json +{ + "personalAccessToken": "glpat-xxx", + "instanceUrl": "https://gitlab.company.com:8443" +} +``` + +**API URL Construction**: +``` +Instance URL: https://gitlab.company.com:8443 +API URL: https://gitlab.company.com:8443/api/v4 +``` + +vTeam automatically preserves the custom port. + +--- + +## GitLab Administrator Configuration + +### Rate Limits + +Self-hosted GitLab administrators can configure custom rate limits. + +**Check Current Limits** (as admin): +1. Admin Area → Settings → Network → Rate Limits +2. Default: Same as GitLab.com (300/min, 10000/hour) + +**Recommended Settings for vTeam**: +- Authenticated API rate limit: 300 requests/minute (default) +- Unauthenticated rate limit: Can be lower +- Protected paths rate limit: Not applicable to vTeam + +**If Users Hit Rate Limits Frequently**: +- Consider increasing limits for authenticated API calls +- Monitor GitLab performance +- Check for misconfigured integrations + +--- + +### Personal Access Token Settings + +Administrators can restrict PAT creation and usage. + +**Settings Location**: +1. Admin Area → Settings → General +2. Expand "Account and limit" +3. Personal Access Token section + +**Recommended Settings**: +- ✅ Personal Access Tokens: **Enabled** +- ✅ Project Access Tokens: Can be disabled (not used by vTeam) +- ⚠️ Token expiration: Enforce 90-day maximum (recommended) +- ⚠️ Limit token lifetime: Yes (security best practice) + +**If PAT Creation is Disabled**: +- Users cannot connect to vTeam +- Administrator must enable PATs +- Or use alternative authentication (not currently supported) + +--- + +### API Access + +**Verify API is Enabled**: +1. Admin Area → Settings → General +2. Expand "Visibility and access controls" +3. Check "Enable access to the GitLab API" is **ON** + +If disabled, vTeam cannot function. + +--- + +## Air-Gapped Environments + +For completely air-gapped GitLab installations: + +### Requirements + +**GitLab Instance**: +- Accessible from Kubernetes cluster (internal network) +- Does NOT need internet access + +**vTeam Backend**: +- Must reach GitLab instance (internal network) +- Does NOT need internet access for GitLab operations + +### Configuration + +Same as standard self-hosted setup - air-gap doesn't affect vTeam → GitLab communication. + +**Network Diagram**: +``` +┌─────────────────────┐ ┌──────────────────┐ +│ vTeam Backend Pods │────────▶│ GitLab Instance │ +│ (Kubernetes) │ HTTPS │ (Self-Hosted) │ +└─────────────────────┘ └──────────────────┘ + Internal Network Only + (No Internet Required) +``` + +--- + +## Multi-Instance Support + +vTeam users can connect to multiple self-hosted GitLab instances simultaneously. + +**Limitation**: Each vTeam user can connect to **one** GitLab instance at a time. + +**Use Cases**: + +**Scenario 1: Different Users, Different Instances** +- User A connects to `https://gitlab-dev.company.com` +- User B connects to `https://gitlab-prod.company.com` +- ✅ Supported - each user has their own connection + +**Scenario 2: One User, Multiple Instances** +- User needs access to both `gitlab-dev` and `gitlab-prod` +- ❌ Not supported - must choose one instance per user +- Workaround: Use different vTeam user accounts + +**Scenario 3: Mixed GitLab.com and Self-Hosted** +- User connects to `https://gitlab.company.com` (self-hosted) +- Same user wants `https://gitlab.com` (SaaS) +- ❌ Not supported - must choose one + +--- + +## Troubleshooting + +### Connection Issues + +**Problem**: "Failed to connect to GitLab instance" + +**Debug Steps**: + +1. **Test from local machine**: + ```bash + curl https://gitlab.company.com + ``` + - If fails: GitLab is down or DNS issue + - If succeeds: Continue to step 2 + +2. **Test from backend pod**: + ```bash + kubectl exec -it -n vteam-backend -- \ + curl https://gitlab.company.com + ``` + - If fails: Firewall or network issue + - If succeeds: Continue to step 3 + +3. **Test GitLab API**: + ```bash + kubectl exec -it -n vteam-backend -- \ + curl -H "Authorization: Bearer glpat-xxx" \ + https://gitlab.company.com/api/v4/user + ``` + - If fails: API disabled or token invalid + - If succeeds: vTeam configuration issue + +4. **Check vTeam logs**: + ```bash + kubectl logs -l app=vteam-backend -n vteam-backend | grep -i gitlab + ``` + +--- + +### API Version Issues + +**Problem**: "API endpoint not found" (404) + +**Cause**: GitLab version too old or API disabled + +**Check GitLab Version**: +```bash +curl https://gitlab.company.com/api/v4/version +``` + +**Expected** (v13.0+): +```json +{ + "version": "14.10.0", + "revision": "abc123" +} +``` + +**Solutions**: +- Upgrade GitLab to 13.0+ +- Contact administrator to enable API + +--- + +### Performance Issues + +**Problem**: Slow GitLab API responses + +**Debug**: + +1. **Check API response times**: + ```bash + time curl -H "Authorization: Bearer glpat-xxx" \ + https://gitlab.company.com/api/v4/user + ``` + - Should complete in < 1 second + - If > 5 seconds: GitLab performance issue + +2. **Check network latency**: + ```bash + kubectl exec -it -n vteam-backend -- \ + ping -c 5 gitlab.company.com + ``` + - Should be < 50ms for same datacenter + - > 200ms indicates network issues + +3. **Contact GitLab Administrator**: + - Check GitLab resource utilization (CPU, memory, disk I/O) + - Review Sidekiq queue length + - Check PostgreSQL query performance + +--- + +## Security Considerations + +### Token Storage + +**Where Tokens Are Stored**: +- Kubernetes Secret: `gitlab-user-tokens` in `vteam-backend` namespace +- Encrypted at rest (Kubernetes default encryption) +- Never logged in plaintext + +**Access Control**: +- Only vTeam backend pods can read secret +- Kubernetes RBAC enforced +- Administrators can view secret but not decode tokens automatically + +**Audit Trail**: +- GitLab logs all API calls with user information +- Check GitLab audit log for token usage +- Admin Area → Monitoring → Audit Events + +--- + +### Network Security + +**Recommendations**: + +1. **Use HTTPS Only** + - Never use HTTP for GitLab + - All tokens sent over HTTPS + +2. **Restrict Network Access** + - Firewall: Only allow vTeam backend pods → GitLab + - No direct user access from pods to GitLab UI needed + +3. **SSL/TLS Configuration** + - Use trusted certificates (Let's Encrypt, etc.) + - If self-signed: Properly configure CA trust + - Never disable SSL verification in production + +4. **Audit Logging** + - Enable GitLab audit logging + - Monitor for unusual API activity + - Review PAT usage regularly + +--- + +### Compliance + +For regulated environments (HIPAA, SOC 2, etc.): + +**Token Security**: +- ✅ Tokens encrypted at rest in Kubernetes Secrets +- ✅ Tokens encrypted in transit (HTTPS only) +- ✅ Tokens automatically redacted in logs +- ✅ Token rotation supported (manually) + +**Audit Trail**: +- ✅ GitLab logs all API calls with user identity +- ✅ vTeam logs all operations with redacted tokens +- ✅ Kubernetes audit logs track secret access + +**Access Control**: +- ✅ RBAC controls who can access vTeam +- ✅ GitLab permissions control repository access +- ✅ No elevated privileges required + +--- + +## Best Practices + +### For GitLab Administrators + +1. **Enable and Monitor Audit Logs** + - Admin Area → Monitoring → Audit Events + - Track PAT creation and usage + - Alert on unusual activity + +2. **Enforce Token Expiration** + - Set maximum token lifetime (90 days recommended) + - Users must rotate tokens regularly + +3. **Configure Rate Limits Appropriately** + - Default limits work for most use cases + - Increase only if legitimate usage hits limits + - Monitor API performance impact + +4. **Maintain GitLab Version** + - Keep GitLab up to date (security patches) + - Test vTeam compatibility before major upgrades + - Minimum: GitLab 13.0+ + +5. **SSL Certificate Management** + - Use trusted certificates (Let's Encrypt, etc.) + - Automate certificate renewal + - Plan for certificate expiration + +--- + +### For vTeam Users + +1. **Use Strong Tokens** + - Create separate token for vTeam + - Use descriptive name: "vTeam Integration" + - Minimum required scopes only + +2. **Rotate Tokens Regularly** + - Every 90 days recommended + - Before expiration date + - Immediately if compromised + +3. **Monitor Token Usage** + - Check "Last Used" date in GitLab + - Revoke unused tokens + - Contact admin if suspicious activity + +4. **Repository Access** + - Request minimum necessary access + - Developer role sufficient for most use cases + - Avoid Owner/Maintainer unless needed + +--- + +## Reference + +### API Endpoints Used by vTeam + +vTeam uses these GitLab API v4 endpoints: + +**Authentication & User**: +``` +GET /api/v4/user +GET /api/v4/personal_access_tokens/self +``` + +**Repository Operations**: +``` +GET /api/v4/projects/:id +GET /api/v4/projects/:id/repository/branches +GET /api/v4/projects/:id/repository/tree +GET /api/v4/projects/:id/repository/files/:file_path +``` + +**Git Operations** (via git protocol, not API): +``` +git clone https://oauth2:TOKEN@gitlab.company.com/owner/repo.git +git push https://oauth2:TOKEN@gitlab.company.com/owner/repo.git +``` + +### Required Minimum Scopes + +| Scope | Purpose | Required | +|-------|---------|----------| +| `api` | Full API access | ✅ Yes | +| `read_api` | Read API access | ✅ Yes | +| `read_user` | User info | ✅ Yes | +| `write_repository` | Git push | ✅ Yes | + +--- + +## Support + +**For Self-Hosted GitLab Issues**: +- Contact your GitLab administrator +- Check GitLab logs: `/var/log/gitlab/` +- GitLab Community Forum: https://forum.gitlab.com + +**For vTeam Integration Issues**: +- vTeam GitHub Issues: https://github.com/natifridman/vTeam/issues +- Check vTeam logs: `kubectl logs -l app=vteam-backend -n vteam-backend` + +**For Network/Firewall Issues**: +- Contact your network/infrastructure team +- Provide: Source IPs (pod network), destination (GitLab), port (443) + +--- + +## Quick Reference + +**Test Connectivity**: +```bash +# From backend pod +kubectl exec -it -n vteam-backend -- \ + curl https://gitlab.company.com +``` + +**Test API**: +```bash +kubectl exec -it -n vteam-backend -- \ + curl -H "Authorization: Bearer glpat-xxx" \ + https://gitlab.company.com/api/v4/user +``` + +**Connect to vTeam**: +```bash +curl -X POST http://vteam-backend:8080/api/auth/gitlab/connect \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{"personalAccessToken":"glpat-xxx","instanceUrl":"https://gitlab.company.com"}' +``` + +**Check Status**: +```bash +curl -X GET http://vteam-backend:8080/api/auth/gitlab/status \ + -H "Authorization: Bearer " +``` + +**View Logs**: +```bash +kubectl logs -l app=vteam-backend -n vteam-backend | grep -i gitlab +``` diff --git a/docs/gitlab-testing-procedures.md b/docs/gitlab-testing-procedures.md new file mode 100644 index 000000000..6af59c59d --- /dev/null +++ b/docs/gitlab-testing-procedures.md @@ -0,0 +1,671 @@ +# GitLab Integration Testing Procedures + +## Quick Start Guide for Testing GitLab Support + +This guide provides step-by-step instructions for manually testing the GitLab integration in vTeam. + +--- + +## Prerequisites + +### 1. GitLab Personal Access Token Setup + +1. **Log in to GitLab**: + - For GitLab.com: https://gitlab.com + - For self-hosted: https://your-gitlab-instance.com + +2. **Navigate to Access Tokens**: + - Click your profile icon (top right) + - Select "Preferences" + - Click "Access Tokens" in left sidebar + - Or direct link: https://gitlab.com/-/profile/personal_access_tokens + +3. **Create New Token**: + - **Token name**: `vTeam Integration Test` + - **Expiration date**: Set 30+ days from now + - **Scopes** (select ALL of these): + - ✅ `api` - Full API access + - ✅ `read_api` - Read API + - ✅ `read_user` - Read user information + - ✅ `write_repository` - Push to repositories + +4. **Copy Token**: + - Click "Create personal access token" + - **IMPORTANT**: Copy the token immediately (starts with `glpat-`) + - Store securely - you won't be able to see it again + +**Example Token**: `glpat-xyz123abc456def789` (yours will be different) + +--- + +### 2. Test Repository Setup + +1. **Create Test Repository** (GitLab.com): + - Go to https://gitlab.com/projects/new + - Project name: `vteam-test-repo` + - Visibility: Private or Public (your choice) + - Initialize with README: ✅ + - Click "Create project" + +2. **Note Repository URL**: + - Clone button → Copy HTTPS URL + - Example: `https://gitlab.com/yourusername/vteam-test-repo.git` + +3. **Verify Access**: + ```bash + git clone https://oauth2:@gitlab.com/yourusername/vteam-test-repo.git + ``` + - Should clone successfully + - Delete cloned folder after verification + +--- + +### 3. vTeam Environment Setup + +1. **Verify Backend Running**: + ```bash + kubectl get pods -n vteam-backend + ``` + - Should show backend pod in Running state + +2. **Get Backend URL**: + ```bash + # Get service URL (adjust for your environment) + kubectl get svc -n vteam-backend + ``` + - Note the backend API URL (e.g., `http://vteam-backend.vteam-backend.svc.cluster.local:8080`) + +3. **Get User Auth Token**: + - Log in to vTeam UI + - Open browser developer console + - Find auth token in localStorage or cookies + - Or use test user token if available + +--- + +## Test Procedures + +### Test 1: Connect GitLab Account + +**Objective**: Verify user can connect their GitLab account to vTeam + +**Steps**: + +1. **Send Connect Request**: + ```bash + curl -X POST http://vteam-backend:8080/api/auth/gitlab/connect \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{ + "personalAccessToken": "glpat-your-actual-token-here", + "instanceUrl": "" + }' + ``` + +2. **Expected Response** (200 OK): + ```json + { + "userId": "user-123", + "gitlabUserId": "789456", + "username": "yourusername", + "instanceUrl": "https://gitlab.com", + "connected": true, + "message": "GitLab account connected successfully" + } + ``` + +3. **Verify in Kubernetes**: + ```bash + # Check secret created + kubectl get secret gitlab-user-tokens -n vteam-backend -o yaml + + # Check configmap created + kubectl get configmap gitlab-connections -n vteam-backend -o yaml + ``` + +**Success Criteria**: +- ✅ HTTP 200 response received +- ✅ Response includes your GitLab username +- ✅ Secret `gitlab-user-tokens` exists +- ✅ ConfigMap `gitlab-connections` exists +- ✅ Your user ID appears in both resources + +--- + +### Test 2: Check Connection Status + +**Objective**: Verify connection status endpoint returns correct information + +**Steps**: + +1. **Send Status Request**: + ```bash + curl -X GET http://vteam-backend:8080/api/auth/gitlab/status \ + -H "Authorization: Bearer " + ``` + +2. **Expected Response** (200 OK): + ```json + { + "connected": true, + "username": "yourusername", + "instanceUrl": "https://gitlab.com", + "gitlabUserId": "789456" + } + ``` + +**Success Criteria**: +- ✅ HTTP 200 response +- ✅ `connected: true` +- ✅ Your GitLab username shown +- ✅ Correct instanceUrl + +--- + +### Test 3: Configure Project with GitLab Repository + +**Objective**: Add GitLab repository to vTeam project + +**Steps**: + +1. **Create or Select Project**: + - Use existing vTeam project or create new one + - Note project namespace (e.g., `my-project`) + +2. **Update ProjectSettings CR**: + ```bash + kubectl edit projectsettings -n + ``` + +3. **Add GitLab Repository**: + ```yaml + spec: + repositories: + - url: "https://gitlab.com/yourusername/vteam-test-repo.git" + branch: "main" + ``` + +4. **Save and Verify**: + ```bash + kubectl get projectsettings -n -o yaml + ``` + +**Success Criteria**: +- ✅ ProjectSettings updated successfully +- ✅ Repository appears in spec +- ✅ Provider auto-detected as `gitlab` + +--- + +### Test 4: Create AgenticSession with GitLab Repository + +**Objective**: Verify session can clone, commit, and push to GitLab + +**Steps**: + +1. **Create AgenticSession CR**: + ```bash + kubectl apply -f - < + spec: + description: "Test GitLab integration by adding a comment to README" + outputRepo: + url: "https://gitlab.com/yourusername/vteam-test-repo.git" + branch: "test-branch" + EOF + ``` + +2. **Monitor Session**: + ```bash + # Watch session status + kubectl get agenticsession test-gitlab-session -n -w + + # View session logs + kubectl logs -l agenticsession=test-gitlab-session -n -f + ``` + +3. **Check for Key Log Messages**: + - "Cloning GitLab repository" + - "Using GitLab token for user" + - "Push succeeded" + - GitLab branch URL in completion notification + +4. **Verify in GitLab UI**: + - Open repository in GitLab: https://gitlab.com/yourusername/vteam-test-repo + - Click "Branches" dropdown + - Find `test-branch` + - Verify commits appear from session + +**Success Criteria**: +- ✅ Session pod starts successfully +- ✅ Repository clones without errors +- ✅ Changes committed locally +- ✅ Push to GitLab succeeds +- ✅ Branch visible in GitLab UI +- ✅ Completion notification includes GitLab URL format: + - `https://gitlab.com/yourusername/vteam-test-repo/-/tree/test-branch` + +--- + +### Test 5: Test Error Handling - Insufficient Permissions + +**Objective**: Verify user-friendly error when token lacks write access + +**Steps**: + +1. **Create Read-Only Token**: + - GitLab → Access Tokens + - Create new token with ONLY these scopes: + - ✅ `read_api` + - ✅ `read_user` + - **DO NOT** select `write_repository` + +2. **Connect with Read-Only Token**: + ```bash + curl -X POST http://vteam-backend:8080/api/auth/gitlab/connect \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{ + "personalAccessToken": "glpat-readonly-token-here", + "instanceUrl": "" + }' + ``` + +3. **Create AgenticSession** (same as Test 4) + +4. **Observe Push Failure**: + - Clone should succeed + - Commit should succeed + - Push should FAIL with user-friendly error + +**Expected Error Message**: +``` +GitLab push failed: Insufficient permissions. Ensure your GitLab token has 'write_repository' scope. You can update your token by reconnecting your GitLab account with the required permissions +``` + +**Success Criteria**: +- ✅ Error message is user-friendly (no stack traces) +- ✅ Error mentions `write_repository` scope +- ✅ Error includes remediation guidance +- ✅ Session status shows failure reason + +--- + +### Test 6: Token Security - Verify Redaction + +**Objective**: Ensure tokens never appear in logs + +**Steps**: + +1. **Search Backend Logs**: + ```bash + # Should find NO raw tokens + kubectl logs -l app=vteam-backend -n vteam-backend | grep "glpat-" + + # Should only find redacted tokens (with ***) + kubectl logs -l app=vteam-backend -n vteam-backend | grep "oauth2:" + ``` + +2. **Search Session Logs**: + ```bash + # Should find NO raw tokens + kubectl logs -l agenticsession=test-gitlab-session -n | grep "glpat-" + + # Git URLs should be redacted + kubectl logs -l agenticsession=test-gitlab-session -n | grep "https://" | grep "gitlab" + ``` + +**Success Criteria**: +- ✅ No raw tokens in backend logs +- ✅ No raw tokens in session logs +- ✅ Git URLs show `oauth2:***@` instead of `oauth2:@` +- ✅ API calls show `Bearer ***` instead of `Bearer ` + +--- + +### Test 7: Disconnect GitLab Account + +**Objective**: Verify user can safely disconnect GitLab + +**Steps**: + +1. **Send Disconnect Request**: + ```bash + curl -X POST http://vteam-backend:8080/api/auth/gitlab/disconnect \ + -H "Authorization: Bearer " + ``` + +2. **Expected Response** (200 OK): + ```json + { + "message": "GitLab account disconnected successfully", + "connected": false + } + ``` + +3. **Verify Removal**: + ```bash + # Check token removed from secret + kubectl get secret gitlab-user-tokens -n vteam-backend -o json | \ + jq '.data | keys' + + # Check connection removed from configmap + kubectl get configmap gitlab-connections -n vteam-backend -o json | \ + jq '.data | keys' + ``` + +4. **Verify Status Shows Disconnected**: + ```bash + curl -X GET http://vteam-backend:8080/api/auth/gitlab/status \ + -H "Authorization: Bearer " + ``` + + Expected: `{"connected": false}` + +**Success Criteria**: +- ✅ HTTP 200 response +- ✅ Token removed from Secret +- ✅ Connection removed from ConfigMap +- ✅ Status endpoint returns `connected: false` + +--- + +### Test 8: Self-Hosted GitLab (Optional) + +**Objective**: Verify self-hosted GitLab instances work + +**Prerequisites**: +- Access to self-hosted GitLab instance +- Repository on self-hosted instance +- PAT from self-hosted instance + +**Steps**: + +1. **Connect with Instance URL**: + ```bash + curl -X POST http://vteam-backend:8080/api/auth/gitlab/connect \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{ + "personalAccessToken": "glpat-self-hosted-token", + "instanceUrl": "https://gitlab.example.com" + }' + ``` + +2. **Verify Response**: + - Check `instanceUrl` matches your self-hosted URL + - Not `https://gitlab.com` + +3. **Create AgenticSession with Self-Hosted Repo**: + ```yaml + spec: + outputRepo: + url: "https://gitlab.example.com/group/project.git" + branch: "test-branch" + ``` + +4. **Verify Operations**: + - Clone uses self-hosted URL + - API calls go to `https://gitlab.example.com/api/v4` + - Push succeeds to self-hosted instance + - Completion URL uses self-hosted domain + +**Success Criteria**: +- ✅ Connection succeeds with custom instanceUrl +- ✅ Self-hosted API URL constructed correctly +- ✅ Clone/push work with self-hosted instance +- ✅ Completion notification shows self-hosted URL + +--- + +### Test 9: Regression - GitHub Still Works + +**Objective**: Verify GitHub functionality unaffected by GitLab changes + +**Steps**: + +1. **Connect GitHub Account** (if not already): + - Use existing GitHub App integration + - Or configure GitHub PAT in runner secrets + +2. **Create AgenticSession with GitHub Repo**: + ```yaml + spec: + outputRepo: + url: "https://github.com/username/repo.git" + branch: "test-branch" + ``` + +3. **Verify GitHub Operations**: + - Clone uses `x-access-token` authentication + - Push succeeds to GitHub + - Completion URL uses GitHub format: `https://github.com/username/repo/tree/test-branch` + +**Success Criteria**: +- ✅ GitHub sessions work identically to before GitLab support +- ✅ GitHub authentication unchanged +- ✅ No errors related to provider detection +- ✅ GitHub and GitLab can coexist in same backend instance + +--- + +## Troubleshooting Guide + +### Issue: Connection Fails with "Invalid Token" + +**Symptoms**: +- HTTP 500 response +- Error: "GitLab token validation failed" + +**Solutions**: +1. Verify token is copied correctly (no extra spaces) +2. Check token hasn't expired in GitLab +3. Verify token has required scopes: + ```bash + curl -H "Authorization: Bearer " \ + https://gitlab.com/api/v4/personal_access_tokens/self + ``` +4. Check backend logs: + ```bash + kubectl logs -l app=vteam-backend -n vteam-backend | grep -i "gitlab" + ``` + +--- + +### Issue: Session Clone Fails + +**Symptoms**: +- Session pod starts but clone fails +- Error: "no GitLab credentials available" + +**Solutions**: +1. Verify GitLab account connected: + ```bash + curl -X GET http://vteam-backend:8080/api/auth/gitlab/status \ + -H "Authorization: Bearer " + ``` +2. Check token exists in Secret: + ```bash + kubectl get secret gitlab-user-tokens -n vteam-backend -o yaml + ``` +3. Verify namespace is correct (`vteam-backend`) +4. Check session logs for detailed error: + ```bash + kubectl logs -n + ``` + +--- + +### Issue: Push Fails with 403 Forbidden + +**Symptoms**: +- Clone and commit succeed +- Push fails with "Insufficient permissions" + +**Solutions**: +1. Verify token has `write_repository` scope: + - GitLab → Access Tokens → View your token + - Check scopes list +2. Regenerate token with correct scopes if needed +3. Reconnect account: + ```bash + # Disconnect + curl -X POST http://vteam-backend:8080/api/auth/gitlab/disconnect \ + -H "Authorization: Bearer " + + # Reconnect with new token + curl -X POST http://vteam-backend:8080/api/auth/gitlab/connect \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{"personalAccessToken": "glpat-new-token", "instanceUrl": ""}' + ``` + +--- + +### Issue: Self-Hosted Instance Not Detected + +**Symptoms**: +- Self-hosted GitLab treated as GitLab.com +- API calls fail with 404 + +**Solutions**: +1. Ensure `instanceUrl` provided when connecting: + ```json + { + "personalAccessToken": "glpat-...", + "instanceUrl": "https://gitlab.example.com" // REQUIRED + } + ``` +2. Verify instance URL format: + - Must include `https://` + - No trailing slash + - No `/api/v4` path +3. Check repository URL includes correct host: + - ✅ `https://gitlab.example.com/group/project.git` + - ❌ `https://gitlab.com/group/project.git` + +--- + +### Issue: Tokens Visible in Logs + +**Symptoms**: +- Raw tokens appear in kubectl logs output + +**CRITICAL SECURITY ISSUE**: +1. Immediately report this issue +2. Rotate all affected tokens in GitLab +3. Check backend logs for redaction failures: + ```bash + kubectl logs -l app=vteam-backend -n vteam-backend | grep -E "(glpat-|oauth2:)" | grep -v "***" + ``` + +--- + +## Test Results Checklist + +After completing all tests, verify: + +**Connection Management**: +- [ ] Connect with valid token works +- [ ] Connect with invalid token shows error +- [ ] Status endpoint accurate (connected/disconnected) +- [ ] Disconnect removes credentials +- [ ] Self-hosted instance works (if tested) + +**Repository Operations**: +- [ ] Provider detection works (HTTPS, SSH) +- [ ] Repository validation works +- [ ] ProjectSettings accepts GitLab URLs + +**AgenticSession**: +- [ ] Clone succeeds with GitLab repo +- [ ] Commit creates changes locally +- [ ] Push succeeds to GitLab +- [ ] Completion notification shows GitLab URL +- [ ] Changes visible in GitLab UI + +**Error Handling**: +- [ ] Insufficient permissions shows user-friendly error +- [ ] Invalid token shows clear error message +- [ ] All errors include remediation guidance + +**Security**: +- [ ] Tokens stored in Kubernetes Secrets +- [ ] Tokens redacted in all logs +- [ ] No plaintext tokens in API responses + +**Regression**: +- [ ] GitHub functionality unchanged +- [ ] Existing projects work correctly +- [ ] No performance degradation + +--- + +## Quick Reference Commands + +### Backend Logs +```bash +kubectl logs -l app=vteam-backend -n vteam-backend -f +``` + +### Session Logs +```bash +kubectl logs -l agenticsession= -n -f +``` + +### Check Secrets +```bash +kubectl get secret gitlab-user-tokens -n vteam-backend -o yaml +``` + +### Check ConfigMaps +```bash +kubectl get configmap gitlab-connections -n vteam-backend -o yaml +``` + +### GitLab API Test +```bash +# Test your token manually +curl -H "Authorization: Bearer glpat-..." \ + https://gitlab.com/api/v4/user +``` + +### Clean Up Test Resources +```bash +# Delete test session +kubectl delete agenticsession test-gitlab-session -n + +# Disconnect GitLab +curl -X POST http://vteam-backend:8080/api/auth/gitlab/disconnect \ + -H "Authorization: Bearer " +``` + +--- + +## Next Steps + +After successful testing: +1. Document any issues found +2. Create bug reports for failures +3. Update test plan with additional scenarios discovered +4. Prepare for production deployment + +For production deployment: +- Review security checklist +- Plan token rotation strategy +- Configure monitoring/alerting +- Prepare user documentation +- Train support team on GitLab integration + +--- + +## Support Resources + +- **GitLab API Docs**: https://docs.gitlab.com/ee/api/ +- **GitLab PAT Docs**: https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html +- **vTeam GitLab Test Plan**: `/docs/gitlab-integration-test-plan.md` +- **GitLab Integration Spec**: `specs/001-gitlab-support/spec.md` diff --git a/docs/gitlab-token-setup.md b/docs/gitlab-token-setup.md new file mode 100644 index 000000000..6cd06d9e2 --- /dev/null +++ b/docs/gitlab-token-setup.md @@ -0,0 +1,649 @@ +# GitLab Personal Access Token Setup Guide + +This guide provides step-by-step instructions for creating a GitLab Personal Access Token (PAT) for use with vTeam. + +## Overview + +**What is a Personal Access Token?** +A Personal Access Token (PAT) is a secure credential that allows vTeam to access your GitLab repositories on your behalf without needing your password. + +**Why do I need one?** +vTeam uses your PAT to: +- Validate your access to repositories +- Clone repositories for AgenticSessions +- Commit and push changes to your GitLab repositories + +**Security Note**: Your token is stored securely in Kubernetes Secrets and never logged in plaintext. + +--- + +## Creating a GitLab Personal Access Token + +### For GitLab.com + +#### Step 1: Log In to GitLab + +1. Open your browser and navigate to: **https://gitlab.com** +2. Sign in with your GitLab credentials +3. If you don't have an account, click "Register" to create one (free for public and private repositories) + +--- + +#### Step 2: Navigate to Access Tokens Page + +**Option A - Via Profile Menu**: +1. Click your **profile icon** in the top-right corner +2. Select **"Preferences"** from the dropdown menu +3. In the left sidebar, click **"Access Tokens"** + +**Option B - Direct Link**: +1. Navigate directly to: https://gitlab.com/-/profile/personal_access_tokens + +--- + +#### Step 3: Create New Token + +On the Personal Access Tokens page, you'll see a form to create a new token: + +**1. Token Name** +- Enter: `vTeam Integration` (or any descriptive name) +- This helps you identify the token later + +**2. Expiration Date** +- **Recommended**: Set 90 days from today +- **Maximum**: GitLab allows up to 1 year +- **Important**: You'll need to create a new token and reconnect vTeam before expiration + +**3. Select Scopes** (IMPORTANT - must select all of these): + +Check the following scopes: + +- ✅ **`api`** - Full API access + - *Required*: Allows vTeam to access GitLab API endpoints + - Grants read and write access to repositories, merge requests, etc. + +- ✅ **`read_api`** - Read API + - *Required*: Allows read-only access to API + - Used for validation and repository browsing + +- ✅ **`read_user`** - Read user information + - *Required*: Allows vTeam to verify your identity + - Used to get your GitLab username and user ID + +- ✅ **`write_repository`** - Write to repository + - *Required*: Allows vTeam to push changes + - Essential for AgenticSessions to commit and push code + +**DO NOT SELECT** (not needed, grants excessive privileges): +- ❌ `sudo` - Admin-level access +- ❌ `admin_mode` - Administrative operations +- ❌ `create_runner` - Register CI runners +- ❌ `manage_runner` - Manage CI runners + +**4. Click "Create personal access token"** + +--- + +#### Step 4: Copy Your Token + +**CRITICAL STEP** - This is your only chance to copy the token! + +1. After clicking "Create", GitLab will display your new token +2. The token starts with **`glpat-`** followed by random characters + - Example: `glpat-xyz123abc456def789ghi012` + +3. **Copy the entire token** to your clipboard + - Click the copy icon next to the token + - Or select all text and copy manually + +4. **Save the token securely**: + - Paste into a password manager (recommended) + - Or save to a secure text file temporarily + - **DO NOT** share the token or commit it to git + +**Warning**: GitLab will NOT show this token again. If you lose it, you must create a new token. + +--- + +### For Self-Hosted GitLab + +The process is identical to GitLab.com, with these differences: + +#### Step 1: Access Your GitLab Instance + +1. Navigate to your organization's GitLab URL + - Example: `https://gitlab.company.com` +2. Sign in with your corporate credentials + +#### Step 2: Navigate to Access Tokens + +The location depends on your GitLab version: + +**GitLab 14.0+**: +- Click profile icon → Preferences → Access Tokens + +**GitLab 13.x**: +- Click profile icon → Settings → Access Tokens + +**Direct URL**: +- `https://gitlab.company.com/-/profile/personal_access_tokens` +- (Replace `gitlab.company.com` with your instance) + +#### Step 3: Create Token (Same as GitLab.com) + +Follow Steps 3-4 from the GitLab.com instructions above. + +**Important Notes for Self-Hosted**: +- Expiration policy may be enforced by your administrator +- Some scopes may be restricted by your admin +- Contact your GitLab administrator if you encounter permission issues +- Your instance may use different authentication (LDAP, SAML, etc.) but PAT creation is the same + +--- + +## Verifying Your Token + +Before using the token with vTeam, verify it works: + +### Using cURL (Command Line) + +```bash +# Test token validity +curl -H "Authorization: Bearer glpat-your-token-here" \ + https://gitlab.com/api/v4/user + +# For self-hosted: +curl -H "Authorization: Bearer glpat-your-token-here" \ + https://gitlab.company.com/api/v4/user +``` + +**Expected Response** (200 OK): +```json +{ + "id": 123456, + "username": "yourname", + "name": "Your Name", + "state": "active", + "avatar_url": "...", + "web_url": "https://gitlab.com/yourname" +} +``` + +**Error Responses**: + +**401 Unauthorized**: +```json +{ + "message": "401 Unauthorized" +} +``` +- Token is invalid or expired +- Create a new token + +**403 Forbidden**: +```json +{ + "message": "403 Forbidden" +} +``` +- Token lacks required scopes +- Recreate token with `api`, `read_api`, `read_user`, `write_repository` + +--- + +### Verify Token Scopes + +```bash +# Check token scopes (GitLab API) +curl -H "Authorization: Bearer glpat-your-token-here" \ + https://gitlab.com/api/v4/personal_access_tokens/self +``` + +**Expected Response**: +```json +{ + "id": 123456, + "name": "vTeam Integration", + "revoked": false, + "created_at": "2025-11-05T10:00:00.000Z", + "scopes": ["api", "read_api", "read_user", "write_repository"], + "user_id": 789, + "active": true, + "expires_at": "2026-02-05" +} +``` + +**Verify**: +- `"revoked": false` - Token is active +- `"active": true` - Token is not expired +- `"scopes"` includes all required: `api`, `read_api`, `read_user`, `write_repository` + +--- + +### Verify Repository Access + +Test access to a specific repository: + +```bash +# Replace owner/repo with your repository +curl -H "Authorization: Bearer glpat-your-token-here" \ + https://gitlab.com/api/v4/projects/owner%2Frepo + +# Example: +curl -H "Authorization: Bearer glpat-xxx" \ + https://gitlab.com/api/v4/projects/myteam%2Fmyproject +``` + +**Expected Response** (200 OK): +```json +{ + "id": 12345, + "name": "myproject", + "path": "myproject", + "path_with_namespace": "myteam/myproject", + "permissions": { + "project_access": { + "access_level": 30, + "notification_level": 3 + } + } +} +``` + +**Access Levels**: +- `10` = Guest (❌ cannot push) +- `20` = Reporter (❌ cannot push) +- `30` = Developer (✅ can push) +- `40` = Maintainer (✅ can push) +- `50` = Owner (✅ can push) + +**Minimum Required**: `30` (Developer) for AgenticSessions + +--- + +## Using Your Token with vTeam + +Once you have your token, connect it to vTeam: + +### Via vTeam UI + +1. Navigate to **Settings** → **Integrations** +2. Find **GitLab** section +3. Click **"Connect GitLab"** button +4. Paste your token in the **Personal Access Token** field +5. (Optional) For self-hosted: Enter **Instance URL** + - Example: `https://gitlab.company.com` +6. Click **"Connect"** +7. Wait for success confirmation + +### Via API (Command Line) + +**For GitLab.com**: +```bash +curl -X POST http://vteam-backend:8080/api/auth/gitlab/connect \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{ + "personalAccessToken": "glpat-your-gitlab-token-here", + "instanceUrl": "" + }' +``` + +**For Self-Hosted GitLab**: +```bash +curl -X POST http://vteam-backend:8080/api/auth/gitlab/connect \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{ + "personalAccessToken": "glpat-your-gitlab-token-here", + "instanceUrl": "https://gitlab.company.com" + }' +``` + +**Success Response**: +```json +{ + "userId": "user-123", + "gitlabUserId": "456789", + "username": "yourname", + "instanceUrl": "https://gitlab.com", + "connected": true, + "message": "GitLab account connected successfully" +} +``` + +--- + +## Token Management + +### Viewing Your Tokens + +**In GitLab**: +1. Navigate to: https://gitlab.com/-/profile/personal_access_tokens +2. Scroll down to **"Active Personal Access Tokens"** +3. You'll see a table with all your tokens: + - Token name + - Scopes + - Created date + - Last used date + - Expiration date + +**Note**: GitLab shows when a token was last used, helping you identify unused tokens. + +--- + +### Revoking a Token + +**When to Revoke**: +- Token compromised or accidentally exposed +- Token no longer needed +- Replacing with new token (after rotating) + +**How to Revoke**: +1. Navigate to: https://gitlab.com/-/profile/personal_access_tokens +2. Find the token in the **"Active Personal Access Tokens"** table +3. Click the **"Revoke"** button next to the token +4. Confirm revocation + +**Important**: +- Revoked tokens CANNOT be un-revoked +- Any application using the token will immediately lose access +- If you revoked the wrong token, create a new one + +--- + +### Rotating Tokens (Recommended Every 90 Days) + +Token rotation improves security by limiting exposure if a token is compromised. + +**Rotation Process**: + +1. **Create New Token**: + - Follow steps above to create new token + - Use same name with date: `vTeam Integration (Nov 2025)` + - Select same scopes + +2. **Test New Token**: + ```bash + curl -H "Authorization: Bearer glpat-new-token" \ + https://gitlab.com/api/v4/user + ``` + +3. **Update vTeam**: + - Disconnect current GitLab connection in vTeam + - Reconnect with new token + +4. **Verify vTeam Works**: + - Check connection status in vTeam + - Test with a simple AgenticSession + +5. **Revoke Old Token**: + - Go to GitLab Access Tokens page + - Revoke the old token + +**Set a Reminder**: Add calendar reminder 7 days before token expiration. + +--- + +## Troubleshooting + +### Token Not Working with vTeam + +**Problem**: vTeam shows "Invalid token" error + +**Solutions**: +1. **Verify token copied correctly**: + - No extra spaces before/after + - Entire token including `glpat-` prefix + - Check for line breaks if copy-pasted from email + +2. **Check token hasn't expired**: + - Go to GitLab Access Tokens page + - Check expiration date + - Create new token if expired + +3. **Verify token is active**: + ```bash + curl -H "Authorization: Bearer glpat-xxx" \ + https://gitlab.com/api/v4/personal_access_tokens/self + ``` + - Check `"active": true` and `"revoked": false` + +4. **For self-hosted**: Verify instance URL is correct + - Must include `https://` + - No trailing slash + - Example: `https://gitlab.company.com` + +--- + +### Insufficient Permissions Error + +**Problem**: vTeam shows "Insufficient permissions" when pushing + +**Solutions**: +1. **Check token scopes**: + ```bash + curl -H "Authorization: Bearer glpat-xxx" \ + https://gitlab.com/api/v4/personal_access_tokens/self + ``` + +2. **Verify all required scopes**: + - ✅ `api` + - ✅ `read_api` + - ✅ `read_user` + - ✅ `write_repository` ← Often missing! + +3. **Recreate token with correct scopes**: + - Create new token with all scopes + - Update vTeam connection + - Revoke old token + +4. **Check repository access**: + - Verify you're at least Developer on the repository + - For private repos: Check you're a member + +--- + +### Rate Limit Exceeded + +**Problem**: "Rate limit exceeded" error + +**Cause**: GitLab.com limits: +- 300 requests per minute per user +- 10,000 requests per hour per user + +**Solutions**: +1. **Wait**: Limits reset after the time window (1 minute or 1 hour) +2. **Check for loops**: Ensure no automated processes hammering API +3. **For self-hosted**: Contact admin about rate limit configuration + +--- + +### Token Revoked Unexpectedly + +**Possible Causes**: +1. **You revoked it**: Check GitLab audit log +2. **Admin revoked it**: Self-hosted instances allow admin token revocation +3. **Token expired**: Check expiration date +4. **Account issue**: Account suspended or password changed on some GitLab versions + +**Solutions**: +- Create new token +- Contact GitLab admin (for self-hosted) +- Check GitLab account status + +--- + +## Security Best Practices + +### DO ✅ + +1. **Set Expiration Dates** + - Always set an expiration (max 90 days recommended) + - Prevents perpetual access if token compromised + +2. **Use Minimum Required Scopes** + - Only select: `api`, `read_api`, `read_user`, `write_repository` + - Avoid `sudo` and `admin_mode` + +3. **Store Tokens Securely** + - Use password manager (1Password, LastPass, etc.) + - Or secure corporate vault + - Never in git repositories + +4. **Rotate Regularly** + - Every 90 days recommended + - Immediately if compromised + +5. **Use Separate Tokens** + - Different token for vTeam vs other applications + - Easier to identify in audit logs + - Can revoke individually + +6. **Monitor Last Used Date** + - Check GitLab Access Tokens page monthly + - Revoke unused tokens + +### DON'T ❌ + +1. **Never Commit Tokens to Git** + ```bash + # BAD - token exposed in git history! + git commit -m "Added token glpat-xxx to config" + ``` + +2. **Never Share Tokens** + - Each user should have their own token + - Team members need individual vTeam connections + +3. **Never Use Sudo Scope** + - Grants excessive admin privileges + - Not needed for vTeam + +4. **Never Set "No Expiration"** + - Security risk if token leaks + - Always set expiration date + +5. **Never Log Tokens** + - Don't print tokens in application logs + - Don't include in error messages + - vTeam automatically redacts tokens + +6. **Never Hardcode Tokens** + ```python + # BAD - token in source code! + gitlab_token = "glpat-xyz123abc456" + ``` + +--- + +## FAQ + +**Q: How long should my token's expiration be?** +A: **90 days** is recommended. This balances security (shorter is better) with convenience (longer reduces rotation overhead). + +**Q: What if I lose my token?** +A: Create a new token and update vTeam. You cannot retrieve a lost token - GitLab only shows it once during creation. + +**Q: Can I use the same token for multiple vTeam projects?** +A: Yes, one token works for all vTeam projects under your user account. + +**Q: Can multiple team members share one token?** +A: **No**. Each person should create their own token and connect individually to vTeam. This ensures proper audit trails. + +**Q: What's the difference between `api` and `write_repository` scopes?** +A: `api` grants full API access (read + write). `write_repository` specifically grants push access to git repositories. Both are needed. + +**Q: Do I need to create a new token for each repository?** +A: No. One token works for all repositories you have access to. + +**Q: What happens when my token expires?** +A: AgenticSessions will fail with "Authentication failed" error. Create a new token and reconnect to vTeam. + +**Q: Can I extend a token's expiration date?** +A: No. You must create a new token with a new expiration date. + +**Q: How do I know if my token was compromised?** +A: Check "Last Used" date in GitLab. If it shows activity you didn't perform, revoke immediately and create new token. + +**Q: Can administrators see my token?** +A: No. GitLab doesn't show token values to anyone, including admins. However, admins can revoke tokens on self-hosted instances. + +**Q: What's the difference between Personal Access Token and Deploy Token?** +A: Personal Access Tokens are tied to your user account. Deploy Tokens are scoped to specific projects and have limited permissions. vTeam requires Personal Access Tokens. + +**Q: Can I use OAuth instead of PAT?** +A: Not currently. vTeam only supports Personal Access Token authentication for GitLab. + +--- + +## Additional Resources + +**GitLab Official Documentation**: +- [Personal Access Tokens](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) +- [GitLab API Authentication](https://docs.gitlab.com/ee/api/index.html#authentication) +- [Token Security](https://docs.gitlab.com/ee/security/token_overview.html) + +**vTeam Documentation**: +- [GitLab Integration Guide](./gitlab-integration.md) +- [Self-Hosted GitLab Configuration](./gitlab-self-hosted.md) +- [Troubleshooting Guide](./gitlab-integration.md#troubleshooting) + +**Security Resources**: +- [GitLab Security Best Practices](https://docs.gitlab.com/ee/security/) +- [OWASP Token Management](https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html) + +--- + +## Support + +Need help with token creation? + +**For GitLab.com Issues**: +- GitLab Support: https://about.gitlab.com/support/ +- GitLab Forum: https://forum.gitlab.com/ + +**For Self-Hosted GitLab**: +- Contact your GitLab administrator +- Check your organization's GitLab documentation + +**For vTeam Integration Issues**: +- vTeam GitHub Issues: https://github.com/natifridman/vTeam/issues +- vTeam Documentation: [Main README](../README.md) + +--- + +## Quick Reference + +**Required Token Scopes**: +``` +✅ api +✅ read_api +✅ read_user +✅ write_repository +``` + +**Token Format**: +``` +glpat-xxxxxxxxxxxxxxxx +``` + +**Test Token**: +```bash +curl -H "Authorization: Bearer glpat-xxx" \ + https://gitlab.com/api/v4/user +``` + +**Connect to vTeam**: +```bash +curl -X POST http://vteam-backend:8080/api/auth/gitlab/connect \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer " \ + -d '{"personalAccessToken":"glpat-xxx","instanceUrl":""}' +``` + +**Check vTeam Connection**: +```bash +curl -X GET http://vteam-backend:8080/api/auth/gitlab/status \ + -H "Authorization: Bearer " +```