Skip to content

Home

Jump to bottom
Phil Hennel edited this page Oct 20, 2025 · 3 revisions

COPIMA CLI Crawler Wiki

Welcome to the comprehensive documentation for the COPIMA CLI Crawler - a powerful command-line tool for extracting and crawling data from GitLab instances via GraphQL and REST APIs.

📚 Quick Navigation

Getting Started

Architecture & Design

Core Concepts

User Guides

API Reference

Development

Troubleshooting & FAQ

🎯 What This Tool Does

The COPIMA CLI Crawler is designed to systematically extract comprehensive data from GitLab instances. It crawls:

  • Groups & Projects - Organizational structure and metadata
  • Users - User profiles and information
  • Issues & Merge Requests - Development workflow artifacts
  • Pipelines - CI/CD execution history
  • Repository Data - Commits, branches, tags, and file contents
  • Security & Compliance - Vulnerabilities, dependencies, and audit logs

All data is stored in structured JSONL (JSON Lines) format for easy processing and analysis.

🚀 Quick Start

# 1. Run interactive setup
copima-cli-crawler setup

# 2. Authenticate with GitLab
copima-cli-crawler auth

# 3. Start crawling
copima-cli-crawler crawl

Output will be saved to ./output directory by default.

🔑 Key Features

Four-Step Crawling Process

  1. Areas - Gather all accessible groups and projects
  2. Users - Collect user information
  3. Resources - Extract issues, MRs, labels, milestones, etc.
  4. Repository - Crawl commits, branches, tags, and files

Advanced Capabilities

  • Resume Support - Pause and resume crawls without data loss
  • Progress Tracking - Real-time YAML progress reports
  • Deduplication - Automatic prevention of duplicate data
  • Custom Callbacks - Hook into the data processing pipeline
  • Flexible Authentication - PAT tokens or OAuth2 flows
  • Hierarchical Storage - Mirror GitLab's group/project structure
  • E2E Testing - Built-in test framework for validation

📖 Documentation Structure

This Wiki follows a modular structure designed for both beginners and advanced users:

  • Getting Started - For new users who want to quickly start using the tool
  • Architecture - For understanding how the system is designed
  • Core Concepts - For deep dives into specific features
  • Guides - For task-oriented how-to documentation
  • API Reference - For developers integrating or extending the tool
  • Development - For contributors and developers
  • Troubleshooting - For solving problems and debugging

🛠 Technology Stack

This project is built with:

  • Stricli - CLI framework
  • Bun - JavaScript runtime and package manager
  • TypeScript - Type-safe JavaScript
  • Winston - Logging framework
  • Jest - Testing framework
  • GraphQL - Primary API interface
  • REST API - Secondary API for REST-only resources

📝 Contributing to This Wiki

Found an error or want to improve the documentation? Contributions are welcome!

  1. Clone the wiki repository: git clone https://github.com/pdiegmann/copima-cli-crawler.wiki.git
  2. Edit or add Wiki pages (markdown files)
  3. Commit and push your changes
  4. Follow the Contributing Guidelines

📄 License

This project is licensed under the MIT License. See the LICENSE file for details.

🔗 Additional Resources

Last Updated: 2025-10-20

Wiki Version: 1.1.0

Clone this wiki locally