Skip to content
BranchesTags

Repository files navigation

GithubSearchEngine

Unified GitHub Search, Download, and Processing Engine with Autonomous Processing Capabilities

GithubSearchEngine combines intelligent GitHub search with autonomous repository processing through browser automation.

Features

Search Capabilities

  • Repository Search - Find repos by keywords, language, stars, topics
  • User Search - Discover developers by location, skills, followers
  • Organization Search - Map organizations and their members
  • Code Search - Find specific code patterns across GitHub
  • Secret Hunting - Detect exposed API keys, tokens, and credentials

Processing Capabilities

  • GitHub Repositories - Download, extract, and analyze
  • PyPI Packages - Process Python packages
  • NPM Packages - Process JavaScript/TypeScript packages
  • Rust Crates - Process crates.io packages
  • Go Packages - Process Go modules
  • Local Directories - Analyze local codebases

Pipeline Features

  • Autonomous Pipeline - Search → Download → Process in one command
  • Continuous Mode - Loop through queries automatically
  • Stealth Mode - Evade rate limits with headless browser
  • Token Rotation - Automatic token management
  • Parallel Processing - Concurrent downloads and analysis

Installation

# Clone the repository
git clone https://github.com/yourrepo/githubstealth.git
cd githubstealth/OpenStealth

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install in development mode
pip install -e .

# Install Playwright browsers
playwright install chromium

Quick Start

Search for Repositories

# Basic search
githubstealth search repos "security scanner"

# With filters
githubstealth search repos "vulnerability scanner" -l rust --stars ">100"

# Save results
githubstealth search repos "pentesting tool" -o results.json -f json

Run Search → Process Pipeline

# Search and process top 10 repos
githubstealth pipeline "security automation" -n 10

# Search only (no processing)
githubstealth pipeline "api gateway" --no-process

# Custom output directory
githubstealth pipeline "cli tool" -o ./my-output

Explore Organizations

# Explore an org and process their repos
githubstealth explore microsoft --max-repos 20

# Explore without processing
githubstealth explore google --no-process

Hunt for Secrets

# Hunt secrets in an organization
githubstealth hunt uber lyft airbnb --org

# Hunt secrets in user repos
githubstealth hunt someuser --user -o secrets.json

Process Individual Repos

# Process a single repo
githubstealth process repo rust-lang/rust

# Process multiple repos
githubstealth process batch rust-lang/rust golang/go python/cpython

Run Continuous Pipeline

# Cycle through multiple queries
githubstealth continuous "security tool" "pentesting" "vulnerability" --delay 300

Modes of Operation

API Mode

Uses GitHub API with token rotation:

# Set tokens (multiple for rotation)
export GITHUB_TOKEN="token1,token2,token3"

# Or add to tokens file
echo "ghp_your_token_here" >> ~/.config/githubstealth/.tokens

githubstealth search repos "query" --mode api

Stealth Mode

Uses headless browser with anti-detection:

githubstealth search repos "query" --mode stealth

Auto Mode (Default)

Tries API first, falls back to stealth:

githubstealth search repos "query" --mode auto

Configuration

Configuration file: ~/.config/githubstealth/config.yaml

search:
  max_results: 100
  delay_ms: 200
  max_concurrent: 5

stealth:
  headless: true
  browser: chromium
  enable_webdriver_evasion: true
  enable_cdp_evasion: true

processing:
  max_concurrent_downloads: 5
  max_files_per_repo: 100
  skip_forks: true
  min_stars: 0

pipeline:
  auto_process: true
  process_top_n: 10
  continuous: false
  loop_delay_seconds: 300

output:
  base_dir: ./output
  format: txt
  include_source_code: true

Initialize Configuration

githubstealth config init

View Configuration

githubstealth config show

Python API

import asyncio
from githubstealth import GithubStealthEngine, Config

async def main():
    async with GithubStealthEngine(mode="auto") as engine:
        # Search for repositories
        results = await engine.search_repos(
            "security scanner",
            language="rust",
            stars=">100"
        )
        print(f"Found {len(results.repositories)} repos")
        
        # Process search results
        processed = await engine.process_search_results(results, top_n=5)
        
        for result in processed:
            if result.success:
                print(f"✓ {result.repo.full_name} -> {result.output_path}")
        
        # Or use the integrated pipeline
        pipeline_result = await engine.search_and_process(
            "vulnerability scanner",
            search_type="repos"
        )

asyncio.run(main())

Explore Organizations

async with GithubStealthEngine() as engine:
    result = await engine.explore_org(
        "microsoft",
        process_repos=True,
        max_repos=10
    )
    print(f"Found {len(result['repos'])} repos")

Hunt for Secrets

async with GithubStealthEngine() as engine:
    results = await engine.hunt_secrets(
        ["uber", "lyft", "airbnb"],
        is_org=True,
        output_dir=Path("./secrets")
    )
    print(f"Found {len(results.secret_matches)} potential secrets")

CLI Commands Reference

Search Commands

Command Description
githubstealth search repos Search repositories
githubstealth search users Search users
githubstealth search orgs Search organizations
githubstealth search code Search code
githubstealth search secrets Hunt for secrets

Process Commands

Command Description
githubstealth process repo Process single repository
githubstealth process batch Process multiple repositories

Pipeline Commands

Command Description
githubstealth pipeline Run search → process pipeline
githubstealth continuous Run continuous pipeline loop
githubstealth explore Explore organization
githubstealth hunt Hunt secrets across targets

Config Commands

Command Description
githubstealth config show Display configuration
githubstealth config init Initialize config files
githubstealth config tokens Show token status

Info Commands

Command Description
githubstealth version Show version
githubstealth status Show engine status

Output Format

Processed repositories are saved as comprehensive text files:

# owner/repo

**Type:** GITHUB
**Generated:** 2024-01-15T12:00:00 UTC

## Repository Information
- **URL:** https://github.com/owner/repo
- **Stars:** 1234
- **Forks:** 56
- **Primary Language:** Rust

## README
[README content]

## Repository Structure
├── src/
│   ├── main.rs
│   └── lib.rs
├── Cargo.toml
└── README.md

## Language Statistics
| Language | Files |
|----------|-------|
| Rust     | 15    |
| TOML     | 2     |

## Source Code

### src/main.rs

```rust
fn main() {
    println!("Hello, world!");
}


## Architecture

githubstealth/ ├── init.py # Package entry ├── cli.py # CLI interface ├── config.py # Configuration management ├── engine.py # Main engine (unified interface) ├── pipeline.py # Pipeline orchestrator ├── processor.py # Repository processor └── processors/ # Package processors ├── github.py # GitHub repos ├── pypi.py # PyPI packages ├── npm.py # NPM packages ├── crates.py # Rust crates ├── go.py # Go packages └── local.py # Local directories


## Included Modules

- **githubstealth** - Main unified engine
- **github_search** - GitHub search capabilities
- **pystealth** - Anti-detection browser automation
- **webscrape2anki** - Web scraping utilities

## License

MIT License - see LICENSE file for details.

## Disclaimer

This tool is for educational and authorized security research purposes only. Always ensure you have proper authorization before scanning or processing repositories. Respect GitHub's terms of service and rate limits.

About

A simple unified GitHub Search, Download, and Processing Engine with Autonomous Processing Capabilities

Resources

Readme
MIT license
Activity

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages