Atlantis Framework - AI Assistant Repository

AI assistant framework for Atlantis - OOP-based composable ETL framework with Blackboard pattern, Pydantic metadata, and Pandera data quality.

Specialized agents for building reusable Atlantis components (SCD2Reader, SCD1Writer, etc.), Azure Synapse pipelines, Pydantic metadata models, and Pandera schema validation.

Atlantis Framework Overview

The Atlantis framework is an object-oriented approach to ETL development:

Blackboard Pattern: Components communicate via shared state, no direct dependencies
Composable Building Blocks: SCD2Reader, CustomerProcessor, SCD1Writer as reusable classes
Pydantic Metadata: Type-safe TableMetadata and ProcessMetadata models
Pandera Validation: Schema-based data quality checks
Azure Synapse: Dedicated SQL pools, Spark pools, pipelines
Monorepo: Core Python package, Azure Functions, IaC, Synapse artifacts

Example Atlantis Pipeline

from atlantis import AtlantisPipeline, Blackboard
from atlantis.components import SCD2Reader, CustomerProcessor, SCD1Writer

# Components don't know about each other - only the blackboard
pipeline = AtlantisPipeline(
    config=process_metadata,  # Pydantic ProcessMetadata model
    components=[
        SCD2Reader(source_metadata),
        CustomerProcessor(transformation_rules),
        SCD1Writer(target_metadata)
    ]
)

result = pipeline.execute()  # Components execute via blackboard

What's New: Data Engineering Focus 🚀

This repository has been transformed from a general software development framework to a specialized data engineering AI framework. See DATA_ENGINEERING_TRANSFORMATION.md for full details.

12 Specialized Agents Across 3 Tiers

Tier 1 (Must-Have): Developer, Unit Test, Code Review, Refactor & Cleaner, Schema & Contract
Tier 2 (High-Leverage): Data Quality, SQL Optimization, Dependency Impact, Documentation
Tier 3 (Advanced): Architecture Guard, Security Scan, Metadata

See DATA_ENGINEERING_AGENTS.md for complete agent documentation.

Getting Started

Quick Start

Install Prerequisites
- Python 3.8 or higher
- VS Code with GitHub Copilot extension
- Git
- Connection to your data warehouse (Snowflake, BigQuery, etc.)
Clone and Install

git clone https://github.com/yourusername/ai-data-engineering.git
cd ai-data-engineering

# Install dependencies
pip install -r requirements.txt

# For development (includes pytest, black, flake8)
pip install -r requirements-dev.txt

Configure Package Manager

python scripts/setup_package_manager.py

Start Using Agents

# Write a data pipeline
@developer.md Write a PySpark pipeline to aggregate customer orders

# Create tests
@unit-test.md Create pytest tests for the aggregation function

# Review code
@code-reviewer.md Review this pipeline for idempotency and partitioning

Full Setup Guide

See QUICKSTART.md for detailed setup instructions including:

Data warehouse connection setup
pytest configuration
Great Expectations setup
dbt integration

Cross-Platform Support

This framework fully supports Windows, macOS, and Linux. All hooks and scripts are written in Python for maximum compatibility.

Package Manager Detection

The system automatically detects your preferred Python package manager (pip, poetry, pipenv, or uv) with the following priority:

Environment variable: COPILOT_PACKAGE_MANAGER
Project config: .github-copilot/package-manager.json
pyproject.toml: Tool sections for poetry/pipenv/uv
Lock file: Detection from requirements.txt, poetry.lock, Pipfile.lock, or uv.lock
Global config: ~/.github-copilot/package-manager.json
Fallback: First available package manager

To set your preferred package manager:

# Via environment variable
export COPILOT_PACKAGE_MANAGER=poetry

# Via setup script - global
python scripts/setup_package_manager.py --global poetry

# Via setup script - project
python scripts/setup_package_manager.py --project poetry

# Detect current setting
python scripts/setup_package_manager.py --detect

Or use the /setup-pm command in VS Code.

What's Inside

This repo is a VS Code GitHub Copilot configuration - use it directly or copy components as needed.

everything-github-copilot/
|-- agents/           # Specialized subagents for delegation
|   |-- planner.md           # Feature implementation planning
|   |-- architect.md         # System design decisions
|   |-- tdd-guide.md         # Test-driven development
|   |-- code-reviewer.md     # Quality and security review
|   |-- security-reviewer.md # Vulnerability analysis
|   |-- build-error-resolver.md
|   |-- e2e-runner.md        # Playwright E2E testing
|   |-- refactor-cleaner.md  # Dead code cleanup
|   |-- doc-updater.md       # Documentation sync
|
|-- skills/           # Workflow definitions and domain knowledge
|   |-- coding-standards/           # Language best practices
|   |-- backend-patterns/           # API, database, caching patterns
|   |-- frontend-patterns/          # React, Next.js patterns
|   |-- continuous-learning/        # Auto-extract patterns from sessions
|   |-- strategic-compact/          # Manual compaction suggestions
|   |-- tdd-workflow/               # TDD methodology
|   |-- security-review/            # Security checklist
|   |-- eval-harness/               # Verification loop evaluation
|   |-- verification-loop/          # Continuous verification
|
|-- commands/         # Slash commands for quick execution
|   |-- tdd.md              # /tdd - Test-driven development
|   |-- plan.md             # /plan - Implementation planning
|   |-- e2e.md              # /e2e - E2E test generation
|   |-- code-review.md      # /code-review - Quality review
|   |-- build-fix.md        # /build-fix - Fix build errors
|   |-- refactor-clean.md   # /refactor-clean - Dead code removal
|   |-- learn.md            # /learn - Extract patterns mid-session
|   |-- checkpoint.md       # /checkpoint - Save verification state
|   |-- verify.md           # /verify - Run verification loop
|   |-- setup-pm.md         # /setup-pm - Configure package manager
|
|-- rules/            # Always-follow guidelines
|   |-- security.md         # Mandatory security checks
|   |-- coding-style.md     # Immutability, file organization
|   |-- testing.md          # TDD, 80% coverage requirement
|   |-- git-workflow.md     # Commit format, PR process
|   |-- agents.md           # When to delegate to subagents
|   |-- performance.md      # Model selection, context management
|
|-- scripts/          # Cross-platform Python scripts
|   |-- lib/                     # Shared utilities
|   |   |-- utils.py             # Cross-platform file/path/system utilities
|   |   |-- package_manager.py   # Package manager detection and selection
|   |-- hooks/                   # Hook implementations
|   |   |-- session_start.py     # Load context on session start
|   |   |-- session_end.py       # Save state on session end
|   |   |-- pre_compact.py       # Pre-compaction state saving
|   |   |-- suggest_compact.py   # Strategic compaction suggestions
|   |   |-- evaluate_session.py  # Extract patterns from sessions
|   |-- setup_package_manager.py # Package manager setup script
|
|-- tests/            # Test suite (pytest)
|   |-- lib/                 # Library tests
|   |   |-- test_utils.py
|   |   |-- test_package_manager.py
|
|-- requirements.txt      # Python dependencies
|-- requirements-dev.txt  # Development dependencies
|-- setup.py              # Python package setup

| |-- e2e-runner.md # Playwright E2E testing | |-- refactor-cleaner.md # Dead code cleanup | |-- doc-updater.md # Documentation sync | |-- skills/ # Workflow definitions and domain knowledge | |-- coding-standards/ # Language best practices | |-- backend-patterns/ # API, database, caching patterns | |-- frontend-patterns/ # React, Next.js patterns | |-- continuous-learning/ # Auto-extract patterns from sessions (Longform Guide) | |-- strategic-compact/ # Manual compaction suggestions (Longform Guide) | |-- tdd-workflow/ # TDD methodology | |-- security-review/ # Security checklist | |-- eval-harness/ # Verification loop evaluation (Longform Guide) | |-- verification-loop/ # Continuous verification (Longform Guide) | |-- commands/ # Slash commands for quick execution | |-- tdd.md # /tdd - Test-driven development | |-- plan.md # /plan - Implementation planning | |-- e2e.md # /e2e - E2E test generation | |-- code-review.md # /code-review - Quality review | |-- build-fix.md # /build-fix - Fix build errors | |-- refactor-clean.md # /refactor-clean - Dead code removal | |-- learn.md # /learn - Extract patterns mid-session (Longform Guide) | |-- checkpoint.md # /checkpoint - Save verification state (Longform Guide) | |-- verify.md # /verify - Run verification loop (Longform Guide) | |-- setup-pm.md # /setup-pm - Configure package manager (NEW) | |-- rules/ # Always-follow guidelines (copy to ~/.claude/rules/) | |-- security.md # Mandatory security checks | |-- coding-style.md # Immutability, file organization | |-- testing.md # TDD, 80% coverage requirement | |-- git-workflow.md # Commit format, PR process | |-- agents.md # When to delegate to subagents | |-- performance.md # Model selection, context management | |-- hooks/ # Trigger-based automations | |-- hooks.json # All hooks config (PreToolUse, PostToolUse, Stop, etc.) | |-- memory-persistence/ # Session lifecycle hooks (Longform Guide) | |-- strategic-compact/ # Compaction suggestions (Longform Guide) | |-- scripts/ # Cross-platform Node.js scripts (NEW) | |-- lib/ # Shared utilities | | |-- utils.js # Cross-platform file/path/system utilities | | |-- package-manager.js # Package manager detection and selection | |-- hooks/ # Hook implementations | | |-- session-start.js # Load context on session start | | |-- session-end.js # Save state on session end | | |-- pre-compact.js # Pre-compaction state saving | | |-- suggest-compact.js # Strategic compaction suggestions | | |-- evaluate-session.js # Extract patterns from sessions | |-- setup-package-manager.js # Interactive PM setup | |-- tests/ # Test suite (NEW) | |-- lib/ # Library tests | |-- hooks/ # Hook tests | |-- run-all.js # Run all tests | |-- contexts/ # Dynamic system prompt injection contexts (Longform Guide) | |-- dev.md # Development mode context | |-- review.md # Code review mode context | |-- research.md # Research/exploration mode context | |-- examples/ # Example configurations and sessions | |-- CLAUDE.md # Example project-level config | |-- user-CLAUDE.md # Example user-level config | |-- mcp-configs/ # MCP server configurations | |-- mcp-servers.json # GitHub, Supabase, Vercel, Railway, etc. | |-- marketplace.json # Self-hosted marketplace config (for /plugin marketplace add)


---

## Installation

### Option 1: Install as Plugin (Recommended)

The easiest way to use this repo - install as a Claude Code plugin:

```bash
# Add this repo as a marketplace
/plugin marketplace add affaan-m/everything-claude-code

# Install the plugin
/plugin install everything-claude-code@everything-claude-code

Or add directly to your ~/.claude/settings.json:

{
  "extraKnownMarketplaces": {
    "everything-claude-code": {
      "source": {
        "source": "github",
        "repo": "affaan-m/everything-claude-code"
      }
    }
  },
  "enabledPlugins": {
    "everything-claude-code@everything-claude-code": true
  }
}

This gives you instant access to all commands, agents, skills, and hooks.

Option 2: Manual Installation

If you prefer manual control over what's installed:

# Clone the repo
git clone https://github.com/affaan-m/everything-claude-code.git

# Copy agents to your Claude config
cp everything-claude-code/agents/*.md ~/.claude/agents/

# Copy rules
cp everything-claude-code/rules/*.md ~/.claude/rules/

# Copy commands
cp everything-claude-code/commands/*.md ~/.claude/commands/

# Copy skills
cp -r everything-claude-code/skills/* ~/.claude/skills/

Add hooks to settings.json

Copy the hooks from hooks/hooks.json to your ~/.claude/settings.json.

Configure MCPs

Copy desired MCP servers from mcp-configs/mcp-servers.json to your ~/.claude.json.

Important: Replace YOUR_*_HERE placeholders with your actual API keys.

Key Concepts

Agents

Subagents handle delegated tasks with limited scope. Example:

---
name: code-reviewer
description: Reviews code for quality, security, and maintainability
tools: Read, Grep, Glob, Bash
model: opus
---

You are a senior code reviewer...

Skills

Skills are workflow definitions invoked by commands or agents:

# TDD Workflow

1. Define interfaces first
2. Write failing tests (RED)
3. Implement minimal code (GREEN)
4. Refactor (IMPROVE)
5. Verify 80%+ coverage

Hooks

Hooks fire on tool events. Example - warn about console.log:

{
  "matcher": "tool == \"Edit\" && tool_input.file_path matches \"\\\\.(ts|tsx|js|jsx)$\"",
  "hooks": [{
    "type": "command",
    "command": "#!/bin/bash\ngrep -n 'console\\.log' \"$file_path\" && echo '[Hook] Remove console.log' >&2"
  }]
}

Rules

Rules are always-follow guidelines. Keep them modular:

~/.claude/rules/
  security.md      # No hardcoded secrets
  coding-style.md  # Immutability, file limits
  testing.md       # TDD, coverage requirements

Running Tests

The project includes a comprehensive test suite using pytest:

# Install test dependencies
pip install -r requirements-dev.txt

# Run all tests
pytest

# Run with coverage
pytest --cov=scripts --cov-report=html

# Run specific test file
pytest tests/lib/test_utils.py -v

# Run tests in watch mode (requires pytest-watch)
pytest-watch

Usage with VS Code GitHub Copilot

This configuration is optimized for use with GitHub Copilot in VS Code:

Install GitHub Copilot extension in VS Code

Set environment variables in your shell profile:

export COPILOT_CONFIG_ROOT="/path/to/AI_setup"
export COPILOT_PACKAGE_MANAGER="poetry"  # optional

Configure hooks by referencing hooks/hooks.json
Use agents as reference templates for Copilot Chat
Follow rules defined in the rules/ directory

Migration from JavaScript/Claude Code

This repository has been converted from the original JavaScript-based Claude Code implementation to Python for use with VS Code GitHub Copilot. See MIGRATION.md for detailed migration information.

Key Differences

Language: Python instead of JavaScript
AI Tool: VS Code GitHub Copilot instead of Claude Code
Package Managers: pip, poetry, pipenv, uv instead of npm, pnpm, yarn, bun
Testing: pytest instead of custom Node.js test runner
Hooks: Python scripts instead of Node.js scripts

Contributing

Contributions are welcome and encouraged.

This repo is meant to be a community resource. If you have:

Useful agents or skills
Clever hooks
Better configurations
Improved rules for Python projects

Please contribute! See CONTRIBUTING.md for guidelines.

Ideas for Contributions

Python framework-specific skills (Django, Flask, FastAPI)
Data science workflows (Jupyter, pandas, scikit-learn)
ML/AI patterns (PyTorch, TensorFlow, transformers)
DevOps agents (Docker, Kubernetes, Terraform)
Testing strategies for Python frameworks
Domain-specific knowledge (data engineering, web scraping, APIs)

License

MIT License - feel free to use this in your projects.

Acknowledgments

This project is adapted from the original "Everything Claude Code" by Affaan Mustafa, converted to Python for use with VS Code GitHub Copilot.

Original repository and inspiration: affaan-m/everything-claude-code

Important Notes

Context Window Management

Critical: Don't enable all MCPs at once. Your 200k context window can shrink to 70k with too many tools enabled.

Rule of thumb:

Have 20-30 MCPs configured
Keep under 10 enabled per project
Under 80 tools active

Use disabledMcpServers in project config to disable unused ones.

Customization

These configs work for my workflow. You should:

Start with what resonates
Modify for your stack
Remove what you don't use
Add your own patterns

Star History

Links

Shorthand Guide (Start Here): The Shorthand Guide to Everything Claude Code
Longform Guide (Advanced): The Longform Guide to Everything Claude Code
Follow: @affaanmustafa
zenith.chat: zenith.chat

License

MIT - Use freely, modify as needed, contribute back if you can.

Star this repo if it helps. Read both guides. Build something great.

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
.claude-plugin		.claude-plugin
.claude		.claude
.vscode		.vscode
agents		agents
commands		commands
contexts		contexts
examples		examples
hooks		hooks
mcp-configs		mcp-configs
plugins		plugins
rules		rules
scripts		scripts
skills		skills
tests		tests
.gitignore		.gitignore
ATLANTIS_SETUP_COMPLETE.md		ATLANTIS_SETUP_COMPLETE.md
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
CONVERSION_SUMMARY.md		CONVERSION_SUMMARY.md
DATA_ENGINEERING_AGENTS.md		DATA_ENGINEERING_AGENTS.md
DATA_ENGINEERING_TRANSFORMATION.md		DATA_ENGINEERING_TRANSFORMATION.md
ISSUES_TO_CREATE.md		ISSUES_TO_CREATE.md
MIGRATION.md		MIGRATION.md
QUICKSTART.md		QUICKSTART.md
README.md		README.md
pyproject.toml		pyproject.toml
requirements-dev.txt		requirements-dev.txt
requirements.txt		requirements.txt
setup.cfg		setup.cfg
setup.py		setup.py

AmbushData/AI_setup

Folders and files

Latest commit

History

Repository files navigation

Atlantis Framework - AI Assistant Repository

Atlantis Framework Overview

Example Atlantis Pipeline

What's New: Data Engineering Focus 🚀

12 Specialized Agents Across 3 Tiers

Getting Started

Quick Start

Full Setup Guide

Cross-Platform Support

Package Manager Detection

What's Inside

Option 2: Manual Installation

Add hooks to settings.json

Configure MCPs

Key Concepts

Agents

Skills

Hooks

Rules

Running Tests

Usage with VS Code GitHub Copilot

Migration from JavaScript/Claude Code

Key Differences

Contributing

Ideas for Contributions

License

Acknowledgments

Important Notes

Context Window Management

Customization

Star History

Links

License

About

Resources

Contributing

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages