ATAC-seq Data Processing Pipeline

A comprehensive Python package for end-to-end ATAC-seq data processing and analysis. This pipeline handles everything from raw read processing to peak calling and cloud integration.

Features

• Core Processing:

  • Read trimming (Cutadapt)
  • Quality filtering
  • Read alignment (Bowtie2/BWA)
  • Duplicate removal
  • Tn5 shift correction
  • Peak calling (MACS2)
  • Quality metrics generation

• Infrastructure:

  • AWS integration for cloud processing
  • Docker support for reproducible environments
  • Automated testing framework

• Utilities:

  • S3 data transfer
  • ENCODE data fetching
  • Parallel processing

Installation

Prerequisites

• Python 3.8+
• Docker (optional)
• AWS CLI (for cloud features)

From source

git clone https://github.com/yourusername/ATACseq-Project.git
cd ATACseq-Project
pip install -e .

Using Docker

docker build -t atacseq .
docker run -it atacseq

Project Structure

ATACseq-Project/
├── src/atacseq/               # Main package
│   ├── processing/            # Core processing modules
│   │   ├── align_reads.py     # Read alignment
│   │   ├── cutadapt_trim.py   # Adapter trimming
│   │   ├── filter_reads.py    # Quality filtering
│   │   ├── peak_calling.py    # MACS2 peak calling
│   │   ├── shift_reads.py     # Tn5 shift correction
│   │   └── *.py               # Other processing steps
│   ├── streaming/             # Data transfer
│   │   ├── encode_fetch.py    # ENCODE data download
│   │   └── s3_upload.py       # AWS S3 upload
│   ├── utils/                 # Utilities
│   │   ├── s3_utils.py        # AWS S3 operations
│   │   └── utils.py           # General utilities
│   ├── api.py                 # Programmatic interface
│   └── __main__.py            # CLI entry point
├── aws/                       # AWS integration
│   ├── ami.py                 # AMI management
│   └── instance.py            # EC2 instance management
├── tests/                     # Test suite
│   ├── test_analysis.py       # Analysis tests
│   ├── test_api.py            # API tests
│   └── test_preprocessing.py  # Preprocessing tests
├── config/
│   └── atac.yml               # Configuration template
├── output/                    # Default output directory
├── pyproject.toml             # Build configuration
├── Makefile                   # Common tasks
└── Dockerfile                 # Container configuration

Usage Examples

Basic Processing Pipeline

from atacseq.processing import run_pipeline

run_pipeline(
    input_fastq="sample.fastq.gz",
    output_dir="results/",
    genome="hg38"
)

AWS Integration

from atacseq.utils.s3_utils import upload_to_s3

upload_to_s3(
    local_path="results/",
    s3_bucket="my-atacseq-bucket",
    s3_prefix="experiment_1/"
)

ENCODE Data Fetch

from atacseq.streaming.encode_fetch import download_encode_data

download_encode_data(
    accession="ENCFF123ABC",
    output_dir="data/"
)

Configuration

Edit atac.yml for pipeline configuration:

reference_genomes:
  hg38: /path/to/hg38
  mm10: /path/to/mm10

aws:
  profile: default
  region: us-east-1

Testing

Run the test suite:

pytest tests/