S3 Import

S3 Import

Migrate data from Amazon S3, Google Cloud Storage, or S3-compatible storage providers to Cloudflare R2 using the Super Slurper API.

---

Overview

The S3 Import feature enables you to migrate existing data from cloud storage providers to R2 without downloading and re-uploading files. Cloudflare's Super Slurper handles the migration on their infrastructure.

Key Features

• Multi-Provider Support - AWS S3, Google Cloud Storage, and S3-compatible endpoints
• Progress Tracking - Monitor active migrations with real-time status
• Job Management - List, view, and abort migration jobs
• Dashboard Fallback - Link to Cloudflare Dashboard if API creation fails

---

Prerequisites

1. Source Storage Credentials: Access Key ID and Secret Access Key with read permissions
2. Destination R2 Bucket: Must exist before starting migration
3. AWS IAM Policy (for S3): Minimum permissions: s3:Get* and s3:List*

Recommended IAM Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:ListBucket",
        "s3:ListBucketVersions"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR-BUCKET-NAME",
        "arn:aws:s3:::YOUR-BUCKET-NAME/*"
      ]
    }
  ]
}

---

Usage

Starting a New Migration

1. Click "Import from S3" in the bucket management toolbar
2. Select the "New Migration" tab
3. Fill in the source configuration:
   • Storage Provider: AWS S3, Google Cloud Storage, or S3-Compatible
   • Source Bucket Name: Name of the bucket to migrate from
   • Access Key ID and Secret Access Key: Credentials with read access
   • Region: For AWS S3, select the bucket's region
   • Custom Endpoint: Required for S3-compatible providers
   • Bucket Subpath: Optional prefix to migrate only specific files
4. Select the Destination R2 Bucket
5. Optionally enable "Overwrite existing objects"
6. Click "Start Migration"

Monitoring Active Jobs

1. Click the "Active Jobs" tab
2. View progress: objects copied, skipped, failed, and bytes transferred
3. Click "Abort" to stop a running job

Viewing History

The "History" tab shows completed, failed, and aborted migrations.

---

API Endpoints

Method	Endpoint	Description
POST	/api/s3-import/jobs	Create a new migration job
GET	/api/s3-import/jobs	List all migration jobs
GET	/api/s3-import/jobs/:jobId	Get job status and details
POST	/api/s3-import/jobs/:jobId/abort	Abort a running job
GET	/api/s3-import/dashboard-url	Get Cloudflare Dashboard URL

Create Job Request Body

{
  "sourceBucketName": "my-s3-bucket",
  "sourceAccessKeyId": "AKIA...",
  "sourceSecretAccessKey": "...",
  "sourceRegion": "us-east-1",
  "sourceProvider": "aws",
  "destinationBucketName": "my-r2-bucket",
  "bucketSubpath": "optional/prefix/",
  "overwriteExisting": false
}

---

Limitations

• Maximum Object Size: Objects larger than 1 TB are skipped
• Credentials: Passed directly to Cloudflare, not stored locally
• Rate Limits: Subject to Cloudflare API rate limits

---

Troubleshooting

Migration job creation fails

• Verify credentials have proper permissions
• Check the source bucket name and region
• Use the "Open Dashboard" button to create jobs via Cloudflare Dashboard

Objects are skipped

• Files larger than 1 TB are automatically skipped
• Check source bucket permissions for specific files

Connection errors

• Verify network connectivity to source storage
• For S3-compatible providers, ensure the endpoint URL is correct

---

📖 See also:

• Cloudflare Super Slurper Documentation
• AWS IAM Best Practices