PDS Migration Toolkit

A comprehensive toolkit for migrating Personal Data Servers (PDS) in the AT Protocol/Bluesky ecosystem. This project provides both a desktop GUI application and a web service API for seamless PDS migration operations.

Overview

This toolkit enables users to migrate their data between Personal Data Servers in the AT Protocol network. It consists of three main packages:

• pdsmigration-common: Core library containing migration logic, cryptographic utilities, and shared API functions
• pdsmigration-gui: Cross-platform desktop GUI application built with egui/eframe
• pdsmigration-web: HTTP web service built with Actix-Web, featuring background job processing and Prometheus metrics

Technology Stack

• Language: Rust (1.97.0 toolchain required)
• GUI Framework: egui/eframe with both glow and wgpu rendering backends
• Web Framework: Actix-Web
• Cryptography: secp256k1, multibase encoding
• Data Formats: IPLD, CBOR, JSON
• Metrics: Prometheus
• Testing: Standard Rust testing with mockall, wiremock

Requirements

System Requirements

• Rust Toolchain: 1.97.0 channel with rustfmt and clippy components
• Platform Support:
  • Native: Windows, macOS, Linux

Installation

# Install Rust 1.97.0 (if not already installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup toolchain install 1.97.0
rustup component add rustfmt clippy

Setup & Running

Using Prebuilt Binaries

Each release on the Releases page contains the following prebuilt binaries:

• pdsmigration-gui.exe for Windows
• a .deb package for Debian, Ubuntu and related distributions
• an .rpm package for Fedora, RHEL, openSUSE and related distributions
• an .AppImage to run on other Linux distributions without installation
• pdsmigration-gui-linux-x86_64, a plain binary for other Linux distributions

To build from source, continue with the steps below.

Building from Source

# Clone the repository
git clone https://github.com/NorthskySocial/pds-migration.git
cd pds-migration

# Build all packages
cargo build --release

# Run GUI application
cargo run -p pdsmigration-gui

# Run web service (requires environment setup)
cargo run -p pdsmigration-web

Package-Specific Commands

Desktop GUI Application

# Build and run native GUI
cargo run -p pdsmigration-gui

# Release build
cargo build -p pdsmigration-gui --release

Web Service

# Run web service (development)
cargo run -p pdsmigration-web

# Build for production
cargo build -p pdsmigration-web --release

# Docker deployment
docker build -t pdsmigration .
docker run -p 9090:9090 --env-file .env pdsmigration

Common Library

# Build library
cargo build -p pdsmigration-common

# Run library tests
cargo test -p pdsmigration-common

Environment Variables

Web Service Configuration

Variable	Required	Default	Description
PLC_DIRECTORY	No	https://plc.directory	PLC directory service URL
SERVER_PORT	No	9090	HTTP server port
WORKER_COUNT	No	2	Number of worker threads
CONCURRENT_TASKS_PER_JOB	No	3	Max concurrent blob processes per job
JOB_RETENTION_SECS	No	3600	How long finished jobs are kept in memory before being pruned
ARTIFACT_RETENTION_SECS	No	86400	How long local blob directories and repo CAR files are kept before being deleted
ARTIFACT_GC_INTERVAL_SECS	No	3600	How often the local artifact garbage collector runs
MIGRATION_LIMIT	No	-1	Maximum distinct ongoing migrations; -1 means capacity is unlimited
MIGRATION_TTL_SECS	No	1800	Seconds before an inactive migration lease expires
PDS_RATE_LIMIT_BYPASS_KEY	No		Key to bypass PDS rate limits; sent in the x-ratelimit-bypass header on PDS requests

Example .env file

SERVER_PORT=9090
WORKER_COUNT=4
CONCURRENT_TASKS_PER_JOB=3
PLC_DIRECTORY=https://plc.directory

API Endpoints

The web service provides the following HTTP POST endpoints for migration:

1. /service-auth - Service authentication
2. /create-account - Create new account on target PDS
3. /export-repo - Export repository data
4. /import-repo - Import repository data
5. /export-blobs - Export blob data
6. /upload-blobs - Upload blobs to target PDS
7. /migrate-preferences - Migrate user preferences
8. /request-token - Request authentication token
9. /migrate-plc - Migrate PLC (Personal Data License)
10. /activate-account - Activate migrated account
11. /deactivate-account - Deactivate old account

Additional endpoints:

• /health - Health check endpoint
• /migrations - Migrations capacity status
• /metrics - Prometheus metrics

Testing

Running Tests

# Run all tests
cargo test --workspace

# Run tests for specific package
cargo test -p pdsmigration-common
cargo test -p pdsmigration-gui
cargo test -p pdsmigration-web

# Run tests with output
cargo test --workspace -- --nocapture

Development

Code Quality

• All code must pass cargo fmt --check
• All code must pass cargo clippy
• Warnings are treated as errors in CI
• Use #[tracing::instrument] for debugging critical functions

Build Targets

• Native: Standard desktop platforms
• Docker: Production containerized deployment

CI/CD

GitHub Actions automatically:

• Runs comprehensive tests
• Checks code formatting and linting
• Builds Docker containers
• Publishes to GitHub Container Registry
• Publishes desktop application releases

Desktop Application Releases

A release workflow runs every Monday and Thursday from the state of main.

If there are new commits that touch pdsmigration-common, pdsmigration-gui or the workspace manifests, it builds the Windows executable and the Linux binary, .deb and .rpm packages and AppImage, and a new GitHub release is published with the related commit messages as release notes.

Every release is tagged with the major and minor workspace version and the build date as the patch number.

Contributing

1. Ensure you have Rust 1.97.0 installed
2. Format code: cargo fmt
3. Run tests: cargo test --workspace
4. Check linting: cargo clippy
5. Build all targets: cargo build --workspace