Singularity

The new pure-go implementation of Singularity provides everything you need to onboard your, or your client's data to Filecoin network, with automatic deal creation and intelligent workflow management.

✨ Key Features

• 🚀 Automatic Deal Creation - Deal schedules created automatically when data preparation completes
• 📦 Data Preparation - Efficient scanning, packing, and CAR file generation
• 🔗 Deal Management - Comprehensive deal scheduling and tracking
• 🏪 Storage Integration - Support for multiple storage backends (local, S3, etc.)
• 📊 Monitoring & Notifications - Real-time status updates and error handling
• 🔧 Flexible Configuration - Extensive customization options for different workflows

🚀 Quick Start

Installation

# Download the latest release
wget https://github.com/data-preservation-programs/singularity/releases/latest/download/singularity-linux-amd64
chmod +x singularity-linux-amd64
sudo mv singularity-linux-amd64 /usr/local/bin/singularity

# Or build from source
git clone https://github.com/data-preservation-programs/singularity.git
cd singularity
go build -o singularity .

Basic Usage

Single command data onboarding with automatic deal creation:

singularity onboard \
  --name "my-dataset" \
  --source "/path/to/data" \
  --auto-create-deals \
  --deal-provider "f01234" \
  --deal-verified \
  --deal-price-per-gb 0.0000001 \
  --start-workers \
  --wait-for-completion

That's it! ✨ This single command will:

1. Create storage connections automatically
2. Set up data preparation with deal parameters
3. Start managed workers to process jobs
4. Automatically progress through scan → pack → daggen
5. Create storage deals when preparation completes
6. Monitor progress until completion

🤖 Auto-Deal System

The Auto-Deal System automatically creates deal schedules when data preparation jobs complete, eliminating manual intervention. The onboard command provides the simplest interface for complete automated workflows.

How It Works

Source Data → Scan → Pack → DAG Gen → Deal Schedule Created ✅

All stages progress automatically with event-driven triggering - no polling or manual monitoring required.

Configuration Options (onboard command)

Flag	Description	Default
--auto-create-deals	Enable automatic deal creation	true
--deal-provider	Storage provider ID (e.g., f01234)	Required
--deal-verified	Create verified deals	false
--deal-price-per-gb	Price per GB per epoch	0
--deal-duration	Deal duration (e.g., "8760h")	535 days
--deal-start-delay	Deal start delay	72h
--validate-wallet	Validate wallets before creating deals	false
--validate-provider	Validate storage provider	false
--start-workers	Start managed workers automatically	true
--wait-for-completion	Monitor until completion	false

Manual Monitoring

# Check preparation status
singularity prep status "my-dataset"

# List all deal schedules
singularity deal schedule list

# Run background processing service
singularity run unified --max-workers 5

📖 Documentation

Read the Full Documentation

🛠️ Advanced Usage

Multiple Storage Providers

Onboard data to different providers with different strategies:

# Hot storage with fast provider
singularity onboard --name "hot-data" --source "/critical/data" \
  --deal-provider "f01234" --deal-price-per-gb 0.000001 --auto-create-deals

# Cold storage with economical provider  
singularity onboard --name "cold-data" --source "/archive/data" \
  --deal-provider "f05678" --deal-price-per-gb 0.0000001 --auto-create-deals

Conditional Auto-Deals

Use validation to control when deals are created:

# Only create deals if wallet has sufficient balance
singularity onboard --name "conditional" --source "/data" --auto-create-deals \
  --deal-provider "f01234" --wallet-validation

# Only create deals if provider is verified  
singularity onboard --name "verified-only" --source "/data" --auto-create-deals \
  --deal-provider "f01234" --sp-validation

Monitoring

# Check preparation status
singularity prep status "my-dataset"

# List all deal schedules
singularity deal schedule list

# Run unified service with monitoring
singularity run unified --max-workers 5

🏗️ Architecture

Simplified Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Onboard        │    │ Worker Manager  │    │ Workflow        │
│  Command        │────▶│                 │────▶│ Orchestrator    │
│                 │    │ • Auto-scaling  │    │                 │
│ • One command   │    │ • Job processing│    │ • Event-driven  │
│ • Full workflow │    │ • Monitoring    │    │ • Auto-progress │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                  │                        │
                                  ▼                        ▼
                    ┌─────────────────────────────┐ ┌──────────────┐
                    │     Auto-Deal Service       │ │ Deal Schedule│
                    │                             │ │    Created   │
                    │ • Check Readiness          │ │      ✅      │
                    │ • Validate Wallets/SPs    │ │              │
                    │ • Create Deal Schedules    │ │              │
                    └─────────────────────────────┘ └──────────────┘

Key Components

• Onboard Command: Single entry point for complete automated workflows
• Worker Manager: Auto-scaling workers that process jobs intelligently
• Workflow Orchestrator: Event-driven progression through data preparation stages
• Auto-Deal Service: Creates deal schedules when preparations complete
• Trigger Service: Handles automatic deal creation logic
• Validation System: Ensures wallets and providers are ready for deals
• Notification System: Provides observability and error reporting

🧪 Testing

# Run auto-deal tests
go test ./service/autodeal/ -v

# Run integration tests
go test ./service/autodeal/ -v -run "TestTrigger"

# Test CLI functionality
singularity onboard --help

🔧 Configuration

Environment Variables

# Lotus connection
export LOTUS_API="https://api.node.glif.io/rpc/v1"
export LOTUS_TOKEN="your-token"

# Database
export DATABASE_CONNECTION_STRING="sqlite:singularity.db"

Runtime Configuration

# Run unified service with custom settings
singularity run unified --max-workers 5

# Run with specific worker configuration
singularity run unified --max-workers 10

🚨 Troubleshooting

Common Issues

Auto-deal not triggering:

• Ensure --auto-create-deals is enabled when using onboard
• Verify wallet is attached: singularity prep list-wallets <prep>
• Check all jobs are complete
• Verify unified service is running: singularity run unified

Deal creation failing:

• Check provider ID is correct
• Ensure wallet has sufficient balance
• Verify network connectivity to Lotus
• Review validation settings

Performance issues:

• Adjust --max-workers in unified service for better throughput
• Monitor database performance and connections
• Use appropriate hardware resources for large datasets

Debug Commands

# Test onboard workflow
singularity onboard --name "test-dataset" --source "/test/data" --auto-create-deals

# View detailed logs
singularity run unified --max-workers 3

# Check preparation status
singularity prep status "my-dataset"

🤝 Migration from Manual Workflows

Existing preparations work unchanged! Auto-deal is completely opt-in:

# Existing workflow (still works)
singularity prep create --name "manual"
singularity deal schedule create --preparation "manual" --provider "f01234"

# New automated workflow
singularity prep create --name "automatic" --auto-create-deals --deal-provider "f01234"

📊 Monitoring & Observability

Key Metrics

• Preparations processed per minute
• Deal schedules created automatically
• Validation success/failure rates
• Error frequencies and types

Log Analysis

# Monitor auto-deal activity
tail -f singularity.log | grep "autodeal-trigger\|auto-deal"

# View successful deal creations
grep "Auto-Deal Schedule Created Successfully" singularity.log

🌟 Benefits

Before Auto-Deal System

• ❌ Manual deal schedule creation required
• ❌ Risk of forgetting to create deals
• ❌ No automation for completed preparations
• ❌ Time-consuming manual monitoring

After Auto-Deal System

• ✅ Zero-touch deal creation for completed preparations
• ✅ Configurable validation and error handling
• ✅ Background monitoring and batch processing
• ✅ Comprehensive logging and notifications
• ✅ Full backward compatibility

🔮 Future Enhancements

• Dynamic provider selection based on reputation/pricing
• Deal success monitoring and automatic retries
• Cost optimization algorithms
• Advanced scheduling (time-based, capacity-based)
• Multi-wallet load balancing
• Integration with deal marketplaces

📞 Support

For issues or questions:

1. Check logs: tail -f singularity.log | grep auto-deal
2. Review notifications: singularity admin notification list
3. Run tests: go test ./service/autodeal/ -v
4. Consult documentation: Full Documentation

Related Projects

• js-singularity - The predecessor that was implemented in Node.js
• js-singularity-import-boost - Automatically import deals to boost for Filecoin storage providers
• js-singularity-browser - A next.js app for browsing singularity made deals
• go-generate-car - The internal tool used by js-singularity to generate car files as well as commp
• go-generate-ipld-car - The internal tool used by js-singularity to regenerate the CAR that captures the unixfs dag of the dataset.

License

Dual-licensed under MIT + Apache 2.0