TFEngine

TFEngine is a production-ready Terraform automation platform that executes infrastructure operations across multiple cloud providers through an asynchronous task queue system.

Features

• Multi-Cloud Support - Alibaba Cloud, AWS, Tencent Cloud, Huawei Cloud, Kingsoft Cloud, Volcano Engine
• Async Task Processing - Powered by Asynq for reliable task distribution
• Plan/Apply Separation - Safe infrastructure changes with approval workflows
• State Management - Automatic state file persistence and recovery
• Graceful Operations - Support for graceful shutdown and task cancellation
• Resource Discovery - Query cloud resources and sync to CMDB

Architecture

┌─────────────────────────────────────────────────────────────┐
│                      API / Business System                   │
└────────────────┬──────────────────────────┬─────────────────┘
                 │                          │
                 │ Enqueue Task             │ Poll Status
                 │                          │
                 ▼                          ▼
         ┌──────────────┐          ┌──────────────┐
         │ Asynq Queue  │          │   Database   │
         │   (Redis)    │          │   (MySQL)    │
         └──────┬───────┘          └──────▲───────┘
                │                         │
                │ Distribute              │ Update Status
                │                         │ Store Artifacts
                ▼                         │
         ┌──────────────────────────────┬─┘
         │    TFEngine Worker Pool      │
         │  ┌────────────────────────┐  │
         │  │  Task Handler          │  │
         │  │    ↓                   │  │
         │  │  Launcher              │  │
         │  │    ↓                   │  │
         │  │  Workspace Manager     │  │
         │  │    ↓                   │  │
         │  │  Credential Injector   │  │
         │  │    ↓                   │  │
         │  │  Terraform Executor    │  │
         │  │    ↓                   │  │
         │  │  Artifact Collector    │  │
         │  └────────────────────────┘  │
         └──────────────┬────────────────┘
                        │
                        ▼
         ┌─────────────────────────────┐
         │   Multi-Cloud Providers     │
         │  Aliyun | AWS | Tencent     │
         │  Huawei | Kingsoft | Volcano│
         └─────────────────────────────┘

Task Types

Type	Description
PLAN	Execute terraform plan and save plan file for review
APPLY	Apply approved plan file or direct apply
DESTROY	Destroy resources with state restoration support
DISCOVERY	Query cloud resource metadata for CMDB sync

Quick Start

Prerequisites

• Go 1.21+
• Redis
• MySQL
• Terraform 1.6+

Installation

# Clone the repository
git clone https://github.com/zhu327/tfengine.git
cd tfengine

# Build the worker
make build

# Run the worker
./tfengine -config config.yaml

Using Docker

# Build Docker image
make docker-build

# Or pull from registry
docker pull zhu327/tfengine:latest

# Run with Docker
docker run -d \
  -v /path/to/config.yaml:/app/config.yaml \
  -v /path/to/templates:/app/templates \
  zhu327/tfengine:latest

Configuration

Create a config.yaml file:

database:
  host: localhost
  port: 3306
  name: tfengine
  user: tfengine
  password: tfengine

asynq:
  redis_addr: localhost:6379
  concurrency: 5
  shutdown_timeout: 10m
  queues:
    critical: 6
    default: 3
    low: 1

terraform:
  binary_path: /usr/bin/terraform
  plugin_dir: /usr/lib/custom-terraform-plugins
  workspace_root: /data/tasks

templates:
  storage_path: ./templates

logging:
  level: info
  add_source: true
  format: text  # json or text

API Reference

TFEngine provides a RESTful API for task management.

Create Task

curl -X POST http://localhost:8080/api/v1/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "task_type": "PLAN",
    "template_dir": "execution/aws/ec2",
    "cloud_account_id": "account-123",
    "variables": {
      "instance_type": "t3.micro",
      "region": "us-east-1"
    }
  }'

Execute Task

curl -X POST http://localhost:8080/api/v1/tasks/{task_id}/execute \
  -H "Content-Type: application/json" \
  -d '{
    "queue": "default"
  }'

Get Task Status

curl http://localhost:8080/api/v1/tasks/{task_id}

List Task Files

curl http://localhost:8080/api/v1/tasks/{task_id}/files

Download Artifact

curl http://localhost:8080/api/v1/tasks/{task_id}/files/{file_name}/download

Development

# Run tests
make test

# Run tests with coverage
make coverage

# Format code
make fmt

# Lint code
make lint

# Start local environment (requires docker-compose)
make local-up

Roadmap

• Web UI for task management
• Terraform module registry integration
• Cost estimation before apply
• Drift detection
• Policy as Code (OPA/Sentinel)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.