terrac

A simple CLI tool to quickly setup a minimal private terraform module registry with your cloud storage service.

• Why
• Design
• Installation
• Tutorial
• Configuration
• Commands
• Backends
• Limitations
• Roadmap
• Development

Why

Sharing terraform module privately is usually necessary when the infrastructure development happens across multiple teams (DevOps vs applications) or multiple repositories (core vs. app infrastructure).

While a module can be downloaded from a git URL, it lacks the support to code versionization and storage management. While other paid solutions (like Terraform Cloud) or open-source solutions (like citizen) exist as a full-feature registry, they are usually very heavy for small teams, in terms of const, features, or maintenance.

The terrac CLI provides a thin layer on your choice of cloud storage service, such as AWS S3 or GCP Cloud Storage, to publish and share private module code. It provides features:

• Publish and download with semver
• Manage storage schema and metadata automatically
• Fully integrated with your infrastructure
• Completely serverless (nothing is hosted)
• Simple commands (npm-style)
• Free and flexible

It is suitable to use as a private terraform registry in small teams (while limitations apply).

Design

The desing of terrac consists of three components:

• Configuration: a JSON file to provide configurations on the module and the cloud storage service
• Commands: a set of commands to provide user interface and interaction
• Backends: a set of standard abstractions for different cloud storage services. All backends expose the same interface to the commands and encapuslate the details of interaction with the remote API.

graph TD;
    Configuration-->Command:publish;
    Command:publish-->Backend:s3;
    Command:publish-->Backend:gcp;
    Command:publish-->Backend:azure;
    Backend:gcp-->GCP;
    Backend:azure-->Azure;
    Backend:s3-->AWS;

Installation

npm

As a node.js application, terrac can be installed with npm or other compatible package managers, such as yarn or pnpm.

npm install -g terrac

manual

Binaries for diefferent platforms are provided as attachment at each GitHub release:

• terrac-linux for Linux
• terrac-macos for MacOS
• terrac-win.exe for Windowns

You can simply download a binary and put it in a directoy included in the PATH environment variable. Note that administrator permission may be required for the action.

bash

The following script automates the manual installation with a bash script in Linux and MacOS.

curl https://raw.githubusercontent.com/haoliangyu/terrac/main/scripts/install.sh | bash

Tutorial

To initialize a module directory and publish

# Create terrac configuration
terrac init

# Pack and publish the module to the remote storage service
terrac publish

# Get the module source URL to use in terraform
terrac get module-name

Configuration

A terrac.json file at the module root directory is used to provide configuration for terrac. It contains two blocks:

• backend to provide the cloud storage configuration
• module to provide the module metadata

The JSON configuration can be populated interactively using the terrac init command and this is an example:

{
  "backend": {
    "type": "s3",
    "bucket": "team-sharing",
    "region": "us-east-1"
  },
  "module": {
    "name": "custom-rds-module",
    "version": "1.6.3"
  }
}

Backend

See the Backends section for more details.

Module

The module object describes the meta information for the module to publish:

• name: module name
• version: module version number. This could be a sematic version or a custom version name.

Commands

• terrac init
• terrac get
• terrac list
• terrac publish

terrac init

Initialize terrac configuration in a directory. It will provide an interactive questionnaire for the selected backend and create the terrac.json configuration file at the local directory.

USAGE
  $ terrac init [--work-directory <value>]

FLAGS
  --work-directory=<value>  [default: .] Root directory of the terraform project

DESCRIPTION
  Initialize terrac configuration in a directory.

See code: src/commands/init.ts

terrac get

Get the module source URL with the given module and version.

USAGE
  $ terrac get NAME [VERSION] [--exact] [--work-directory <value>]

ARGUMENTS
  NAME     Module name
  VERSION  Module version. This could be a version name (like latest), a semver, or a semver component.
           By default, terrac will verify the release with the input version and generate the source URL.
           If it needs to resolve to an exact version, use the --exact flag.

FLAGS
  --exact                   Whether to resolve to an exact version if a named version or a semver component is given
  --work-directory=<value>  [default: .] Root directory of the module configuration

DESCRIPTION
  Get the module source URL of the given module and version.

EXAMPLES
  $ terrac get my-module

  $ terrac get my-module latest

  $ terrac get my-module 1.3.2

  $ terrac get my-module 1.3

  $ terrac get --exact my-module 1.3

The command will try to automatically resolve to the desired version. If the version is found at the backend, it will provide the download URL as output

The release 1.3.2 is found and available at s3::https://s3-us-east-1.amazonaws.com/test-bucket/test-module/1.3.2/module.zip

The output URL can be used in the module source property. For example,

module "example" {
  source = "s3::https://s3-us-east-1.amazonaws.com/test-bucket/test-module/1.3.2/module.zip"
}

See code: src/commands/get.ts

terrac list

List available modules and their versions.

USAGE
  $ terrac list [NAME] [--work-directory <value>]

ARGUMENTS
  NAME  Module name

FLAGS
  --work-directory=<value>       [default: .] Root directory for the terrac configuration file

DESCRIPTION
  List available modules and versions.

EXAMPLES
  $ terrac list

  $ terrac list my-module

If the module name is provided, it will print a list of available versions for the given module. Otherwise, it will list all published modules in the backend.

See code: src/commands/list.ts

terrac publish

Publish a terraform module using the metadata from the terrac.json configuration. If a version name (like beta) is specified, it will publish a single release with the version beta. If a semver is specified, it will publish multile releaes with each semver component. For example, the version 1.3.2 will create three releases (1, 1.3, 1.3.2) and each will have it own downloadable source URL.

A latest release will be updated at every publication.

USAGE
  $ terrac publish [--overwrite] [--work-directory <value>]

FLAGS
  --overwrite               Overwrite a published version with a new package
  --work-directory=<value>  [default: .] Root directory of the module project

DESCRIPTION
  Publish a terraform module.

EXAMPLES
  $ terrac publish

  $ terrac publish --overwrite

Once the publication is successful, it will list all updated releases with the download URLs.

See code: src/commands/publish.ts

Backends

Terrac supports a variety of storage backends for artifact hosting:

• Local
• AWS S3
• GCP Storage
• Azure Block Storage

To set a backend for module publication, update the backend block in the terrac.json configuration file.

Local Directory

The local backend type uses a local directory for package storage. It is typically used for local testing.

// terrac.json
{
  "backend": {
    "type": "local",
    // path to the storage directory
    "path": "./"
  }
}

AWS S3

The s3 backend type uses an AWS S3 bucket for artifact storage. It utilizes the AWS SDK for JavaScript to communicate with AWS and requires proper authentication setup (see documentation).

// terrac.json
{
  "backend": {
    "type": "s3",
    // bucket name
    "bucket": "module-bucket",
    // bucket region
    "region": "us-east-1"
  }
}

GCP Cloud Storage

The gcp backend type uses a GCP Cloud Storage bucket for artifact storage. It uses the GCP Node.js SDK to communicate with GCP and requires proper authentication setup (see documentation).

// terrac.json
{
  "backend": {
    "type": "gcp",
    // bucket name
    "bucket": "module-bucket",
    // project id
    "projectId": "my-module-registry"
  }
}

Azure Blob Storage

The azure backend type uses a Azure Blog Storage container for artifact storage.

// terrac.json
{
  "backend": {
    "type": "azure",
    // Azure storage account name
    "account": "terrac",
    // Azure storage container name
    "conatiner": "terrac-test",
    // [Optional] File name prefix
    "fileNamePrefix": "name/prefix"
  }
}

Limitations

The purpose of terrac is to provide a consistent and simple interface for terraform module hosting with a variety of storage backends. It focuses on the module publication and download. However, it doesn't provide or guarantee some advanced features of modern artifact registry, such as:

• Authentication
• High availability
• High scalability
• Atomic write
• Conflict control
• Permission control

It may be possible to configure a storage backend for these features but this is out of the scope of this project.

Roadmap

• Features

  • Add overwrite option to the publish command
  • Add init command to interatively initialize a module project
  • Add schema check to the terrac configuration file
  • Add support to any custom version name in the get and publish commands
  • Add support to using partial semver in the get command
  • Install with brew
  • Install with bash script
  • Install with standalone executable in different OS
  • Use backend configuration schema to configure init command questionnaire

• Backends

  • GCP Cloud Storage
  • Azure Blob Storage

• Maintenance

  • Add E2E tests with terraform and OpenTofu
  • Unit tests for init command
  • Automate release process to cut GitHub release

Development

In order to run tests locally, it requires the following packages to be installed:

• localstack
• awscli-local