Move all Go packages under internal/

[apparentlymart]

Aside from some historical exceptions under helper/ (which have since moved to the plugin SDK, all of the Go packages in this repository have always been for Terraform's internal use only, and not supported for importing by other codebases as if Terraform were a library.

However, this codebases predates the addition of Internal Packages in the Go toolchain, and so our packages were not positioned in such a way as to make it clear (using the usual Go idiom) that they are not suitable for external imports. Unfortunately this has meant that several times over the years we've refactored these internal packages and then heard from folks who mistakenly thought we were trying to create a public API surface because we'd broken their external system. That's a reasonable misunderstanding to make given how our repository is structured, but unfortunate that someone would find this out only after having invested time building something against an unsupported library API.

Although we've been using the internal directory for newly-added packages, we've long put off moving our existing packages under internal due to the disruption that would inevitably cause to existing code contributions that are not yet merged, and make it harder to backport commits to older releases.

However, we use tags in this repository to represent official Terraform releases and do so using a syntax that happens to match what the Go toolchain expects for semantic versioning of library modules. Due to that overlap, once we make a tag which looks like an initial semver release we will -- per normal Go idiom and Go toolchain assumptions -- unfortunately also be unintentionally communicating that this is a stable API surface that other Go modules can depend on.

Although there is never a good time for a change like this, it's time to bite the bullet and get this migration done so that we're communicating our intentions in the way the Go community expects, before we have a release tag in this repository that could be misinterpreted as a semantic versioning initial release and thus inadvertently encourage new dependents.

The only exception here is that I've not moved the github.com/hashicorp/terraform/version package to github.com/hashicorp/terraform/internal/version. Although we don't intend for that to be a public interface either, it's pragmatic to leave it where it is because our automatic release processes update the version numbers in that package and so leaving these alone avoids the need for the release process to handle two different possible paths to update, so that we can continue releasing against older major versions if we need to.

This sort of change is not practical to review in a diff-like form, so I can only promise that all I've done here is mechanically move directories around and rewrite import paths. There is no change to any code here aside from import statements, and so this new tree should be functionally equivalent to what came before it.

I'm intending to backport this to v0.15 too so that the current release branch is in the same directory layout as main in case we need to backport further changes.

[jbardin]

LGTM! 😉

[mildwonkey]

WOOHOO!

[apparentlymart]

Backported to v0.15.

[raphink]

@apparentlymart I totally understand the rationale behind this move and respect it.

In our Terraboard project, we rely on several of Terraform's "internal" APIs, in particular to parse Terraform State files (and its various formats over the years).

It makes sense for quite a few projects to be able to parse State files (and even Plans) to integrate with Terraform. Would it be possible for Hashicorp to provide a public API for this (whether or not it ends up being hosted in this repo or another), or do you consider these file formats to be strictly private APIs?

[alisdair]

The state and plan file formats are not considered stable public interfaces. Instead, the JSON output from terraform show -json is the point of integration that we'd recommend. If there is missing information in the JSON representations, we'd be interested in hearing about that via a feature request issue.

[raphink]

Thanks for the reply @alisdair.

In Terraboard, we parse remote State files directly using Go, so having to run a command is very impractical. Would it be possible to provide an official public lib, even if it's outside of Terraform itself (maybe as a dep of Terraform)?

While we could parse the State files ourselves, the format is rather complex so we'll have to import the whole structure and parsing methods, + there's already been 5 different formats for these states, and the (now internal) Terraform lib provides an abstraction for this. For backwards compatibility, we want to support all previous versions of Terraform states so we can import them, but it's a bit counter-productive to copy this whole abstraction logic from internal Terraform lib.

What would you suggest we do?

[apparentlymart]

Our only supported integration point is running Terraform CLI as a separate program. We don't have the resources to also maintain a separate stable library API.

Any approach using a separate library will end up just being a snapshot of the logic that was in Terraform CLI anyway, and not be able to respond automatically to Terraform CLI upgrades. If automatically supporting new versions of Terraform without changes to your system is not important to you then you can indeed feel free to copy the relevant parts of this codebase out into your own codebase. Of course, if you take that path then it'll be up to you to decide when to update your copy if Terraform's own behavior changes in a newer version.

An alternative is to follow a similar principle that Terraform Cloud does: after each successful run it runs terraform show -json to capture a snapshot of the plan or state (depending on what sort of operation it just ran) and saves that snapshot so it'll be available for other downstream features, such as Terraform Cloud's own view of a plan or state, to parse and render. In other words, it's the responsibility of whatever automation is running Terraform to capture the resulting artifacts and upload them to whatever other consumers need it, rather than having those consumers request the data directly from Terraform CLI.

If you'd like to discuss this further, I'd encourage you to open a topic in the community forum. There we can discuss more details about the two strategies above, although as noted above there is no path to Terraform having a Go library interface in addition to its CLI interface for the foreseeable future, because we need to keep our focus on the CLI tool.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.