simplify the backend interface; promote locking to a first class citizen

[steeling]

Hi there,

Looking at promoting locking to a first class function for terraform backends. See #26422

Starting a draft here to allow a fail-fast, in case this is a non-starter or undesirable (this would result in a large refactor, and changes to each backend implementation).

Anyways, I'm thinking that backends should support simple operations, and place the complexity of calling, locking, and coordinating various operations in the main terraform code. The result is backends will only need to implement the following:

// Backend is the minimal interface that must be implemented to enable Terraform.
type Backend interface {
        // Basic state operations a backend must implement.
	Get(workspace string) (*Payload, error)
	Put(workspace string, data []byte) error
	Delete(workspace string) error
	// List worksapces.
	List() (string, error)
	Lock(info *LockInfo) (string, error)
	Unlock(id string) error

        // Meta operations required to validate and create backends
	ConfigSchema() *configschema.Block
	PrepareConfig(cty.Value) (cty.Value, tfdiags.Diagnostics)
	Configure(cty.Value) tfdiags.Diagnostics
}

I understand there are some implementations (consul, any others?) that require a live session for locking, however it is feasible to implement locking without that. For now I'd like to stray away from how we will achieve this, and those specific implementation details, but whether this is desirable for the calling code within terraform, or if there are any other non-starters.

[steeling]

Another reason to pull the logic out of individual providers: #26593

[steeling]

Better yet for the above proposal, would be adding RLock for read locks which would help parallelize those workloads that heavily use terraform_remote_state

[pkolyvas]

@steeling Sadly, we are unlikely to tackle any broad changes to the current state storage backend interface in the next 6 months to a year. Internally we are considering a careful move to a plugin-type architecture as vendor-specific (3rd party) code shouldn't be shipping with core as the product matures. This would mean a radical redesign of the way backends are used (and interface) with core and will be in coordination with other Terraform teams, including our SDK and provider teams.

At the moment, we are focused on getting to a stable 1.0 release. For us, that means identifying problematic areas first. The current remote state storage implementation is workable, even if many of the lessons we have learned over the years have not found roots in our codebase yet.

Getting the existing remote state backends maintained is a big challenge for us at Terraform Core. The effort necessary for significant refactoring of each backend is something we will push for once they are pluggable. Hopefully in that case the advantages would be clear to the individuals, teams and organizations that develop & maintain them. That means mostly leaving the existing backends and backend implementation, as-is for now.

We do deeply appreciate the effort that goes into contributing ideas like this and we hope that as we do start to tackle making pluggable remote state a reality, you'll join us on that adventure.

[steeling]

@pkolyvas thanks for the detailed response! I'm guessing the "plugin" model will simply be a new interface that each plugin must conform to? If so, do you have ideas on what that interface is like, or if the one mentioned above may fit that? Happy to lend a hand in this space!

[hashicorp-cla]

All committers have signed the CLA.