Data source caching

[florisvdg]

Current Terraform Version

Terraform v0.11.11

Use-cases

Some data sources can be slightly more expensive to fetch than others and do not change often or only under specific predictable circumstances. Terraform is doing quite a lot of refreshing of data sources, which is fine for smaller data sources, but can be slow and wasteful for heavier ones. It would be great if providers could check whether a data source needs to be actually fetched.

Attempted Solutions

I would expect that a provider could on every read check whether the data needs refetching, using (computed) attributes from a previous run. However, Terraform seems to always reset the ResourceData on every refresh.

func dataSourceHeavyRead(d *schema.ResourceData, m interface{}) error {
	if needsUpdate(d.Get("last_update")) {
		// Only fetch if data is out of sync
		...
		lastUpdate := ...
		d.Set("last_update", lastUpdate)
	}
	return nil
}

This does not work, because last_update gets reset by Terraform on every read.

Proposal

My proposal would be to not reset the data source attributes on every read. That way, providers have the option to look at the previous state and decide for themselves whether to refetch the data or not.

I don't see how this change could have an impact on existing providers, because they override the attributes on every read anyway.

References

hashicorp/terraform#17034

[paultyng]

Merging with #7

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.