Remove Dependency on Provider Plugins When Emitting Outputs

[yhakbar]

Summary

If possible, remove the requirement to load provider plugins when fetching outputs from state.

Problem Statement

The increased memory utilization and delay from having to load provider plugins in order to fetch outputs significantly impacts performance of OpenTofu.

User-facing description

tofu output has significantly improved in performance due to optimizations in state parsing and output fetching.

Technical Description

One opportunity to improve performance would be to skip parsing state in its entirety when loading outputs. It's possible that optimizations to avoid fetching state prior to determining outputs via ETag headers would allow for outputs to be loaded faster.

Another opportunity would be to embed the schema into the state file, or provide a way for state files to explicitly include which provider plugins have to be loaded so that state files that can be parsed without loading provider plugins can do so.

Rationale and alternatives

The impact of this feature would be a substantial increase in the speed of output fetching, and a reduction in the memory footprint of tofu output executions.

Plan outputs might be usable without having to load provider plugins via tofu show. This would significantly improve the UX of interacting with saved plans in CI, especially if users are downloading provider plugins just to run tofu show.

Downsides

It's entirely possible that provider plugins have to be loaded in order for state files to be parsed, and for outputs to be fetched. I don't know much about the internals of why provider plugins have to be loaded in order to interact with outputs.

Unresolved Questions

I really don't know if any of this is possible. Just documenting the impact of requiring that provider plugins are loaded before outputs are rendered in hopes that it's actually addressable.

Related Issues

No response

Proof of Concept

No response

[janosdebugs]

Thanks @yhakbar , @Yantrio has volunteered to investigate.