Formalize our usage of the Pulumi configuration

[mehalter]

Currently we have many different methods/models of interacting with the Pulumi configuration and it is fragmented and hard to recover at a given point what is really going on in any arbitrary place in the configuration. This PR is an investigation into formalizing, centralizing, and improving how we handle configuration through the infrastructure.

Plan

This is the current plan that is up to changing at any moment through development and discussion, will be kept up to date.

• Manage all interaction of the Pulumi configuration with a util.config library
• Centralize the idea of the "configuration" rather than placing that information arbitrarily
  • Keep in mind that we want to maintain the ease of adding/removing pieces in the configuration
• Make it so the configuration is interlaced with the defaults and "fully resolved" as step 1 in the infrastructure to make sure it gets the understanding of configuration finalized before the infrastructure starts
  • This is happening at the component level and each component is in charge of the configuration for themselves. If components want to access each others configurations, they should do so by accessing the .config variable from a component that is created and initialized

TO TEST

Run pulumi preview on main and this branch and see that there are no differences

Closes #69

[mehalter]

I just finished a first pass at implementing this PR and I believe everything is at a point where it is ready to start getting reviewed.

Important

This is a pretty significant change and would be best gone through over a call

This takes a very modular and integrated approach to configuration management. It builds a model where the configuration of a component is managed by the component's class itself.

Lifecycle of Configuration

Initialization of a component

During initialization the component fetches the configuration either from Pulumi or if a configuration is passed into it (this part is useful if components built out many other components such as datalakhouses building several tributaries which build several buckets). Each component can define it's own default_config property which is used to fill in missing configuration values that are required for operation.

Accessing configuration within a component

These configuration objects after resolution are provided as CapeConfig objects, under the config property of the component, which have a get function which can be given an arbitrary number of keys to search into the configuration as well as have a fallback, default value that is returned if it is not able to use the exact path of keys provided to resolve a value. This is the main entry point to interacting with the configuration and should be considered the "final configuration source of truth" for the component. If the component creates other components, parts of the configuration may be accessed and passed into the constructor of the other components, each of which will manage their own default configuration and handling the the new CapeConfig objects that it receives.

Accessing configuration across components

There may be times when a component needs to access the configuration of other components. This should be done by accessing configuration directly through an instantiation of said component with their .config property. This is imperative because configuration objects are not considered, "final" or "fully resolved" until the initialization of a component. Accessing configuration of a component before it is initialized indicates semantic blurring of contexts and indicates that something may be logically incorrect.

[thecaffiend]

One stale comment to remove. One thing to explain me. One review to rule them all