Figure out and document the recommended way to establish organizational defaults for individual resource types for Config Connector

[bgrant0607]

Copying a good question from slack:

If I have a complex application consisting of several GCP resources such as a bucket, pubsub topic and cloudsql, what kind of granularity in packages and their scope should I be considering?

The additional context here is that I want these GCP resources to have sane defaults for our environment, so I'd likely already create packages for many resources, just for that (and likely accompany things like cloudsql with a loggingsink).

Will everything end up in 1 package, with cloudsql, the bucket, etc packages being cloned from the blueprints I created?

Or will the user deploy each and every blueprint, with all the issues that involves?

I guess my question consists out of two:

• how to provide my own defaults for most resources?
• what does the end result look like from a user perspective / how are things composed?

The existing de facto approach to set defaults for individual resource types has been granular nested subpackages with setters.

Here's an example:
https://github.com/GoogleCloudPlatform/blueprints/tree/main/catalog/bucket

That repo has other examples, of redis, spanner, and so on.

However, I've cautioned strongly against setters #3131, as they are manual and error-prone, and obstruct automation. For instance, setters are an obstacle to operating on blueprints with the package orchestrator, to editing them with a UI, to changing settings with security remediation automation, and to creating blueprints automatically.

And I've recommended that users should not need to create packages for individual resource types.

Packages are pretty heavyweight for that granularity, especially for deployment, and also often degenerate to the "every attribute is a parameter" pattern anyway (see the setter issue above).

I've even suggested that we reconsider whether we need nested subpackages #3343.

Furthermore, users should be able to author resources easily enough, such as via:

• Backstage UI plugin
• Generator CLI command, such as kubectl create or kube-gen.sh.
• CLI with interactive prompts
• Shell with menu-like autocompletion
• IDE plugin, such as Google Cloud Code
• Terminal console
• Mobile app
• Generator KRM function
• Copy/paste from documentation
• Fork a tutorial
• Export from live state

and so on.

Imperative generators have fewer issues than declarative ones (#2528), and authoring is an imperative process.

Enabling easy configuration authoring is a goal of the project. There's a lot that could be done to improve the UX via user interface surfaces #3145: recommended defaults, smart defaults, step-by-step guidance, and so on.

For Config Connector specifically, we need improvements and additions to the function catalog, such as:

• Horizontal functions that change names, labels, namespaces, etc. need to be configured to be aware of cross references and selectors in Config Connector resource types
• We need functions suitable for implementing the variant-constructor pattern
• set-project-id needs improvement: set-project-id function should configure GCP project ID and project Number in KCC resources #2562, set-project-id relies on apply-setter to edit project  #2690
• We need set-gcp-location: New function: set-gcp-location #2658
• We need set-gcp-domain: New function: set-domain #2677
• We need set-gcp-org-id: Add organization-id as an auto-setter #1400
• We might want set-gcp-displaynames (no issue yet AFAIK)
• We should be able to automatically capture the gcloud context: GcloudConfig as Default functionConfig data  #2715
• We probably want a function to set kcc lifecycle directives by default based on resource type, such as for storage resources

cc @diviner524

[bgrant0607]

I've been looking at other tools that support resource or package/module composition, such as terragrunt and kapitan.dev.

There are some common concepts or patterns:

1. A workspace, or container for the composition.
2. Shared properties within the workspace, including some well known properties.
3. Components being composed, with their own customized property values.
4. Targets / environments / variants that more or all of the components are provisioned into, with their own property values.

We may want to think about how to support this pattern more explicitly.

cc @justinsb