Add customPlugins and unsupportedBuiltInPlugins sections to spire-server

[kfox1111]

This patch enables end users to configure external plugins in the spire-server config. Unsupported internal plugins are not able to be set.

[marcofranssen]

I prefer to have a couple settings in here like we already have.

spire-server:
  nodeAttestor:
    xYZ:
      enabled: true

and same for other plugins.

spire-server:
  upstreamAuthority:
    xYZ:
      enabled: true

We could then loop the keys in those objects and people can still add their own plugins within there.

This probably adds some duplication in templates, but it also makes it more straight forward and less magic to merge things in the templates, so easier to understand and digest.

Futhermore the API will me better documented.

[kfox1111]

I prefer to have a couple settings in here like we already have.

spire-server:
  nodeAttestor:
    xYZ:
      enabled: true

and same for other plugins.

spire-server:
  upstreamAuthority:
    xYZ:
      enabled: true

We could then loop the keys in those objects and people can still add their own plugins within there.

This probably adds some duplication in templates, but it also makes it more straight forward and less magic to merge things in the templates, so easier to understand and digest.

Futhermore the API will me better documented.

There are... dragons there... But seems like there may be them here either way, maybe just less so with the plugin section?...

The issue we're going to hit, is in toYaml'ing the sections. If we're taking values, such as xYZ.enabled and jsoning it over to spire, its going to get confused by settings that are for helm, not for the actual server. We could remove them from the doc before toPrettyJson, but that can get messy too. With plugin being a different section, it allowed for the k8s config and the raw yaml things to be separate. Its all still able to be validated, as its proper yaml in helm values, so its not so bad.

I'll do a quick inventory though and see how bad removing the k8s stuff from the json at runtime would be....

[marcofranssen]

LGTM 🚀

[marcofranssen]

@edwbuck I think I incorporated all the feedback you gave via slack. Thanks again for the review.

Would be good to have the feedback in the PR. I see there is currently still a block by @edwbuck even though there is no change requested in this PR.

code has changed a bit since the review, and it is not current

[edwbuck]

@edwbuck I think I incorporated all the feedback you gave via slack. Thanks again for the review.

Would be good to have the feedback in the PR. I see there is currently still a block by @edwbuck even though there is no change requested in this PR.

My concern was that the configuration section for the item was open ended. Basically it took a YAML object.

There are a number of subtle issues here. First, the config file output is not YAML, it is HCL. The syntax won't mesh up under many scenarios. Also, since we are simply passing a whole object, there isn't checking that the object is an array or an object, or something even more simple (a value).

That latter part can be fixed, but to have valid HCL output, you might need to force this into a multi-line string. Finally, the strings depend on the kind of the custom plugin / non-custom plugin, where some non-custom plugins are obviously without configuration, while others contain standard configurations (memory keyStore vs disk keyStore) that differ from partially required fields as in a custom keyStore.

[kfox1111]

@edwbuck I think I incorporated all the feedback you gave via slack. Thanks again for the review.

Would be good to have the feedback in the PR. I see there is currently still a block by @edwbuck even though there is no change requested in this PR.

My concern was that the configuration section for the item was open ended. Basically it took a YAML object.

There are a number of subtle issues here. First, the config file output is not YAML, it is HCL. The syntax won't mesh up under many scenarios. Also, since we are simply passing a whole object, there isn't checking that the object is an array or an object, or something even more simple (a value).

That latter part can be fixed, but to have valid HCL output, you might need to force this into a multi-line string. Finally, the strings depend on the kind of the custom plugin / non-custom plugin, where some non-custom plugins are obviously without configuration, while others contain standard configurations (memory keyStore vs disk keyStore) that differ from partially required fields as in a custom keyStore.

We need some relief valve for users able to do unsupported stuff until we can support them. They have to sign up for it being unsupported though, which I think is a good tradeoff.

The config file output isn't YAML. We build up the config in YAML since thats easy to do in helm, then convert it to JSON (also easy to do in helm) and spire has the ability to take its config in either HCL or JSON. so we have a direct path via JSON. This avoids all the ugliness of generating HCL. This was done a while ago in #113