node.extensions does not distinguish between compiled-in and configured bootstrap extensions

[golden-shoe]

Description:
Bootstrap extensions that are compiled-in get reported as node.extensions, even when it's not explicitly configured.
However, there are no attributes for the control plane to distinguish between a bootstrap extension that is "compiled-in" vs "compiled-in and configured"

Repro Steps:

1. Extension is pulled in via extensions_build_config.bzl

"envoy.bootstrap.internal_listener": "//source/extensions/bootstrap/internal_listener:config",

2. We want to vend an internal listener on nodes that have envoy.bootstrap.internal_listener configured in bootstrap.json, e.g:

"bootstrapExtensions": [
    {
      "name": "envoy.bootstrap.internal_listener",
      "typedConfig": {
        "@type": "type.googleapis.com/envoy.extensions.bootstrap.internal_listener.v3.InternalListener"
      }
    }
  ],

3. However, some nodes may not have internal listener configured despite running the same binary.
4. Control Plane see "envoy.bootstrap.internal_listener" in node.extensions, but has no way of knowing whether the extension is explicitly configured
5. The node that does not explicit register internal listener then sees error:

[external/envoy/source/extensions/config_subscription/grpc/delta_subscription_state.cc:282] delta config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating           
listener(s) tunneling_encap_listener: error adding listener named 'tunneling_encap_listener': InternalListener bootstrap extension is mandatory 

A few notes

1. Current workaround is to use node metadata to signal which bootstrap extension is configured
2. Control plane should have a way to distinguish whether an extension is configured or not - either via node or extension state.

[golden-shoe]

cc @bplotnick @mathetake @KBaichoo

[tonya11en]

@golden-shoe can you clarify your request? It's unclear where the node identifier is falling short for this.

[golden-shoe]

yeah today node.extensions lists compiled-in extensions, including bootstrap extensions. However, bootstrap extensions being compiled in does not automatically mean they are supported. They also need to be configured in bootstrap.json.

With our example with internal listener, the control plane sees "envoy.bootstrap.internal_listener" in the extension list, sends out an internal listener, then gets rejected because the node didn't actually configure the bootstrap extension in the bootstrap.json. Today there is no way for control plane to tell whether the extension is actually configured in bootstrap.

The request is to somehow embed information in node such that control plane can tell if a bootstrap extension is configured.

A few thoughts

1. Add bootstrap_extensions to node to explicitly indicate configured bootstrap extensions
2. Or, mark the "compiled-in, but not configured" bootstrap extensions to be disabled
3. Or, remove the requirement that internal listener needs to be explicitly configured in order to accept a dynamic listener

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.