-
Notifications
You must be signed in to change notification settings - Fork 4.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Thrift to Metadata filter proto design (#33919)
For apache thrift compatible HTTP requests and responses, this filter parses the thrift metadata and put them into filter dynamic metadata for other filter usage. This is the initial proto design, which refers to other filters like json_to_metadata and payload_to_metadata. Risk Level: low Testing: build Docs Changes: yes #29371 Signed-off-by: kuochunghsu <kuochunghsu@pinterest.com>
- Loading branch information
Showing
18 changed files
with
741 additions
and
0 deletions.
There are no files selected for viewing
Validating CODEOWNERS rules …
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
13 changes: 13 additions & 0 deletions
13
api/envoy/extensions/filters/http/thrift_to_metadata/v3/BUILD
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # DO NOT EDIT. This file is generated by tools/proto_format/proto_sync.py. | ||
|
|
||
| load("@envoy_api//bazel:api_build_system.bzl", "api_proto_package") | ||
|
|
||
| licenses(["notice"]) # Apache 2 | ||
|
|
||
| api_proto_package( | ||
| deps = [ | ||
| "//envoy/extensions/filters/network/thrift_proxy/v3:pkg", | ||
| "@com_github_cncf_xds//udpa/annotations:pkg", | ||
| "@com_github_cncf_xds//xds/annotations/v3:pkg", | ||
| ], | ||
| ) |
190 changes: 190 additions & 0 deletions
190
api/envoy/extensions/filters/http/thrift_to_metadata/v3/thrift_to_metadata.proto
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,190 @@ | ||
| syntax = "proto3"; | ||
|
|
||
| package envoy.extensions.filters.http.thrift_to_metadata.v3; | ||
|
|
||
| import "envoy/extensions/filters/network/thrift_proxy/v3/thrift_proxy.proto"; | ||
|
|
||
| import "google/protobuf/struct.proto"; | ||
|
|
||
| import "xds/annotations/v3/status.proto"; | ||
|
|
||
| import "udpa/annotations/status.proto"; | ||
| import "validate/validate.proto"; | ||
|
|
||
| option java_package = "io.envoyproxy.envoy.extensions.filters.http.thrift_to_metadata.v3"; | ||
| option java_outer_classname = "ThriftToMetadataProto"; | ||
| option java_multiple_files = true; | ||
| option go_package = "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/http/thrift_to_metadata/v3;thrift_to_metadatav3"; | ||
| option (udpa.annotations.file_status).package_version_status = ACTIVE; | ||
| option (xds.annotations.v3.file_status).work_in_progress = true; | ||
|
|
||
| // [#protodoc-title: Thrift-To-Metadata Filter] | ||
| // | ||
| // The Thrift to Metadata filter serves for thrift over HTTP traffic, expecting serialized | ||
| // Thrift request and response bodies in the HTTP payload. It extracts *thrift metadata* from the | ||
| // HTTP body and put them into the *filter metadata*. This is useful for matching load balancer | ||
| // subsets, logging, etc. | ||
| // | ||
| // Thrift to Metadata :ref:`configuration overview <config_http_filters_thrift_to_metadata>`. | ||
| // [#extension: envoy.filters.http.thrift_to_metadata] | ||
|
|
||
| enum Field { | ||
| // The Thrift method name, string value. | ||
| METHOD_NAME = 0; | ||
|
|
||
| // The Thrift protocol name, string value. Values are "binary", "binary/non-strict", and "compact", with "(auto)" suffix if | ||
| // :ref:`protocol <envoy_v3_api_field_extensions.filters.http.thrift_to_metadata.v3.ThriftToMetadata.protocol>` | ||
| // is set to :ref:`AUTO_PROTOCOL<envoy_v3_api_enum_value_extensions.filters.network.thrift_proxy.v3.ProtocolType.AUTO_PROTOCOL>` | ||
| PROTOCOL = 1; | ||
|
|
||
| // The Thrift transport name, string value. Values are "framed", "header", and "unframed", with "(auto)" suffix if | ||
| // :ref:`transport <envoy_v3_api_field_extensions.filters.http.thrift_to_metadata.v3.ThriftToMetadata.transport>` | ||
| // is set to :ref:`AUTO_TRANSPORT<envoy_v3_api_enum_value_extensions.filters.network.thrift_proxy.v3.TransportType.AUTO_TRANSPORT>` | ||
| TRANSPORT = 2; | ||
|
|
||
| // The Thrift message type, singed 16-bit integer value. | ||
| HEADER_FLAGS = 3; | ||
|
|
||
| // The Thrift sequence ID, singed 32-bit integer value. | ||
| SEQUENCE_ID = 4; | ||
|
|
||
| // The Thrift message type, string value. Values in request are "call" and "oneway", and in response are "reply" and "exception". | ||
| MESSAGE_TYPE = 5; | ||
|
|
||
| // The Thrift reply type, string value. This is only valid for response rules. Values are "success" and "error". | ||
| REPLY_TYPE = 6; | ||
| } | ||
|
|
||
| message KeyValuePair { | ||
| // The namespace — if this is empty, the filter's namespace will be used. | ||
| string metadata_namespace = 1; | ||
|
|
||
| // The key to use within the namespace. | ||
| string key = 2 [(validate.rules).string = {min_len: 1}]; | ||
|
|
||
| // When used for on_present case, if value is non-empty it'll be used instead | ||
| // of the field value. | ||
| // | ||
| // When used for on_missing case, a non-empty value must be provided. | ||
| google.protobuf.Value value = 3; | ||
| } | ||
|
|
||
| message FieldSelector { | ||
| option (xds.annotations.v3.message_status).work_in_progress = true; | ||
|
|
||
| // field name to log | ||
| string name = 1 [(validate.rules).string = {min_len: 1}]; | ||
|
|
||
| // field id to match | ||
| int32 id = 2 [(validate.rules).int32 = {lte: 32767 gte: -32768}]; | ||
|
|
||
| // next node of the field selector | ||
| FieldSelector child = 3; | ||
| } | ||
|
|
||
| // [#next-free-field: 6] | ||
| message Rule { | ||
| // The field to match on. If set, takes precedence over field_selector. | ||
| Field field = 1; | ||
|
|
||
| // Specifies that a match will be performed on the value of a field in the thrift body. | ||
| // If set, the whole http body will be buffered to extract the field value, which | ||
| // may have performance implications. | ||
| // | ||
| // It's a thrift over http version of | ||
| // :ref:`field_selector<envoy_v3_api_field_extensions.filters.network.thrift_proxy.filters.payload_to_metadata.v3.PayloadToMetadata.Rule.field_selector>`. | ||
| // | ||
| // See also `payload-to-metadata <https://www.envoyproxy.io/docs/envoy/latest/configuration/other_protocols/thrift_filters/payload_to_metadata_filter>`_ | ||
| // for more reference. | ||
| // | ||
| // Example: | ||
| // | ||
| // .. code-block:: yaml | ||
| // | ||
| // method_name: foo | ||
| // field_selector: | ||
| // name: info | ||
| // id: 2 | ||
| // child: | ||
| // name: version | ||
| // id: 1 | ||
| // | ||
| // The above yaml will match on value of ``info.version`` in the below thrift schema as input of | ||
| // :ref:`on_present<envoy_v3_api_field_extensions.filters.http.thrift_to_metadata.v3.Rule.on_present>` or | ||
| // :ref:`on_missing<envoy_v3_api_field_extensions.filters.http.thrift_to_metadata.v3.Rule.on_missing>` | ||
| // while we are processing ``foo`` method. This rule won't be applied to ``bar`` method. | ||
| // | ||
| // .. code-block:: thrift | ||
| // | ||
| // struct Info { | ||
| // 1: required string version; | ||
| // } | ||
| // service Server { | ||
| // bool foo(1: i32 id, 2: Info info); | ||
| // bool bar(1: i32 id, 2: Info info); | ||
| // } | ||
| // | ||
| FieldSelector field_selector = 2 [(xds.annotations.v3.field_status).work_in_progress = true]; | ||
|
|
||
| // If specified, :ref:`field_selector<envoy_v3_api_field_extensions.filters.http.thrift_to_metadata.v3.Rule.field_selector>` | ||
| // will be used to extract the field value *only* on the thrift message with method name. | ||
| string method_name = 3 [(xds.annotations.v3.field_status).work_in_progress = true]; | ||
|
|
||
| // The key-value pair to set in the *filter metadata* if the field is present | ||
| // in *thrift metadata*. | ||
| // | ||
| // If the value in the KeyValuePair is non-empty, it'll be used instead | ||
| // of field value. | ||
| KeyValuePair on_present = 4; | ||
|
|
||
| // The key-value pair to set in the *filter metadata* if the field is missing | ||
| // in *thrift metadata*. | ||
| // | ||
| // The value in the KeyValuePair must be set, since it'll be used in lieu | ||
| // of the missing field value. | ||
| KeyValuePair on_missing = 5; | ||
| } | ||
|
|
||
| // The configuration for transforming thrift metadata into filter metadata. | ||
| // | ||
| // [#next-free-field: 7] | ||
| message ThriftToMetadata { | ||
| // The list of rules to apply to http request body to extract thrift metadata. | ||
| repeated Rule request_rules = 1; | ||
|
|
||
| // The list of rules to apply to http response body to extract thrift metadata. | ||
| repeated Rule response_rules = 2; | ||
|
|
||
| // Supplies the type of transport that the Thrift proxy should use. Defaults to | ||
| // :ref:`AUTO_TRANSPORT<envoy_v3_api_enum_value_extensions.filters.network.thrift_proxy.v3.TransportType.AUTO_TRANSPORT>`. | ||
| network.thrift_proxy.v3.TransportType transport = 3 | ||
| [(validate.rules).enum = {defined_only: true}]; | ||
|
|
||
| // Supplies the type of protocol that the Thrift proxy should use. Defaults to | ||
| // :ref:`AUTO_PROTOCOL<envoy_v3_api_enum_value_extensions.filters.network.thrift_proxy.v3.ProtocolType.AUTO_PROTOCOL>`. | ||
| // Note that :ref:`TWITTER<envoy_v3_api_enum_value_extensions.filters.network.thrift_proxy.v3.ProtocolType.TWITTER>` is | ||
| // not supported due to deprecation in envoy. | ||
| network.thrift_proxy.v3.ProtocolType protocol = 4 [(validate.rules).enum = {defined_only: true}]; | ||
|
|
||
| // Allowed content-type for thrift payload to filter metadata transformation. | ||
| // Default to ``{"application/x-thrift"}``. | ||
| // | ||
| // Set ``allow_empty_content_type`` if empty/missing content-type header | ||
| // is allowed. | ||
| repeated string allow_content_types = 5 | ||
| [(validate.rules).repeated = {items {string {min_len: 1}}}]; | ||
|
|
||
| // Allowed empty content-type for thrift payload to filter metadata transformation. | ||
| // Default to false. | ||
| bool allow_empty_content_type = 6; | ||
| } | ||
|
|
||
| // Thrift to metadata configuration on a per-route basis, which overrides the global configuration for | ||
| // request rules and responses rules. | ||
| message ThriftToMetadataPerRoute { | ||
| // The list of rules to apply to http request body to extract thrift metadata. | ||
| repeated Rule request_rules = 1; | ||
|
|
||
| // The list of rules to apply to http response body to extract thrift metadata. | ||
| repeated Rule response_rules = 2; | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
99 changes: 99 additions & 0 deletions
99
docs/root/configuration/http/http_filters/_include/thrift-to-metadata-filter.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| static_resources: | ||
| listeners: | ||
| - address: | ||
| socket_address: | ||
| address: 0.0.0.0 | ||
| port_value: 80 | ||
| filter_chains: | ||
| - filters: | ||
| - name: envoy.filters.network.http_connection_manager | ||
| typed_config: | ||
| "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager | ||
| codec_type: AUTO | ||
| stat_prefix: ingress_http | ||
| route_config: | ||
| name: local_route | ||
| virtual_hosts: | ||
| - name: app | ||
| domains: | ||
| - "*" | ||
| routes: | ||
| - match: | ||
| prefix: "/" | ||
| route: | ||
| cluster: versioned-cluster | ||
| http_filters: | ||
| - name: envoy.filters.http.thrift_to_metadata | ||
| typed_config: | ||
| "@type": type.googleapis.com/envoy.extensions.filters.http.thrift_to_metadata.v3.ThriftToMetadata | ||
| request_rules: | ||
| - field: PROTOCOL | ||
| on_present: | ||
| metadata_namespace: envoy.lb | ||
| key: protocol | ||
| on_missing: | ||
| metadata_namespace: envoy.lb | ||
| key: protocol | ||
| value: "unknown" | ||
| - field: TRANSPORT | ||
| on_present: | ||
| metadata_namespace: envoy.lb | ||
| key: transport | ||
| on_missing: | ||
| metadata_namespace: envoy.lb | ||
| key: transport | ||
| value: "unknown" | ||
| response_rules: | ||
| - field: MESSAGE_TYPE | ||
| on_present: | ||
| metadata_namespace: envoy.filters.http.thrift_to_metadata | ||
| key: response_message_type | ||
| on_missing: | ||
| metadata_namespace: envoy.filters.http.thrift_to_metadata | ||
| key: response_message_type | ||
| value: "exception" | ||
| - field: REPLY_TYPE | ||
| on_present: | ||
| metadata_namespace: envoy.filters.http.thrift_to_metadata | ||
| key: response_reply_type | ||
| on_missing: | ||
| metadata_namespace: envoy.filters.http.thrift_to_metadata | ||
| key: response_reply_type | ||
| value: "error" | ||
| - name: envoy.filters.http.router | ||
| typed_config: | ||
| "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router | ||
|
|
||
| clusters: | ||
| - name: versioned-cluster | ||
| type: STRICT_DNS | ||
| lb_policy: ROUND_ROBIN | ||
| lb_subset_config: | ||
| fallback_policy: ANY_ENDPOINT | ||
| subset_selectors: | ||
| - keys: | ||
| - protocol | ||
| - transport | ||
| load_assignment: | ||
| cluster_name: versioned-cluster | ||
| endpoints: | ||
| - lb_endpoints: | ||
| - endpoint: | ||
| address: | ||
| socket_address: | ||
| address: 127.0.0.1 | ||
| port_value: 8080 | ||
| metadata: | ||
| filter_metadata: | ||
| envoy.lb: | ||
| default: "true" | ||
| - lb_endpoints: | ||
| - endpoint: | ||
| address: | ||
| socket_address: | ||
| address: 127.0.0.1 | ||
| port_value: 8081 | ||
| metadata: | ||
| filter_metadata: | ||
| envoy.lb: | ||
| version: "1.0" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
50 changes: 50 additions & 0 deletions
50
docs/root/configuration/http/http_filters/thrift_to_metadata_filter.rst
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| .. _config_http_filters_thrift_to_metadata: | ||
|
|
||
| Envoy Thrift-To-Metadata Filter | ||
| =============================== | ||
| * This filter should be configured with the type URL ``type.googleapis.com/envoy.extensions.filters.http.thrift_to_metadata.v3.ThriftToMetadata``. | ||
| * :ref:`v3 API reference <envoy_v3_api_msg_extensions.filters.http.thrift_to_metadata.v3.ThriftToMetadata>` | ||
|
|
||
| The Thrift to Metadata filter serves for thrift over HTTP traffic, expecting serialized Thrift request and response bodies | ||
| in the HTTP payload. This filter is configured with rules that will be matched against Apache thrift compatible requests and | ||
| responses in HTTP payload. The filter will parse the thrift body, extract *thrift metadata* or *thrift payload*, and add them to | ||
| *dynamic filter metadata* based on the configuration of the rule. | ||
|
|
||
| The *filter metadata* can then be used for load balancing decisions, consumed from logs, etc. | ||
|
|
||
| A typical use case for this filter is to dynamically match a specified thrift method of requests | ||
| with rate limit. For this, thrift method name is attached to the request as dynamic filter metadata which | ||
| would then be used to match a rate limit action on filter metadata. | ||
|
|
||
| Example | ||
| ------- | ||
|
|
||
| A sample filter configuration to route traffic to endpoints based on the presence or | ||
| absence of a version attribute could be: | ||
|
|
||
| .. literalinclude:: _include/thrift-to-metadata-filter.yaml | ||
| :language: yaml | ||
| :lines: 25-55 | ||
| :lineno-start: 25 | ||
| :linenos: | ||
| :caption: :download:`thrift-to-metadata-filter.yaml <_include/thrift-to-metadata-filter.yaml>` | ||
|
|
||
| Statistics | ||
| ---------- | ||
|
|
||
| The thrift to metadata filter outputs statistics in the *http.<stat_prefix>.thrift_to_metadata.* namespace. The :ref:`stat prefix | ||
| <envoy_v3_api_field_extensions.filters.network.http_connection_manager.v3.HttpConnectionManager.stat_prefix>` | ||
| comes from the owning HTTP connection manager. | ||
|
|
||
| .. csv-table:: | ||
| :header: Name, Type, Description | ||
| :widths: 1, 1, 2 | ||
|
|
||
| rq_success, Counter, Total requests that succeed to parse the thrift body. | ||
| rq_mismatched_content_type, Counter, Total requests that mismatch the content type | ||
| rq_no_body, Counter, Total requests without content body | ||
| rq_invalid_thrift_body, Counter, Total requests with invalid thrift body | ||
| resp_success, Counter, Total responses that succeed to parse the thrift body. | ||
| resp_mismatched_content_type, Counter, Total responses that mismatch the content type | ||
| resp_no_body, Counter, Total responses without content body | ||
| resp_invalid_thrift_body, Counter, Total responses with invalid thrift body |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.