http: add StringMatcher in HeaderMatcher and deprecate the old fields (exact, prefix, etc.) #17119

yangminzhu · 2021-06-24T01:03:55Z

Signed-off-by: Yangmin Zhu ymzhu@google.com

Commit Message: Add StringMatcher in HeaderMatcher and deprecate the old fields (exact, prefix, etc.)
Additional Description:
Risk Level: Low
Testing: Unit Tests
Docs Changes:
Release Notes: Added notes for the new field string_match and the deprecation of the old fields (exact, prefix, etc.).
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]
[Optional API Considerations:]

…aderMatcher Signed-off-by: Yangmin Zhu <ymzhu@google.com>

repokitteh-read-only · 2021-06-24T01:04:02Z

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to api/envoy/.
envoyproxy/api-shepherds assignee is @htuch
CC @envoyproxy/api-watchers: FYI only for changes made to api/envoy/.

🐱

Caused by: #17119 was opened by yangminzhu.

see: more, trace.

yangminzhu · 2021-06-24T23:06:18Z

Could someone take a look at this? The envoy-presubmit failed seems not related to the PR, thanks.

soulxu

@yangminzhu the envoy-presubmit doesn't seems your fault. :)

source/common/http/header_utility.cc

test/common/http/header_utility_test.cc

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

api/envoy/config/route/v3/route_components.proto

test/common/http/header_utility_test.cc

htuch

Thanks, seems reasonable, just some minor comments.
/wait

api/envoy/config/route/v3/route_components.proto

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

yangminzhu · 2021-06-29T00:47:15Z

@htuch @markdroth updated to use StringMatcher now, PTAL, thanks.

yangminzhu · 2021-06-29T00:50:22Z

api/envoy/type/matcher/v3/string.proto

@@ -62,8 +62,8 @@ message StringMatcher {
    string contains = 7 [(validate.rules).string = {min_len: 1}];
  }

-  // If true, indicates the exact/prefix/suffix matching should be case insensitive. This has no
-  // effect for the safe_regex match.
+  // If true, indicates the exact/prefix/suffix/contains matching should be case insensitive. This


note: the containts field is also supported but just not documented in the API.

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

markdroth

This approach looks great to me! I'll let Harvey do the API approval.

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

configs/terminate_http2_post.yaml

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

yangminzhu · 2021-06-30T22:59:38Z

@htuch would you mind to take a look at the PR again? I changed to use StringMatcher directly suggested by Mark and updated existing config and code to use it accordingly, thanks.

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

htuch

LGTM, just two small things.

test/tools/router_check/router.cc

test/common/http/header_utility_test.cc

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

htuch

LGTM, just some small test questions.
/wait

htuch · 2021-07-09T00:28:16Z

test/common/http/header_utility_test.cc

@@ -201,7 +201,7 @@ name: test-header
  EXPECT_EQ(HeaderUtility::HeaderMatchType::Present, header_data.header_match_type_);
 }

-TEST(HeaderDataConstructorTest, ExactMatchSpecifier) {
+TEST(HeaderDataConstructorTest, DEPRECATED_FEATURE_TEST(ExactMatchSpecifier)) {


I guess I'm still vaguely curious why this was passing CI (with compile_time_options rejecting deprecated fields)?

I'm not sure, by looking at the code, parseHeaderMatcherFromYaml() calls the TestUtility::loadFromYaml(yaml, header_matcher) which seems essentially called the MessageUtil::loadFromYaml() that does not check for the deprecated fields?

test/common/http/header_utility_test.cc

markdroth · 2021-07-09T21:44:33Z

api/envoy/config/route/v3/route_components.proto

+    ];
+
+    // If specified, header match will be performed based on the string match of the header value.
+    type.matcher.v3.StringMatcher string_match = 13;


@snowp is copying the StringMatcher proto to the xds repo in cncf/xds#8. It might be a good idea to wait for that to land and then use the new type here, since that will save us the trouble of migrating this later.

the current StringMatcher is used everywhere in many places, is it better to wait and change all usages at once? It is already a large change for control plane to use the StringMatcher to replace the deprecated fields, just want to make it less complicated by not depending on the extra xds repo for now.

I believe that @snowp was going to change the existing code that uses StringMatcher to be templated so that it would work with either copy of StringMatcher. Once that's done, I think it should be trivial to use the new type here.

I think this just comes down to a question of timing. Snow, how soon do you think you can land the StringMatcher change, including any necessary templatizing of code in Envoy?

I'm busy the next two days for company perf season, but I'll have more time to work on it starting Wednesday. Early review of #17096 could be helpful in speeding this up to make sure that there is agreement on the direction.

yangminzhu · 2021-07-10T03:39:34Z

I'm not sure why the following tests only failed in CI but passed in my local run, @htuch @mattklein123 @qiwzhang do you might have any ideas about this? I checked the jwt filter code but has no idea how is it even affected by this PR. thanks.

[ RUN      ] Protocols/RemoteJwksIntegrationTest.WithGoodTokenAsyncFetchFast/IPv4_HttpDownstream_Http2Upstream
[2021-07-09 23:02:04.849][15090][critical][assert] [test/integration/http_integration.cc:470] assert failure: 0. Details: Timed out waiting for new connection.

[ RUN      ] Protocols/RemoteJwksIntegrationTest.WithGoodTokenAsyncFetchFast/IPv4_HttpDownstream_Http2Upstream
[       OK ] Protocols/RemoteJwksIntegrationTest.WithGoodTokenAsyncFetchFast/IPv4_HttpDownstream_Http2Upstream (68 ms)
[ RUN      ] Protocols/RemoteJwksIntegrationTest.WithGoodTokenAsyncFetchFast/IPv4_Http2Downstream_HttpUpstream
[2021-07-09 23:02:22.138][16531][critical][assert] [test/integration/http_integration.cc:470] assert failure: 0. Details: Timed out waiting for new connection.

htuch · 2021-07-12T00:40:23Z

@yangminzhu not sure, but with integration tests, machine load and timing can have an impact on flaky tests. Try the advice at https://github.com/envoyproxy/envoy/blob/main/test/integration/README.md#debugging-integration-tests.

yangminzhu · 2021-07-12T23:52:04Z

@yangminzhu not sure, but with integration tests, machine load and timing can have an impact on flaky tests. Try the advice at https://github.com/envoyproxy/envoy/blob/main/test/integration/README.md#debugging-integration-tests.

Thanks for the suggestion. I reproduced the test flaky with --runs_per_test=1000 in a clean repo at main branch which makes me think this PR is unrelated to the test failure. I created #17303 for tracking the issue.

@mattklein123 @htuch Could you approve the PR if there are no other comments so that we can get this merged while @qiwzhang is working on the test flaky? thanks.

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

htuch

LGTM, thanks!

… (exact, prefix, etc.) (envoyproxy#17119) Signed-off-by: Yangmin Zhu <ymzhu@google.com>

http: support ignore_case for exact/prefix/suffic/contain match in He…

2d1ed7f

…aderMatcher Signed-off-by: Yangmin Zhu <ymzhu@google.com>

repokitteh-read-only bot added the api label Jun 24, 2021

repokitteh-read-only bot assigned htuch Jun 24, 2021

soulxu reviewed Jun 25, 2021

View reviewed changes

source/common/http/header_utility.cc Outdated Show resolved Hide resolved

test/common/http/header_utility_test.cc Outdated Show resolved Hide resolved

test/common/http/header_utility_test.cc Outdated Show resolved Hide resolved

yangminzhu added 2 commits June 25, 2021 14:36

update test

e9ba434

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

Merge branch 'main' into header-matcher

548d08c

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

htuch reviewed Jun 27, 2021

View reviewed changes

api/envoy/config/route/v3/route_components.proto Outdated Show resolved Hide resolved

htuch reviewed Jun 27, 2021

View reviewed changes

test/common/http/header_utility_test.cc Outdated Show resolved Hide resolved

htuch requested changes Jun 27, 2021

View reviewed changes

repokitteh-read-only bot added the waiting label Jun 27, 2021

markdroth reviewed Jun 28, 2021

View reviewed changes

api/envoy/config/route/v3/route_components.proto Outdated Show resolved Hide resolved

api/envoy/config/route/v3/route_components.proto Outdated Show resolved Hide resolved

use StringMatcher

ed274b0

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

repokitteh-read-only bot removed the waiting label Jun 29, 2021

yangminzhu commented Jun 29, 2021

View reviewed changes

fix format

affb6b3

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

markdroth reviewed Jun 29, 2021

View reviewed changes

yangminzhu added 3 commits June 29, 2021 10:06

fix format

91baeaa

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

fix test

2eb8966

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

fix test failure in example config

1a1de1e

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

soulxu reviewed Jun 30, 2021

View reviewed changes

configs/terminate_http2_post.yaml Show resolved Hide resolved

fix other examples and usage

b2bcee5

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

yangminzhu requested review from alyssawilk, lizan, mattklein123 and zuercher as code owners June 30, 2021 22:37

fix router_tool_test

36f2acc

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

yangminzhu changed the title ~~http: support ignore_case for exact/prefix/suffic/contain match in HeaderMatcher~~ http: add StringMatcher in HeaderMatcher and deprecate the old fields (exact, prefix, etc.) Jul 1, 2021

htuch reviewed Jul 8, 2021

View reviewed changes

test/tools/router_check/router.cc Outdated Show resolved Hide resolved

test/common/http/header_utility_test.cc Show resolved Hide resolved

address comments

7c05be9

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

htuch reviewed Jul 9, 2021

View reviewed changes

repokitteh-read-only bot added the waiting label Jul 9, 2021

markdroth reviewed Jul 9, 2021

View reviewed changes

Merge branch 'main' into header-matcher

b8fd421

repokitteh-read-only bot removed the waiting label Jul 9, 2021

yangminzhu added 4 commits July 13, 2021 20:07

Merge branch 'main' into header-matcher

9841e14

fix merge issue

92284de

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

Merge branch 'main' into header-matcher

ee35d9d

resolve conflicts

5a92d83

Signed-off-by: Yangmin Zhu <ymzhu@google.com>

htuch approved these changes Jul 15, 2021

View reviewed changes

repokitteh-read-only bot removed the api label Jul 15, 2021

mattklein123 merged commit 68e38cb into envoyproxy:main Jul 15, 2021

yangminzhu deleted the header-matcher branch July 15, 2021 23:14

qiwzhang mentioned this pull request Sep 14, 2021

Replace deprecated HeaderMatch_ExactMatch GoogleCloudPlatform/esp-v2#601

Merged

yangminzhu mentioned this pull request Sep 29, 2021

replace deprecated HeaderMatchSpecifier with HeaderMatcher_StringMatch istio/istio#35409

Closed

leyao-daily pushed a commit to leyao-daily/envoy that referenced this pull request Sep 30, 2021

http: add StringMatcher in HeaderMatcher and deprecate the old fields…

cb77fc4

… (exact, prefix, etc.) (envoyproxy#17119) Signed-off-by: Yangmin Zhu <ymzhu@google.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

http: add StringMatcher in HeaderMatcher and deprecate the old fields (exact, prefix, etc.) #17119

http: add StringMatcher in HeaderMatcher and deprecate the old fields (exact, prefix, etc.) #17119

yangminzhu commented Jun 24, 2021 •

edited

repokitteh-read-only bot commented Jun 24, 2021

yangminzhu commented Jun 24, 2021

soulxu left a comment

htuch left a comment

yangminzhu commented Jun 29, 2021

yangminzhu Jun 29, 2021 •

edited

markdroth left a comment

yangminzhu commented Jun 30, 2021

htuch left a comment

htuch left a comment

htuch Jul 9, 2021

yangminzhu Jul 9, 2021

markdroth Jul 9, 2021

yangminzhu Jul 9, 2021

markdroth Jul 12, 2021

snowp Jul 12, 2021

yangminzhu commented Jul 10, 2021

htuch commented Jul 12, 2021

yangminzhu commented Jul 12, 2021

htuch left a comment

http: add StringMatcher in HeaderMatcher and deprecate the old fields (exact, prefix, etc.) #17119

http: add StringMatcher in HeaderMatcher and deprecate the old fields (exact, prefix, etc.) #17119

Conversation

yangminzhu commented Jun 24, 2021 • edited

repokitteh-read-only bot commented Jun 24, 2021

yangminzhu commented Jun 24, 2021

soulxu left a comment

Choose a reason for hiding this comment

htuch left a comment

Choose a reason for hiding this comment

yangminzhu commented Jun 29, 2021

yangminzhu Jun 29, 2021 • edited

Choose a reason for hiding this comment

markdroth left a comment

Choose a reason for hiding this comment

yangminzhu commented Jun 30, 2021

htuch left a comment

Choose a reason for hiding this comment

htuch left a comment

Choose a reason for hiding this comment

htuch Jul 9, 2021

Choose a reason for hiding this comment

yangminzhu Jul 9, 2021

Choose a reason for hiding this comment

markdroth Jul 9, 2021

Choose a reason for hiding this comment

yangminzhu Jul 9, 2021

Choose a reason for hiding this comment

markdroth Jul 12, 2021

Choose a reason for hiding this comment

snowp Jul 12, 2021

Choose a reason for hiding this comment

yangminzhu commented Jul 10, 2021

htuch commented Jul 12, 2021

yangminzhu commented Jul 12, 2021

htuch left a comment

Choose a reason for hiding this comment

yangminzhu commented Jun 24, 2021 •

edited

yangminzhu Jun 29, 2021 •

edited