config: allow unknown fields flag (take 2) #4096

kyessenov · 2018-08-09T04:50:12Z

Description: Add a flag allow-unknown-fields that controls whether proto conversion ignores unknown fields. Proto structs are uniformly used for extension configuration. A smaller fix than #4094.

The validation is applied in the following places:

Any conversion (e.g. xDS payloads)
Struct conversion (e.g. filter configs)
bootstrap proto binary

Risk Level: Medium. Allow unknown fields is set to false by default, meaning unknown fields will be rejected by envoy. A missing check for xDS type checking was discovered.

Testing: Unit tests were updated to have it strict by default. Some wrong filter configs were discovered.

Docs Changes: None

Release Notes:
Add a flag allow-unknown-fields to control validation of protobuf configurations for unknown fields.

Fixes #4053

/cc @PiotrSikora

Signed-off-by: Kuat Yessenov <kuat@google.com>

htuch

How come we aren't using proto reflection to prohibit unknown fields in proto input in this version?

htuch · 2018-08-09T14:04:39Z

source/common/protobuf/utility.cc

+  }
+}
+
+void MessageUtil::loadFromJsonLenient(const std::string& json, Protobuf::Message& message) {


Nit: could factor out some of the body of these two to a shared util.

The other way of doing this is to have one function, and make the parameter for leniency be an enum instead of bool. That would be my vote as you would share code and have callsite clarity.

Thanks. I combined the logic into one function which is only called from two places (json convert and a test). The old loadFromJson used in many places in tests, which makes me hesitant to change its signature.

htuch · 2018-08-09T14:06:03Z

tools/protodoc/protodoc.bzl

@@ -92,4 +91,5 @@ proto_doc_aspect = aspect(
            cfg = "host",
        ),
    },
+    implementation = _proto_doc_aspect_impl,


I assume this is just some Buildifier artifact..

Yes, not sure why check_format is eager to refactor all bazel files on my machine.

zuercher · 2018-08-09T16:12:58Z

test/extensions/filters/network/thrift_proxy/integration_test.cc

-                    cluster: "cluster_0"
-                - match:
-                    method_name: "poke"
+                - match: {}


Oops. Could you revert the original and switch method_name to method? That should eliminate the unknown fields. (I'll circle back and add a second cluster to this test and use it for the one of the routes.)

Actually, that'll just make half the tests fail. You can leave this and I'll fix it myself.

Yes, I am getting errors if I put it back...

Signed-off-by: Kuat Yessenov <kuat@google.com>

kyessenov · 2018-08-09T19:33:28Z

source/common/config/utility.h

@@ -52,7 +52,14 @@ class Utility {
    Protobuf::RepeatedPtrField<ResourceType> typed_resources;
    for (const auto& resource : response.resources()) {
      auto* typed_resource = typed_resources.Add();
-      resource.UnpackTo(typed_resource);


note: we were ignoring mismatched type errors during unpacking.

Signed-off-by: Kuat Yessenov <kuat@google.com>

htuch

Looks good, I think this is close to the complete solution, just some refactor cleanups requested. Thanks.

htuch · 2018-08-09T22:48:53Z

source/common/protobuf/utility.cc

@@ -68,6 +78,11 @@ void MessageUtil::loadFromFile(const std::string& path, Protobuf::Message& messa
  if (StringUtil::endsWith(path, ".pb")) {
    // Attempt to parse the binary format.
    if (message.ParseFromString(contents)) {
+      if (!MessageUtil::allow_unknown_fields &&


Can you factor this out to a common utility?

Signed-off-by: Kuat Yessenov <kuat@google.com>

htuch

LGTM, two minor comments.

htuch · 2018-08-10T03:19:48Z

source/common/protobuf/utility.h

+  static void checkUnknownFields(const Protobuf::Message& message) {
+    if (!MessageUtil::allow_unknown_fields &&
+        !message.GetReflection()->GetUnknownFields(message).empty()) {
+      throw EnvoyException("Protobuf Any (type " + message.GetTypeName() + ") has unknown fields");


This is also used when loading from file, so not always Any.

Done, thanks

htuch · 2018-08-10T03:21:32Z

test/common/protobuf/utility_test.cc

  const ProtobufWkt::Struct obj = MessageUtil::keyValueStruct("test_key", "test_value");
  envoy::config::bootstrap::v2::Bootstrap bootstrap;
  EXPECT_NO_THROW(MessageUtil::jsonConvert(obj, bootstrap));
+  MessageUtil::allow_unknown_fields = false;
 }

 TEST(UtilityTest, JsonConvertFail) {


Optionally also add a test to ensure server options have the desired effect.

mattklein123

Generally looks great. Happy to see us iterate on this. Please add docs and release notes for the new CLI flag.

mattklein123 · 2018-08-10T04:02:34Z

source/common/protobuf/utility.cc

+  }
+}
+
+void MessageUtil::loadFromJsonLenient(const std::string& json, Protobuf::Message& message) {


The other way of doing this is to have one function, and make the parameter for leniency be an enum instead of bool. That would be my vote as you would share code and have callsite clarity.

Signed-off-by: Kuat Yessenov <kuat@google.com>

kyessenov

Thanks for the feedback.

kyessenov · 2018-08-10T18:18:14Z

source/common/protobuf/utility.cc

+  }
+}
+
+void MessageUtil::loadFromJsonLenient(const std::string& json, Protobuf::Message& message) {


Thanks. I combined the logic into one function which is only called from two places (json convert and a test). The old loadFromJson used in many places in tests, which makes me hesitant to change its signature.

mattklein123 · 2018-08-10T18:25:24Z

source/common/protobuf/utility.cc

@@ -48,9 +48,16 @@ ProtoValidationException::ProtoValidationException(const std::string& validation
  ENVOY_LOG_MISC(debug, "Proto validation error; throwing {}", what());
 }

+bool MessageUtil::allow_unknown_fields = false;


Just for clarity, can we make this an enum with 2 states? Then the function can take the enum value like I mentioned before (and CLI can convert to enum).

Done, let me know if I need to change the names of anything to conform with the style.

mattklein123 · 2018-08-10T18:25:42Z

source/common/protobuf/utility.cc

+  MessageUtil::loadFromJsonCustom(json, message, false);
+}
+
+void MessageUtil::loadFromJsonCustom(const std::string& json, Protobuf::Message& message,


loadFromJsonEx ? (Not sure what custom means exactly)

Signed-off-by: Kuat Yessenov <kuat@google.com>

htuch · 2018-08-10T18:52:07Z

source/common/protobuf/utility.h

@@ -158,7 +160,19 @@ class MessageUtil {
    return HashUtil::xxHash64(text);
  }

+  static ProtoUnknownFieldsMode proto_unknown_fields;


Nit: For readability, can you do an explicit initialization to Strict here.

Can you give me an example how to initialize a non-const static in the header?
I have explicit initialization in the .cc file.

Ah, missed that, LGTM, thanks.

mattklein123

LGTM other than @htuch comment. Thank you!

htuch

Thanks for this!

* origin/master: (38 commits) test: add tests for corner-cases around sending requests before run() starts or after run() ends. (envoyproxy#4114) perf: reduce the memory usage of LC Trie construction (envoyproxy#4117) test: moving redundant code in websocket_integration_test to utilities (envoyproxy#4127) test: make YamlLoadFromStringFail less picky about error msg. (envoyproxy#4141) rbac: add rbac network filter. (envoyproxy#4083) fuzz: route lookup and header finalization fuzzer. (envoyproxy#4116) Set content-type and content-length (envoyproxy#4113) fault: use FractionalPercent for percent (envoyproxy#3978) test: Fix inverted exact match logic in IntegrationTcpClient::waitForData() (envoyproxy#4134) Added cluster_name to load assignment config for static cluster (envoyproxy#4123) ssl: refactor ContextConfig to use TlsCertificateConfig (envoyproxy#4115) syscall: refactor OsSysCalls for deeper errno latching (envoyproxy#4111) thrift_proxy: fix oneway bugs (envoyproxy#4025) Do not crash when converting YAML to JSON fails (envoyproxy#4110) config: allow unknown fields flag (take 2) (envoyproxy#4096) Use a jittered backoff strategy for handling HdsDelegate stream/connection failures (envoyproxy#4108) bazel: use GCS remote cache (envoyproxy#4050) Add thread local cache of overload action states (envoyproxy#4090) Added TCP healthcheck capabilities to the HdsDelegate (envoyproxy#4079) secret: add secret provider interface and use it for TlsCertificates (envoyproxy#4086) ... Signed-off-by: Snow Pettersen <snowp@squareup.com>

smaller fix

999b785

Signed-off-by: Kuat Yessenov <kuat@google.com>

kyessenov requested a review from zuercher as a code owner August 9, 2018 04:50

kyessenov changed the title ~~config: allow unknown fields flag~~ config: allow unknown fields flag (take 2) Aug 9, 2018

alyssawilk assigned zuercher Aug 9, 2018

htuch reviewed Aug 9, 2018

View reviewed changes

htuch self-assigned this Aug 9, 2018

zuercher reviewed Aug 9, 2018

View reviewed changes

add tests

88c6422

Signed-off-by: Kuat Yessenov <kuat@google.com>

kyessenov commented Aug 9, 2018

View reviewed changes

kyessenov added 2 commits August 9, 2018 12:34

comment fix

2360b5e

Signed-off-by: Kuat Yessenov <kuat@google.com>

fix format

5bdb7cb

Signed-off-by: Kuat Yessenov <kuat@google.com>

htuch requested changes Aug 9, 2018

View reviewed changes

cleanup duplicate code

995b278

Signed-off-by: Kuat Yessenov <kuat@google.com>

htuch reviewed Aug 10, 2018

View reviewed changes

mattklein123 reviewed Aug 10, 2018

View reviewed changes

review

7d51f8e

Signed-off-by: Kuat Yessenov <kuat@google.com>

kyessenov commented Aug 10, 2018

View reviewed changes

mattklein123 reviewed Aug 10, 2018

View reviewed changes

use enum

416f490

Signed-off-by: Kuat Yessenov <kuat@google.com>

htuch reviewed Aug 10, 2018

View reviewed changes

mattklein123 reviewed Aug 10, 2018

View reviewed changes

htuch approved these changes Aug 12, 2018

View reviewed changes

htuch merged commit 2662bf1 into envoyproxy:master Aug 12, 2018

ramaraochavali mentioned this pull request Oct 11, 2018

issue with allow-unknown-fields #4687

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

config: allow unknown fields flag (take 2) #4096

config: allow unknown fields flag (take 2) #4096

kyessenov commented Aug 9, 2018 •

edited

Loading

htuch left a comment

htuch Aug 9, 2018

mattklein123 Aug 10, 2018

kyessenov Aug 10, 2018

htuch Aug 9, 2018

kyessenov Aug 9, 2018

zuercher Aug 9, 2018

zuercher Aug 9, 2018

kyessenov Aug 9, 2018

kyessenov Aug 9, 2018

htuch left a comment

htuch Aug 9, 2018

kyessenov Aug 9, 2018

htuch left a comment

htuch Aug 10, 2018

kyessenov Aug 10, 2018

htuch Aug 10, 2018

mattklein123 left a comment

mattklein123 Aug 10, 2018

kyessenov left a comment

kyessenov Aug 10, 2018

mattklein123 Aug 10, 2018

kyessenov Aug 10, 2018

mattklein123 Aug 10, 2018

kyessenov Aug 10, 2018

htuch Aug 10, 2018

kyessenov Aug 10, 2018

htuch Aug 12, 2018

mattklein123 left a comment

htuch left a comment

config: allow unknown fields flag (take 2) #4096

config: allow unknown fields flag (take 2) #4096

Conversation

kyessenov commented Aug 9, 2018 • edited Loading

htuch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

htuch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

htuch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattklein123 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kyessenov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattklein123 left a comment

Choose a reason for hiding this comment

htuch left a comment

Choose a reason for hiding this comment

kyessenov commented Aug 9, 2018 •

edited

Loading