Kickoff sds api (#1932)

* update envoy Signed-off-by: Kuat Yessenov <kuat@google.com> * update Envoy's SHA to latest to include the SNI inspecting method, for using SNI in telemetry and policies Signed-off-by: Vadim Eisenberg <vadime@il.ibm.com> * implement destination.principal Signed-off-by: Kuat Yessenov <kuat@google.com> * missing test Signed-off-by: Kuat Yessenov <kuat@google.com> * review Signed-off-by: Kuat Yessenov <kuat@google.com> * add AttributeName::kConnectionRequestedServerName * fix format * add GetRequestedServerName() to TCP CheckData * add building attribute ConnectionRequestedServerName * test building attribute ConnectionRequestedServerName * add GetRequestedServerName() to tcp mixer filter * fix compilation errors * use explicit conversion from absl::string_view to std::string * check that the requested server name is not emtpy in attributes builder * fixed a compilation error * add GetRequestedServerName to http mixer filters (check_data) * add GetRequestedServerName to http MockCheckData * specify the class of a method * add setting connection.requested_server_name to the http attributes * qualify Return by testing:: * use connection_ as a pointer * add explicit conversion from absl::string_view to std::string * add missing mock call * Revert "Add connection requested server name attribute to TCP read filter" * update the API SHA (istio/api#575) * add the connection.requested_server_name attribute to tcp and http filters * add missing parentheses * add AttributeName::kConnectionRequestedServerName * fix cardinality of a mock method * fix the signature of TCP Filter::GetRequestedServerName() * add declarations of GetRequestedServerName() to tcp Filter/http CheckData classes * add peer and local distinction Signed-off-by: Kuat Yessenov <kuat@google.com> * Add connection open event. * update comment. * Replace boolean with enum. * extract origin IP Signed-off-by: Kuat Yessenov <kuat@google.com> * Update Envoy SHA to latest. (#1852) Signed-off-by: Piotr Sikora <piotrsikora@google.com> * Revert "Update Envoy SHA to latest. (#1852)" (#1854) This reverts commit f85f49c. * update api sha (#1855) * update api sha * update api sha * Update Envoy SHA to latest. (#1857) * Update Envoy SHA to latest. Signed-off-by: Piotr Sikora <piotrsikora@google.com> * review: install pkg-config on CircleCI. Signed-off-by: Piotr Sikora <piotrsikora@google.com> * review: install pkg-config on build image. Signed-off-by: Piotr Sikora <piotrsikora@google.com> * Update Envoy SHA for WebSocket fix (#1859) Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com> * Revert "Update Envoy SHA for WebSocket fix (#1859)" (#1860) This reverts commit f47be07. * Update Envoy SHA to latest. (#1862) Signed-off-by: Piotr Sikora <piotrsikora@google.com> * Update istio.dep as valid json (#1863) * Update software in the build image used by CircleCI. (#1874) Signed-off-by: Piotr Sikora <piotrsikora@google.com> * Improve build times on CircleCI. (#1875) 1. Stop trashing build's cache. 2. Split ASan and TSan tests into separate targets. 3. Fix output directory, so that caching works on macOS. Before: - build : 49 mins - macos : 30 mins After: - build : 27 mins - linux_asan: 20 mins - linux_tsan: 16 mins - macos : 30 mins After (with warm cache): - build : 3 mins - linux_asan: 4 mins - linux_tsan: 2 mins - macos : 5 mins Fixes #1815. Signed-off-by: Piotr Sikora <piotrsikora@google.com> * Install ninja in the build image used by CircleCI. (#1888) Signed-off-by: Piotr Sikora <piotrsikora@google.com> * merge 1.0 branch to master (#1885) * Reduce log level for jwt filter (#1866) * Update_Dependencies (#1873) * Correctly clean up headers used for payload from JWT authentication (#1879) * Correctly clean up headers used for payload from JWT authentication * Clang * Update_Dependencies (#1883) * destination.principal derivation fix (#1884) * fix attribute extraction Signed-off-by: Kuat Yessenov <kuat@google.com> * seed mock Signed-off-by: Kuat Yessenov <kuat@google.com> * merge 1.0 to master * Update API SHA (#1891) * add needed dependencies for circle ci * Remove fallback for attribute builder when authn filter is not available (#1887) * Update Envoy SHA to latest with ORIGINAL_DST_LB fix. (#1894) Signed-off-by: Piotr Sikora <piotrsikora@google.com> * Using request dynamic metadata to pass data from JWT filter to authn filter. (#1893) * Using dynamicMetadata to pass data between filters instead of headers * Lint * Populate authn result to dynamic data only. * Integration test for authn * Clean up and verify all tests * Remove unused test configs * Address reviews * Lint * Add the groups claim to the attribute request.auth.groups (#1896) * Add the groups JWT claims to the attribute request.auth.groups * Fix lint errors * Simplify the code * Fix lint error * Simplify the code * Add a test * Fix the test error * add context.proxy_error_code to report attributes (#1897) * Add integration test with JWT+AuthN+Mixer filter chain (#1899) * Add integration test with JWT+AuthN+Mixer filter chain * Lint * Rename helper function * Lint * Review * send rbac shadow policies metrics to mixer (#1900) * send rbac shadow policies metrics to mixer * rename shadow -> permissive * address comments * address comments * address comments * Use dynamic metadata to for authentication filter output. (#1901) * Use dynamic metadata to for authentication filter output. * Clean up comments. * Lint * Reviews * Lint * Fix test * Remove mis-type * Lint * Add two new attributes: request.url_path and request.queries (#1837) * Add two new attributes: request.url_path and request.queries * Update api in repositories.bzl * update ENVOY_SHA (#1904) * update ENVOY_SHA * change WORKSPACE file * skip empty sni (#1909) Signed-off-by: Kuat Yessenov <kuat@google.com> * add rbac filter to istio http integration test. (#1907) * add rbac filter to istio http integration test. * rename issueer for rbac test. * Fix macOS build on CircleCI. (#1916) Apparently, automake is now installed automatically, which broke the brew install step. Signed-off-by: Piotr Sikora <piotrsikora@google.com> * Install clang-format in the build image used by CircleCI. (#1917) Signed-off-by: Piotr Sikora <piotrsikora@google.com> * Provide source version information in the binary. (#1915) Before: $ bazel-bin/src/envoy/envoy --version bazel-bin/src/envoy/envoy version: 0/1.8.0-dev//DEBUG After: $ bazel-bin/src/envoy/envoy --version bazel-bin/src/envoy/envoy version: f315a32fc7c6f727fc9645cc1ca27d4160c1d0e0/1.8.0-dev/Clean/DEBUG Fixes #1803. Signed-off-by: Piotr Sikora <piotrsikora@google.com> * fixed broken links to dev guide and contribution guide (#1913) * Update clang to 6.0 and use it for release binaries. (#1914) Signed-off-by: Piotr Sikora <piotrsikora@google.com> * Update Envoy SHA to latest with LcTrie optimizations. (#1918) Signed-off-by: Piotr Sikora <piotrsikora@google.com> * add debug logs for collecting rbac attributes (#1922) * populate source user without authn (#1926) Signed-off-by: Kuat Yessenov <kuat@google.com> * update API sha. (#1927) * Add a check cache test for string map sub keys (#1931) Signed-off-by: Wayne Zhang <qiwzhang@google.com> * update Envoy SHA
istio · Aug 21, 2018 · 99222fa · 99222fa
1 parent b7b73ff
commit 99222fa
Show file tree

Hide file tree

Showing 92 changed files with 2,052 additions and 1,327 deletions.
diff --git a/.circleci/Dockerfile b/.circleci/Dockerfile
@@ -10,24 +10,24 @@ FROM circleci/openjdk:latest
 
 # clang is used for TSAN and ASAN tests
 RUN sudo sh -c 'curl http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -'
-RUN sudo sh -c 'echo "deb http://apt.llvm.org/stretch/ llvm-toolchain-stretch-5.0 main" > /etc/apt/sources.list.d/llvm.list'
+RUN sudo sh -c 'echo "deb http://apt.llvm.org/stretch/ llvm-toolchain-stretch-6.0 main" > /etc/apt/sources.list.d/llvm.list'
 
 RUN sudo apt-get update && \
     sudo apt-get -y install \
-    wget software-properties-common make cmake python python-pip \
+    wget software-properties-common make cmake python python-pip pkg-config \
     zlib1g-dev bash-completion bc libtool automake zip time g++-6 gcc-6 \
-    clang-5.0 rsync
+    clang-6.0 clang-format-6.0 rsync ninja-build
 
 # ~100M, depends on g++, zlib1g-dev, bash-completions
-RUN curl -Lo /tmp/bazel.deb https://github.com/bazelbuild/bazel/releases/download/0.11.0/bazel_0.11.0-linux-x86_64.deb && \
+RUN curl -Lo /tmp/bazel.deb https://github.com/bazelbuild/bazel/releases/download/0.15.2/bazel_0.15.2-linux-x86_64.deb && \
     sudo dpkg -i /tmp/bazel.deb && rm /tmp/bazel.deb
 
 
 # Instead of "apt-get -y install  golang"
 RUN cd /tmp && \
-    wget https://redirector.gvt1.com/edgedl/go/go1.9.2.linux-amd64.tar.gz && \
+    wget https://redirector.gvt1.com/edgedl/go/go1.10.3.linux-amd64.tar.gz && \
     sudo rm -rf /usr/local/go && \
-    sudo tar -C /usr/local -xzf go1.9.2.linux-amd64.tar.gz && \
+    sudo tar -C /usr/local -xzf go1.10.3.linux-amd64.tar.gz && \
     sudo chown -R circleci /usr/local/go && \
     sudo ln -s /usr/local/go/bin/go /usr/local/bin
 

diff --git a/.circleci/Makefile b/.circleci/Makefile
@@ -2,7 +2,7 @@ HUB ?=
 PROJECT ?= istio
 
 # Using same naming convention as istio/istio
-VERSION ?= go1.9-bazel0.11
+VERSION ?= go1.10-bazel0.15-clang6.0
 IMG ?= ci
 
 # Build a local image, can be used for testing with circleci command line.

diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -3,62 +3,97 @@ version: 2
 jobs:
   build:
     docker:
-      - image: istio/ci:go1.9-bazel0.11
+      - image: istio/ci:go1.10-bazel0.15-clang6.0
     environment:
       - BAZEL_TEST_ARGS: "--test_env=ENVOY_IP_TEST_VERSIONS=v4only --test_output=all"
     resource_class: xlarge
     steps:
       - checkout
       - restore_cache:
           keys:
-            - bazel-cache-{{ checksum "WORKSPACE" }}
-      - restore_cache:
-          keys:
-            - repo-cache-{{ checksum "WORKSPACE" }}
+            - linux_fastbuild-bazel-cache-{{ checksum "WORKSPACE" }}
       # To build docker containers or run tests in a docker
       - setup_remote_docker
       - run: rm ~/.gitconfig
       - run: make check
-      - run: make deb BAZEL_BUILD_ARGS="-j 4"
+      - run: make deb
       - run: make test
-      - run: make test_asan
-      - run: make test_tsan
-      - save_cache:
-          key: repo-cache-{{ checksum "WORKSPACE" }}
-          paths:
-            - /home/circleci/.repo
       - save_cache:
-          key: bazel-cache-{{ checksum "WORKSPACE" }}
+          key: linux_fastbuild-bazel-cache-{{ checksum "WORKSPACE" }}
           paths:
             - /home/circleci/.cache/bazel
       - store_artifacts:
           path: /home/circleci/project/bazel-bin/tools/deb/istio-proxy.deb
           destination: /proxy/deb
       - store_artifacts:
-          path: /home/circleci/project/bazel-bin/src/envoy/mixer/envoy
+          path: /home/circleci/project/bazel-bin/src/envoy/envoy
           destination: /proxy/bin
+  linux_asan:
+    docker:
+      - image: istio/ci:go1.10-bazel0.15-clang6.0
+    environment:
+      - BAZEL_TEST_ARGS: "--test_env=ENVOY_IP_TEST_VERSIONS=v4only --test_output=all"
+    resource_class: xlarge
+    steps:
+      - checkout
+      - restore_cache:
+          keys:
+            - linux_asan-bazel-cache-{{ checksum "WORKSPACE" }}
+      # To build docker containers or run tests in a docker
+      - setup_remote_docker
+      - run: rm ~/.gitconfig
+      - run: make test_asan
+      - save_cache:
+          key: linux_asan-bazel-cache-{{ checksum "WORKSPACE" }}
+          paths:
+            - /home/circleci/.cache/bazel
+  linux_tsan:
+    docker:
+      - image: istio/ci:go1.10-bazel0.15-clang6.0
+    environment:
+      - BAZEL_TEST_ARGS: "--test_env=ENVOY_IP_TEST_VERSIONS=v4only --test_output=all"
+    resource_class: xlarge
+    steps:
+      - checkout
+      - restore_cache:
+          keys:
+            - linux_tsan-bazel-cache-{{ checksum "WORKSPACE" }}
+      # To build docker containers or run tests in a docker
+      - setup_remote_docker
+      - run: rm ~/.gitconfig
+      - run: make test_tsan
+      - save_cache:
+          key: linux_tsan-bazel-cache-{{ checksum "WORKSPACE" }}
+          paths:
+            - /home/circleci/.cache/bazel
   macos:
     macos:
       xcode: "9.3.0"
     environment:
+      - BAZEL_STARTUP_ARGS: "--output_base /Users/distiller/.cache/bazel"
       - BAZEL_TEST_ARGS: "--test_env=ENVOY_IP_TEST_VERSIONS=v4only --test_output=all"
+      - CC: clang
+      - CXX: clang++
     steps:
       - run: sudo ntpdate -vu time.apple.com
-      - run: brew install automake bazel cmake coreutils go libtool wget
+      - run: brew install bazel cmake coreutils go libtool ninja wget
       - checkout
       - restore_cache:
           keys:
-            - bazel-cache-{{ checksum "WORKSPACE" }}-macos
+            - macos_fastbuild_v2-bazel-cache-{{ checksum "WORKSPACE" }}
       - run: rm ~/.gitconfig
+      - run: make build_envoy
       - run: make test
       - save_cache:
-          key: bazel-cache-{{ checksum "WORKSPACE" }}-macos
+          key: macos_fastbuild_v2-bazel-cache-{{ checksum "WORKSPACE" }}
           paths:
-            - /home/circleci/.cache/bazel
+            - /Users/distiller/.cache/bazel
 
 workflows:
   version: 2
   all:
     jobs:
       - build
+      - linux_asan
+      - linux_tsan
       - macos
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -1,5 +1,5 @@
 # Contribution guidelines
 
 So, you want to hack on the Istio proxy? Yay! Please refer to Istio's overall
-[contribution guidelines](https://github.com/istio/istio/blob/master/DEV-GUIDE.md)
+[contribution guidelines](https://github.com/istio/community/blob/master/CONTRIBUTING.md)
 to find out how you can help.
diff --git a/Makefile b/Makefile
@@ -17,30 +17,42 @@ TOP := $(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))
 SHELL := /bin/bash
 LOCAL_ARTIFACTS_DIR ?= $(abspath artifacts)
 ARTIFACTS_DIR ?= $(LOCAL_ARTIFACTS_DIR)
-BAZEL_STARTUP_ARGS ?= --batch
+BAZEL_STARTUP_ARGS ?=
 BAZEL_BUILD_ARGS ?=
 BAZEL_TEST_ARGS ?=
 HUB ?=
 TAG ?=
+ifeq "$(origin CC)" "default"
+CC := clang-6.0
+endif
+ifeq "$(origin CXX)" "default"
+CXX := clang++-6.0
+endif
 
 build:
-	@bazel $(BAZEL_STARTUP_ARGS) build $(BAZEL_BUILD_ARGS) //...
+	CC=$(CC) CXX=$(CXX) bazel $(BAZEL_STARTUP_ARGS) build $(BAZEL_BUILD_ARGS) //...
+	@bazel shutdown
 
 # Build only envoy - fast
 build_envoy:
-	bazel $(BAZEL_STARTUP_ARGS) build $(BAZEL_BUILD_ARGS) //src/envoy/mixer:envoy
+	CC=$(CC) CXX=$(CXX) bazel $(BAZEL_STARTUP_ARGS) build $(BAZEL_BUILD_ARGS) //src/envoy:envoy
+	@bazel shutdown
 
 clean:
 	@bazel clean
+	@bazel shutdown
 
 test:
-	bazel $(BAZEL_STARTUP_ARGS) test $(BAZEL_TEST_ARGS) //...
+	CC=$(CC) CXX=$(CXX) bazel $(BAZEL_STARTUP_ARGS) test $(BAZEL_TEST_ARGS) //...
+	@bazel shutdown
 
 test_asan:
-	CC=clang-5.0 CXX=clang++-5.0 bazel $(BAZEL_STARTUP_ARGS) test $(BAZEL_TEST_ARGS) --config=clang-asan //...
+	CC=$(CC) CXX=$(CXX) bazel $(BAZEL_STARTUP_ARGS) test $(BAZEL_TEST_ARGS) --config=clang-asan //...
+	@bazel shutdown
 
 test_tsan:
-	CC=clang-5.0 CXX=clang++-5.0 bazel $(BAZEL_STARTUP_ARGS) test $(BAZEL_TEST_ARGS) --config=clang-tsan //...
+	CC=$(CC) CXX=$(CXX) bazel $(BAZEL_STARTUP_ARGS) test $(BAZEL_TEST_ARGS) --config=clang-tsan //...
+	@bazel shutdown
 
 check:
 	@script/check-license-headers
@@ -50,7 +62,8 @@ artifacts: build
 	@script/push-debian.sh -c opt -p $(ARTIFACTS_DIR)
 
 deb:
-	@bazel build tools/deb:istio-proxy ${BAZEL_BUILD_ARGS}
+	CC=$(CC) CXX=$(CXX) bazel $(BAZEL_STARTUP_ARGS) build $(BAZEL_BUILD_ARGS) //tools/deb:istio-proxy
+	@bazel shutdown
 
 
 .PHONY: build clean test check artifacts
diff --git a/WORKSPACE b/WORKSPACE
@@ -30,7 +30,7 @@ bind(
 )
 
 # When updating envoy sha manually please update the sha in istio.deps file also
-ENVOY_SHA = "57751946a80cf6841586d47bb3b1c842090f546d"
+ENVOY_SHA = "aa8053b1e8f7d02e1bb0c2a44e78688643248ad1"
 
 http_archive(
     name = "envoy",

diff --git a/include/istio/control/http/check_data.h b/include/istio/control/http/check_data.h
@@ -19,7 +19,7 @@
 #include <map>
 #include <string>
 
-#include "src/istio/authn/context.pb.h"
+#include "google/protobuf/struct.pb.h"
 
 namespace istio {
 namespace control {
@@ -38,15 +38,18 @@ class CheckData {
   // Get downstream tcp connection ip and port.
   virtual bool GetSourceIpPort(std::string *ip, int *port) const = 0;
 
-  // If SSL is used, get origin user name.
-  virtual bool GetSourceUser(std::string *user) const = 0;
+  // If SSL is used, get peer or local certificate SAN URI.
+  virtual bool GetPrincipal(bool peer, std::string *user) const = 0;
 
   // Get request HTTP headers
   virtual std::map<std::string, std::string> GetRequestHeaders() const = 0;
 
   // Returns true if connection is mutual TLS enabled.
   virtual bool IsMutualTLS() const = 0;
 
+  // Get requested server name, SNI in case of TLS
+  virtual bool GetRequestedServerName(std::string *name) const = 0;
+
   // These headers are extracted into top level attributes.
   // This is for standard HTTP headers.  It supports both HTTP/1.1 and HTTP2
   // They can be retrieved at O(1) speed by environment (Envoy).
@@ -78,14 +81,18 @@ class CheckData {
   virtual bool FindCookie(const std::string &name,
                           std::string *value) const = 0;
 
-  // If the request has a JWT token and it is verified, get its payload as
-  // string map, and return true. Otherwise return false.
-  virtual bool GetJWTPayload(
-      std::map<std::string, std::string> *payload) const = 0;
+  // Returns a pointer to the authentication result from request info dynamic
+  // metadata, if available. Otherwise, returns nullptr.
+  virtual const ::google::protobuf::Struct *GetAuthenticationResult() const = 0;
+
+  // Get request url path, which strips query part from the http path header.
+  // Return true if url path is found, otherwise return false.
+  virtual bool GetUrlPath(std::string *url_path) const = 0;
 
-  // If the request has authentication result in header, parses data into the
-  // output result; returns true if success. Otherwise, returns false.
-  virtual bool GetAuthenticationResult(istio::authn::Result *result) const = 0;
+  // Get request queries with string map format. Return true if query params are
+  // found, otherwise return false.
+  virtual bool GetRequestQueryParams(
+      std::map<std::string, std::string> *query_params) const = 0;
 };
 
 // An interfact to update request HTTP headers with Istio attributes.

diff --git a/include/istio/control/http/report_data.h b/include/istio/control/http/report_data.h
@@ -40,12 +40,20 @@ class ReportData {
     uint64_t response_body_size;
     std::chrono::nanoseconds duration;
     int response_code;
+    std::string response_flags;
   };
   virtual void GetReportInfo(ReportInfo* info) const = 0;
 
   // Get destination ip/port.
   virtual bool GetDestinationIpPort(std::string* ip, int* port) const = 0;
 
+  // Get Rbac attributes.
+  struct RbacReportInfo {
+    std::string permissive_resp_code;
+    std::string permissive_policy_id;
+  };
+  virtual bool GetRbacReportInfo(RbacReportInfo* report_info) const = 0;
+
   // Get upstream host UID. This value overrides the value in the report bag.
   virtual bool GetDestinationUID(std::string* uid) const = 0;
 

diff --git a/include/istio/control/tcp/check_data.h b/include/istio/control/tcp/check_data.h
@@ -31,12 +31,15 @@ class CheckData {
   // Get downstream tcp connection ip and port.
   virtual bool GetSourceIpPort(std::string* ip, int* port) const = 0;
 
-  // If SSL is used, get origin user name.
-  virtual bool GetSourceUser(std::string* user) const = 0;
+  // If SSL is used, get peer or local certificate SAN URI.
+  virtual bool GetPrincipal(bool peer, std::string* user) const = 0;
 
   // Returns true if connection is mutual TLS enabled.
   virtual bool IsMutualTLS() const = 0;
 
+  // Get requested server name, SNI in case of TLS
+  virtual bool GetRequestedServerName(std::string* name) const = 0;
+
   // Get downstream tcp connection id.
   virtual std::string GetConnectionId() const = 0;
 };

diff --git a/include/istio/control/tcp/report_data.h b/include/istio/control/tcp/report_data.h
@@ -42,6 +42,14 @@ class ReportData {
 
   // Get upstream host UID. This value overrides the value in the report bag.
   virtual bool GetDestinationUID(std::string* uid) const = 0;
+
+  // ConnectionEvent is used to indicates the tcp connection event in Report
+  // call.
+  enum ConnectionEvent {
+    OPEN = 0,
+    CLOSE,
+    CONTINUE,
+  };
 };
 
 }  // namespace tcp

diff --git a/include/istio/control/tcp/request_handler.h b/include/istio/control/tcp/request_handler.h
@@ -36,16 +36,8 @@ class RequestHandler {
       CheckData* check_data, ::istio::mixerclient::CheckDoneFunc on_done) = 0;
 
   // Make report call.
-  // This can be called multiple times for long connection.
-  // TODO(JimmyCYJ): Let TCP filter use
-  // void Report(ReportData* report_data, bool is_final_report), and deprecate
-  // this method.
-  virtual void Report(ReportData* report_data) = 0;
-
-  // Make report call.
-  // If is_final_report is true, report all attributes. Otherwise, report delta
-  // attributes.
-  virtual void Report(ReportData* report_data, bool is_final_report) = 0;
+  virtual void Report(ReportData* report_data,
+                      ReportData::ConnectionEvent event) = 0;
 };
 
 }  // namespace tcp

diff --git a/include/istio/utils/BUILD b/include/istio/utils/BUILD
@@ -41,4 +41,4 @@ cc_library(
         "attribute_names.h",
     ],
     visibility = ["//visibility:public"],
-)
+)