GRR Fleetspeak with Envoy sandbox (#164)

Adds Envoy sandbox documentation and configuration files.

.gitignore

@@ -1,3 +1,4 @@
 _build/**
 .*/**
 
+*.pem

fleetspeak/index.md

@@ -7,4 +7,5 @@ maxdepth: 1
 Installing from a release DEB (recommended) <from-release-deb.md>
 Installing from source <from-source.md>
 Scaling a fleetspeak setup <scaling.md>
+Running GRR and Fleetspeak with Envoy <sandbox.md>
 ```

fleetspeak/sandbox.md

@@ -0,0 +1,79 @@
+# Running GRR and Fleetspeak with Envoy
+
+The following diagram shows the high level architecture of the sandbox environment.
+
+![sandbox fleetspeak](../images/sandbox_fleetspeak.png)
+
+This sandbox uses [Docker Compose](https://docs.docker.com/compose/) to demonstrate how to run [GRR](https://github.com/google/grr) and [Fleetspeak](https://github.com/google/fleetspeak) with [Envoy](https://www.envoyproxy.io/) leveraging Fleetspeak in the HTTPS header frontend mode.  
+This frontend mode lends itself to showcase an example of Fleetspeak's more involved frontend modes.  
+Fleetspeak offers several other frontend modes to run GRR/Fleetspeak in varying contexts. You can find more information about them in the [Fleetspeak repository](https://github.com/google/fleetspeak/docs/sandboxes.md).   
+
+The sandbox also features the sample configurations that can be used to learn about Fleetspeak's frontend mode options and how to model your own configurations.  
+
+Before you begin you will need to setup the sandbox environment.
+It is important that you follow all the steps in the [setup instructions](#setup-instructions) section before running the sandbox.
+
+## Setup the sandbox environment
+- [Install Docker](#install-docker)
+- [Install docker compose](#install-docker-compose)
+- [Install Git](#install-git)
+- [Clone the GRR Doc repository](#clone-the-grr-doc-repository)
+- [Create Configurations](#create-configurations)
+
+### Install docker
+Ensure that you have a recent versions of ```docker``` installed.
+
+You will need a minimum version of ```19.03.0+```.
+
+Version ```20.10``` is well tested, and has the benefit of included ```compose```.
+
+The user account running the examples will need to have permission to use Docker on your system.
+
+Full instructions for installing Docker can be found on the [Docker website](https://docs.docker.com/get-docker/).  
+
+### Install docker compose
+The examples use [Docker compose configuration version 3.8](https://docs.docker.com/compose/compose-file/compose-versioning/#version-38).
+
+You will need to a fairly recent version of [Docker Compose](https://docs.docker.com/compose/).  
+
+### Install Git
+The GRR Doc repository is managed using [Git](https://git-scm.com/).
+
+You can [find instructions for installing Git on various operating systems here](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).  
+
+### Clone the GRR Doc repository
+If you have not cloned the GRR Doc repository already, clone it with:
+
+```
+git clone https://github.com/google/grr-doc
+```
+
+### Create configurations
+
+```
+cd grr-doc/fleetspeak/sandbox
+./createConfig.sh
+```
+
+## Run the GRR Fleetspeak sandbox demo
+```
+docker compose up --build -d
+```
+When you run the sandbox for the first time Docker will have to build the containers before running them.
+This will take a couple of minutes so you might want to grab a cup of coffee now.  
+The next runs will be a lot faster.  
+
+## Connect to the GRR UI
+Once the containers are up and running you can connect to the GRR UI.
+You will have to create a ```demo``` user first. The script will ask you for a password. We suggest you use ```demo``` for simplicity.  
+
+You can create the ```demo``` user by executing the command listed in the box below.
+```
+docker exec -it sandbox-grr-admin-1 /grr/venv/bin/grr_config_updater add_user demo
+```
+With the ```demo``` user in place you can now point your browser to the GRR UI: [http://localhost:8000](http://localhost:8000)
+
+## Stop the GRR Fleetspeak sandbox demo
+```
+docker compose down
+```

fleetspeak/sandbox/config/fleetspeak-admin/components.textproto

@@ -0,0 +1,5 @@
+mysql_data_source_name: "fleetspeak-user:fleetspeak-password@tcp(mysql-fleetspeak:3306)/fleetspeak"
+admin_config: <
+  listen_address: "0.0.0.0:9091"
+>
+notification_use_http_notifier: true

fleetspeak/sandbox/config/fleetspeak-admin/services.textproto

@@ -0,0 +1,13 @@
+services {
+  name: "GRR"
+  factory: "GRPC"
+  config {
+    [type.googleapis.com/fleetspeak.grpcservice.Config] {
+      target: "grr-frontend:11111"
+      insecure: true
+    }
+  }
+}
+broadcast_poll_time {
+  seconds: 1
+}

fleetspeak/sandbox/config/fleetspeak-frontend/components.textproto

@@ -0,0 +1,15 @@
+mysql_data_source_name: "fleetspeak-user:fleetspeak-password@tcp(mysql-fleetspeak:3306)/fleetspeak"
+https_config: <
+  listen_address: "0.0.0.0:9090"
+  certificates: "FRONTEND_CERTIFICATE"
+  key: "FRONTEND_KEY"
+  frontend_config: <
+    https_header_checksum_config: <
+      client_certificate_header: "x-client-certificate" 
+      client_certificate_checksum_header: "x-client-cert-hash"
+    >
+  >
+>
+health_check_config: <
+  listen_address: "0.0.0.0:8080"
+>

fleetspeak/sandbox/config/fleetspeak-frontend/services.textproto

@@ -0,0 +1,13 @@
+services {
+  name: "GRR"
+  factory: "GRPC"
+  config {
+    [type.googleapis.com/fleetspeak.grpcservice.Config] {
+      target: "grr-frontend:11111"
+      insecure: true
+    }
+  }
+}
+broadcast_poll_time {
+  seconds: 1
+}

fleetspeak/sandbox/config/grr-client/client.yaml

@@ -0,0 +1,6 @@
+Client.fleetspeak_enabled: true
+Client.foreman_check_frequency: 30
+Logging.verbose: true
+Logging.engines: file,stderr
+Logging.path: /
+Logging.filename: /grr-client.log

fleetspeak/sandbox/config/grr-client/config.textproto

@@ -0,0 +1,9 @@
+client_certificate_header: "x-client-certificate"
+trusted_certs: "FRONTEND_CERTIFICATE"
+server: "fleetspeak-server:10000"
+client_label: ""
+filesystem_handler: <
+  configuration_directory: "/config/"
+  state_file: "/fleetspeak.state"
+>
+streaming: true

fleetspeak/sandbox/config/grr-client/textservices/grr.service

@@ -0,0 +1,11 @@
+name: "GRR"
+factory: "Daemon"
+config: {
+  [type.googleapis.com/fleetspeak.daemonservice.Config]: {
+    argv: "/grr/venv/bin/python"
+    argv: "-m"
+    argv: "grr_response_client.grr_fs_client"
+    argv: "--config"
+    argv: "/grr/venv/install_data/etc/client.yaml"
+  }
+}

fleetspeak/sandbox/config/grr-server/server.local.yaml

@@ -0,0 +1,42 @@
+    Database.implementation: MysqlDB
+    Blobstore.implementation: DbBlobStore
+    Mysql.host: mysql-grr
+    Mysql.port: 3306
+    Mysql.database_name: grr
+    Mysql.database: grr
+    Mysql.database_username: grr-user
+    Mysql.username: grr-user
+    Client.server_urls:
+    - http://localhost:8080/
+    AdminUI.url: http://localhost:8000
+    AdminUI.bind: 0.0.0.0
+    AdminUI.headless: true
+    Logging.domain: localhost
+    Logging.verbose: true
+    Logging.engines: file,stderr
+    Logging.path: /grr
+    Logging.filename: /grr/grr-server.log
+    Monitoring.alert_email: grr-monitoring@localhost
+    Monitoring.emergency_access_email: grr-emergency@localhost
+    Mysql.database_password: grr-password
+    Mysql.password: grr-password
+    Mysql.flow_processing_threads_max: 20
+
+    Frontend.certificate: |
+FRONTEND_CERTIFICATE
+
+    AdminUI.csrf_secret_key: O$YTx01F2BuxgT4DoVSI hJVuhWOxyuuxJhg#lg96ARF_fWLabzN5kjAy0UBcI9GDrUPo1v
+      b7PyF3vj4pqRrc8oegeJ8XgIyI5X
+    Server.fleetspeak_enabled: true
+    Client.fleetspeak_enabled: true
+    ClientBuilder.fleetspeak_bundled: true
+    Target:Linux:
+      ClientBuilder.fleetspeak_client_config: /grr/venv/fleetspeak-server-bin/etc/fleetspeak-server/linux_client.config
+    Target:Windows:
+      ClientBuilder.fleetspeak_client_config: /grr/venv/fleetspeak-server-bin/etc/fleetspeak-server/windows_client.config
+    Target:Darwin:
+      ClientBuilder.fleetspeak_client_config: /grr/venv/fleetspeak-server-bin/etc/fleetspeak-server/darwin_client.config
+    Server.fleetspeak_server: fleetspeak-admin:9091
+    FleetspeakFrontend Context:
+      Server.fleetspeak_message_listen_address: 0.0.0.0:11111
+    Server.initialized: true

fleetspeak/sandbox/containers/envoy/Dockerfile

@@ -0,0 +1,98 @@
+ARG ENVOY_IMAGE="${ENVOY_IMAGE:-envoyproxy/envoy}"
+ARG ENVOY_VARIANT="${ENVOY_VARIANT:-dev}"
+
+
+FROM ${ENVOY_IMAGE}:${ENVOY_VARIANT} as envoy-base
+ARG ENVOY_CONFIG=envoy.yaml
+ENV ENVOY_CONFIG="$ENVOY_CONFIG"
+ENV DEBIAN_FRONTEND=noninteractive
+RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
+    --mount=type=cache,target=/var/lib/apt/lists,sharing=locked \
+    rm -f /etc/apt/apt.conf.d/docker-clean \
+    && echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' | tee /etc/apt/apt.conf.d/keep-cache \
+    && apt-get -qq update -y \
+    && apt-get -qq install --no-install-recommends -y curl
+COPY --chmod=777 "$ENVOY_CONFIG" /etc/envoy.yaml
+COPY --chmod=644 cert.pem /etc/cert.pem
+COPY --chmod=644 key.pem /etc/key.pem
+CMD ["/usr/local/bin/envoy", "-c", "/etc/envoy.yaml"]
+
+FROM envoy-base as envoy-admin
+ARG ENVOY_ADMIN_PORT=10001
+ENV ENVOY_ADMIN_PORT="$ENVOY_ADMIN_PORT"
+HEALTHCHECK \
+    --interval=1s \
+    --timeout=1s \
+    --start-period=1s \
+    --retries=3 \
+    CMD curl -s "localhost:${ENVOY_ADMIN_PORT}/stats?filter=server.state" | grep 0 \
+             && curl -s "localhost:${ENVOY_ADMIN_PORT}/stats?filter=listener_manager.workers_started" | grep 1
+
+FROM envoy-base as envoy-fault-injection
+RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
+    --mount=type=cache,target=/var/lib/apt/lists,sharing=locked \
+    apt-get -qq update -y \
+    && apt-get -qq install --no-install-recommends -y tree
+COPY enable_delay_fault_injection.sh disable_delay_fault_injection.sh enable_abort_fault_injection.sh disable_abort_fault_injection.sh send_request.sh /
+
+
+FROM envoy-base as envoy-jaeger-native
+#
+# for discussion on jaeger binary compatibility, and the source of the file, see here:
+#  https://github.com/envoyproxy/envoy/issues/11382#issuecomment-638012072
+#
+RUN echo "4a7d17d4724ee890490bcd6cfdedb12a02316a3d33214348d30979abd201f1ca /usr/local/lib/libjaegertracing_plugin.so" > /tmp/checksum \
+         && curl -Ls https://github.com/envoyproxy/misc/releases/download/jaegertracing-plugin/jaegertracing-plugin-centos.tar.gz \
+              | tar zxf - -C /usr/local/lib \
+         && mv /usr/local/lib/libjaegertracing.so.0.4.2 /usr/local/lib/libjaegertracing_plugin.so \
+         && sha256sum -c /tmp/checksum \
+         && rm /tmp/checksum
+
+
+FROM envoy-base as envoy-load-balancing
+RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
+    --mount=type=cache,target=/var/lib/apt/lists,sharing=locked \
+    apt-get -qq update -y \
+    && apt-get -qq install --no-install-recommends -y python3
+COPY ./client.py /client.py
+EXPOSE 8001
+
+
+FROM envoy-base as envoy-double-proxy-base
+COPY --chmod=777 ./certs/ca.crt /certs/cacert.pem
+
+
+FROM envoy-double-proxy-base as envoy-double-proxy-frontend
+COPY  --chmod=777 ./certs/postgres-frontend.example.com.crt /certs/clientcert.pem
+COPY  --chmod=777 ./certs/example.com.key /certs/clientkey.pem
+
+
+FROM envoy-double-proxy-base as envoy-double-proxy-backend
+COPY --chmod=777 ./certs/postgres-backend.example.com.crt /certs/servercert.pem
+COPY --chmod=777 ./certs/example.com.key /certs/serverkey.pem
+
+
+FROM envoy-base as envoy-certs
+COPY --chmod=777 ./certs /certs
+
+
+FROM envoy-base as envoy-lua
+ADD  --chmod=777 ./lib/mylibrary.lua /lib/mylibrary.lua
+
+
+FROM envoy-base as envoy-go
+ENV GODEBUG=cgocheck=0
+COPY --chmod=777 ./lib/simple.so /lib/simple.so
+
+
+FROM envoy-base as envoy-ext_authz
+COPY --chmod=777 ./config /etc/envoy-config
+COPY --chmod=777 ./run_envoy.sh /run_envoy.sh
+CMD ["/bin/sh", "/run_envoy.sh"]
+
+
+FROM envoy-base as envoy-dynamic-fs
+COPY --chmod=777 ./configs /var/lib/envoy
+
+
+FROM envoy-base

fleetspeak/sandbox/containers/fleetspeak/Dockerfile

@@ -0,0 +1,46 @@
+# Copyright 2023 Google LLC
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+FROM golang:1.21 as builder
+
+RUN apt update && \
+    apt install -y python3-venv && \
+    apt install -y pip && \
+    apt install -y git
+
+WORKDIR /
+
+SHELL ["/bin/bash", "-c"]
+
+RUN git clone https://github.com/google/fleetspeak.git && \
+    cd fleetspeak && \
+    go get -u golang.org/x/lint/golint && \
+    go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28.0 && \
+    go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2
+
+RUN cd /fleetspeak && \
+    python3 -m venv $HOME/.venv/FSENV && \
+    source $HOME/.venv/FSENV/bin/activate && \
+    pip install wheel pytest && \
+    pip install -e ./fleetspeak_python[test] && \
+    pip install -e ./frr_python && \
+    ./fleetspeak/generate_go_py_protos.sh && \ 
+    ./fleetspeak/build.sh
+
+FROM golang:1.21
+
+RUN mkdir -p /app/bin
+
+COPY --from=builder /fleetspeak/fleetspeak/src/server/server/server /app/bin/
+COPY --from=builder /fleetspeak/fleetspeak/src/client/client/client /app/bin/
+COPY --from=builder /fleetspeak/fleetspeak/src/config/fleetspeak_config /app/bin/

fleetspeak/sandbox/containers/grr/client/Dockerfile

@@ -0,0 +1,19 @@
+# Copyright 2023 Google LLC
+# 
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+FROM fleetspeak AS fleetspeak
+
+FROM grr
+COPY --from=fleetspeak /app/bin/client /usr/local/bin/fleetspeak-client
+COPY config/grr-client/client.yaml /grr/venv/install_data/etc/client.yaml
+WORKDIR /