[pull] main from pixie-io:main

DEVELOPMENT.md

@@ -9,21 +9,24 @@ The Pixie project currently consists of three main components:
 This document outlines the process for setting up the development environment for each of these components.
 
 ## Setting up the Environment
+To set up the developer environment required to start building Pixie's components, run the `run_docker.sh` script. The following script will run the Docker container and dump you out inside the docker container console from which you can run all the necessary tools to build, test, and deploy Pixie in development mode.
 
-To setup the developer environment required to start building Pixie's components, it is easiest to use our docker image which has the necessary tools and packages. If you plan to use a non-minikube environment (gke, eks, etc) for running Vizier/Pixie Cloud, this can be done by running our script:
+1. Since this script runs a Docker container, you must have Docker installed. To install it follow these instructions [here](https://docs.docker.com/get-docker/).
 
-```
-./scripts/run_docker.sh
-```
+2. `run_docker.sh` requires the realpath command which is part of the coreutils package that you may need to install:
+    * Ubuntu: `sudo apt-get install coreutils`
+    * OS X: `brew install coreutils`
 
-Otherwise, to launch the dev environment with minikube, run:
+3. Finally, run the following script to start the Docker container:
+    ```
+    ./scripts/run_docker.sh
+    ```
 
-```
-make dev-env-start
-```
+4. Since development of Pixie requires a Kubernetes cluster to deploy to, you must have Minikube installed and running. Follow the instructions [here](https://docs.px.dev/installing-pixie/setting-up-k8s/minikube-setup/). Note that Pixie development scripts use the output of `kubectl config current-context` to determine which Kubernetes cluster to deploy to. So make sure if you have multiple clusters, the context is pointing to the correct target cluster.
 
 
 ## Pixie Cloud
+Pixie Cloud manages users, authentication, and proxying “passthrough” mode. If you want to make changes to Pixie Cloud, then you will need to spin up a self-hosted version in development mode to test those changes. If you aren't changing Pixie Cloud, feel free to use officially released Pixie Cloud options listed in our Install Guides.
 
 1. Load the config maps and secrets.
 
@@ -93,25 +96,30 @@ After which, you can rerun a `px auth login` to authenticate with the dev cloud
 Make sure to `px delete --clobber` if running a prior instance of Vizier pointing to another cloud instance.
 
 ## Vizier
+Vizier is Pixie’s data collector that runs on each cluster. It is responsible for query execution and managing PEMs.
 
-Deploying a development version of Vizier is a 2-step process. A release-version of Vizier must first be deployed (through the CLI or YAMLs), which will set up specific cluster-secrets that are not deployed via Skaffold. Skaffold can then be run to build and deploy a version of Vizier based on local changes.
+### Getting Started
+Verify that you can build Pixie Vizier and run the unit tests.
 
-1. If deploying Vizier pointing to a dev cloud instance, make sure to export the following environment variables:
+The following will build Pixie Vizier and run a unit test of your choice in Pixie's Vizier module using [Bazel](https://bazel.build/). If you are running the development environment via Docker, the following should be run inside the Docker container.
 
-    ```
-    export PL_CLOUD_ADDR=dev.withpixie.dev:443
-    export PL_TESTING_ENV=dev
-    ```
+```
+bazel test //src/<path to unit test file> --test_output=errors -j $(nproc)
+```
 
-2. Deploy Vizier through CLI/YAML. If deploying Vizier to a dev cloud instance, make sure to export the necessary dev-specific environment variables mentioned in `Pixie Cloud` above.
+### Deploying
+Deploying a development version of Vizier is a 2-step process. An official release-version of Vizier must first be deployed (through the Pixie CLI or YAMLs) and can then be run to build and deploy a local development version of Vizier.
+
+1. If you wish to test development changes made to both Pixie Cloud and Vizier, export the following environment variables that will point to the development Pixie Cloud instance:
 
     ```
-    px deploy
+    export PL_CLOUD_ADDR=dev.withpixie.dev:443
+    export PL_TESTING_ENV=dev
     ```
 
-3. Wait for the Vizier to successfully connect to the cloud instance, to ensure all secrets and configs have been set up properly.
+2. Install the Pixie CLI and run `px deploy`. Depending on whether you are pointing to a self-hosted Pixie Cloud or the official Community Pixie Cloud follow the appropriate installation guide [here](https://docs.px.dev/installing-pixie/install-guides/). `px deploy` will set up specific cluster-secrets, etc that are not deployed via Skaffold. Wait for this command to successfully complete and Vizier to successfully connect to Pixie Cloud, to ensure all secrets and configs have been set up properly. Note you will not need to run this to deploy again unless you connect to a different cluster.
 
-4. Deploy a local version of Vizier using skaffold.
+3. Deploy a local development version of Pixie Vizier using  [Skaffold](https://skaffold.dev/). Note each time you make a code change you will need to run this command to build and deploy the new version.
 
    ```
    skaffold run -f skaffold/skaffold_vizier.yaml

Jenkinsfile

@@ -1258,27 +1258,69 @@ def buildScriptForCloudProdRelease = {
   postBuildActions()
 }
 
+def copybaraTemplate(String name, String copybaraFile) {
+  DefaultCopybaraPodTemplate(name) {
+    deleteDir()
+    checkout scm
+    container('copybara') {
+    sshagent (credentials: ['pixie-copybara-git']) {
+      withCredentials([
+        file(
+          credentialsId: 'copybara-private-key-asc',
+          variable: 'COPYBARA_GPG_KEY_FILE'),
+        string(
+          credentialsId: 'copybara-gpg-key-id',
+          variable: 'COPYBARA_GPG_KEY_ID'),
+        ]) {
+          sh "GIT_SSH_COMMAND='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' \
+          ./ci/run_copybara.sh ${copybaraFile}"
+        }
+      }
+    }
+  }
+}
 
-def buildScriptForCopybaraTemplate(String name, String copybaraFile) {
+def buildScriptForCopybaraPublic() {
   try {
     stage('Copybara it') {
-      DefaultCopybaraPodTemplate(name) {
+      copybaraTemplate("public-copy", "tools/copybara/public/copy.bara.sky")
+    }
+    stage('Copy tags') {
+      DefaultGCloudPodTemplate("public-copy-tags") {
         deleteDir()
-        checkout scm
-        container('copybara') {
-        sshagent (credentials: ['pixie-copybara-git']) {
-          withCredentials([
-            file(
-              credentialsId: 'copybara-private-key-asc',
-              variable: 'COPYBARA_GPG_KEY_FILE'),
-            string(
-              credentialsId: 'copybara-gpg-key-id',
-              variable: 'COPYBARA_GPG_KEY_ID'),
-            ]) {
-              sh "GIT_SSH_COMMAND='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' \
-              ./ci/run_copybara.sh ${copybaraFile}"
-            }
-          }
+        checkout([
+          changelog: false,
+          poll: false,
+          scm: [
+            $class: 'GitSCM',
+            branches: [[name: 'main']],
+            extensions: [
+              [$class: 'RelativeTargetDirectory', relativeTargetDir: 'pixie-private'],
+              [$class: 'CloneOption', noTags: false, reference: '', shallow: false]
+            ],
+            userRemoteConfigs: [
+              [credentialsId: 'build-bot-ro', url: 'git@github.com:pixie-labs/pixielabs.git']
+            ]
+          ]
+        ])
+        checkout([
+          changelog: false,
+          poll: false,
+          scm: [
+            $class: 'GitSCM',
+            branches: [[name: 'main']],
+            extensions: [
+              [$class: 'RelativeTargetDirectory', relativeTargetDir: 'pixie-oss'],
+              [$class: 'CloneOption', noTags: false, reference: '', shallow: false]
+            ],
+            userRemoteConfigs: [
+              [credentialsId: 'pixie-copybara-git', url: 'git@github.com:pixie-io/pixie.git']
+            ]
+          ]
+        ])
+        dir('pixie-private') {
+          sh "GIT_SSH_COMMAND='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' \
+          ./ci/copy_release_tags.sh ../pixie-oss"
         }
       }
     }
@@ -1291,6 +1333,19 @@ def buildScriptForCopybaraTemplate(String name, String copybaraFile) {
   }
 }
 
+def buildScriptForCopybaraPxAPI() {
+  try {
+    stage('Copybara it') {
+      copybaraTemplate("pxapi-copy", "tools/copybara/pxapi_go/copy.bara.sky")
+    }
+  }
+  catch (err) {
+    currentBuild.result = 'FAILURE'
+    echo "Exception thrown:\n ${err}"
+    echo 'Stacktrace:'
+    err.printStackTrace()
+  }
+}
 
 if (isNightlyTestRegressionRun) {
   buildScriptForNightlyTestRegression()
@@ -1305,9 +1360,9 @@ if (isNightlyTestRegressionRun) {
 } else if (isCloudProdBuildRun) {
   buildScriptForCloudProdRelease()
 } else if(isCopybaraPublic) {
-  buildScriptForCopybaraTemplate("public-copy", "tools/copybara/public/copy.bara.sky")
+  buildScriptForCopybaraPublic()
 } else if(isCopybaraPxAPI) {
-  buildScriptForCopybaraTemplate("pxapi-copy", "tools/copybara/pxapi_go/copy.bara.sky")
+  buildScriptForCopybaraPxAPI()
 }else {
   buildScriptForCommits()
 }

ci/copy_release_tags.sh

@@ -0,0 +1,62 @@
+#!/bin/bash -e
+
+# Copyright 2018- The Pixie Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+if [ "$#" -ne 1 ]; then
+  echo "This script requires exactly one argument with the location to the checkout for the public repo."
+  exit 1
+fi
+pubRepoLoc=$1
+
+pushd "${pubRepoLoc}" &> /dev/null || exit 1
+
+typeset -A privateHashToPublicHash
+mapfile -t hashMappings < <(git --no-pager log --pretty="format:%(trailers:key=GitOrigin-RevId,valueonly,separator=%x00) %H")
+
+for hashMapping in "${hashMappings[@]}"; do
+  IFS=" " read -r -a privToPub <<< "${hashMapping}"
+  privHash="${privToPub[0]}"
+  pubHash="${privToPub[1]}"
+  privateHashToPublicHash["${privHash}"]="${pubHash}"
+done
+
+echo "Found ${#privateHashToPublicHash[@]} commit mappings."
+
+popd &> /dev/null || exit 1
+
+mapfile -t tagRefs < <(git show-ref --tags -d | grep -E release | grep -E '\^\{\}' | grep -E -v 'v\d+\.\d+\.\d+-' | cut -d'^' -f1 | sed 's|refs/tags/||g')
+
+echo "Found ${#tagRefs[@]} tag refs."
+
+for tagRef in "${tagRefs[@]}"; do
+  IFS=" " read -r -a hashToTag <<< "${tagRef}"
+  hash="${hashToTag[0]}"
+  tag="${hashToTag[1]}"
+  pubHash=${privateHashToPublicHash[$hash]}
+  tagMessage=$(git tag -l --format='%(contents)' "${tag}")
+  if [[ -n "${pubHash}" ]]; then
+    pushd "${pubRepoLoc}" &> /dev/null || exit 1
+    if [[ $(git tag -l "${tag}") ]]; then
+      echo "Tag ${tag} already exists. Skipping."
+    else
+      echo "Creating and pushing tag ${tag}"
+      git tag -a "${tag}" -m "${tagMessage}" "${pubHash}"
+      git push origin "${tag}"
+    fi
+    popd &> /dev/null || exit 1
+  fi
+done

k8s/cloud/staging/cloud_proxy_backend_config.yaml

@@ -1,5 +1,5 @@
 ---
-apiVersion: cloud.google.com/v1beta1
+apiVersion: cloud.google.com/v1
 kind: BackendConfig
 metadata:
   name: cloud-proxy-backend-config
@@ -12,6 +12,6 @@ spec:
   cdn:
     enabled: true
     cachePolicy:
-      includeHost: false
-      includeProtocol: false
-      includeQueryString: false
+      includeHost: true
+      includeProtocol: true
+      includeQueryString: true

k8s/cloud/testing/cloud_proxy_backend_config.yaml

@@ -1,5 +1,5 @@
 ---
-apiVersion: cloud.google.com/v1beta1
+apiVersion: cloud.google.com/v1
 kind: BackendConfig
 metadata:
   name: cloud-proxy-backend-config
@@ -12,6 +12,6 @@ spec:
   cdn:
     enabled: true
     cachePolicy:
-      includeHost: false
-      includeProtocol: false
-      includeQueryString: false
+      includeHost: true
+      includeProtocol: true
+      includeQueryString: true

scripts/run_docker.sh

@@ -47,6 +47,7 @@ docker run --rm -it \
   "${configs[@]}" \
   -v /var/run/docker.sock:/var/run/docker.sock \
   -v "${workspace_root}:/pl/src/px.dev/pixie" \
+  --network host \
   "${PX_RUN_DOCKER_EXTRA_ARGS[@]}" \
   "${docker_image_with_tag}" \
   "${exec_cmd[@]}"