Security observability (#398)

* Work in progress Analysis tools

* Thursday updates

* Friday fixes and improvements

* Friday code

* Separate analyzers into a separate container

* Wire in analyzers container in the Helm chart

* Hide analysis UI features when not enabled

* Fix sidepanel bug with fallback metadata

* Bug fix for change in way tab links are hidden

* Remove debug logging

* Add refresh button to report selector drop down. Change no reports icon

* Add support for adding breadcrumbs in the sub nav bar

* Fix unit tests

* Fix format issues

* Final front-end unit test fixes

* Fix issues when deploying via Helm with mariadb

* Analyzers container fix. Allow helm chart to be packaged.

* Build script fixes

* Remove file

* WIP: Add support for Clair image scanning

* Use klar

* Remove binary

* Add clair helm chart for dev

* Fixes

* USe end var for clair server address

* Latest updates

* Improvements

* Minor fixes

* Tweak

* Fix 1.16 detecton issue with the analyzers

* Chart fixes

* Changes following first run of script

* Changes following npm install

* Update custom-src to new model
- expose custom module's module's
- Add routing module
- Tweak stratos.config.ts log output
- remove custom-src dir

* update naming... custom extensions --> suse extensions

* A few tidyups to help review

* Fix build issue due to merge

* Fixes following merge from upstream

* Remove clair from this PR

* Ignore example packages when there's a stratos config file

* Changes following review

* Changes following merge

* Update dir names, remove examples folder

* Add back in custom-src deploy content, also add product version to config

* Revert change needed downstream... (only needed when suse extension is included)

* Remove unused wip report viewers

* Fix after merge

* Move new terminal & config code to plugin, fix more build files

* Fix imports and add doc

* Fix compilation issues

* Change following merge

* Tweaks to logging

* Fix bug where report can not be deleted

* Fix kube config connect after merge, also fix subtype & error on connect

* Fix e2e

* Improve drop-down menu

* Remove strange merge artifacts

* Remove build file

* Fix graph overview

* Numerous improvements to graph parsing and presentation

* Remove logging. Add no reports message to workload analysis

* Add support for CRDs. KubeCF renders correctly.

* Allow which engines are enabled to be configured

* Fix issue where reports are not filtered by endpoint

* Minor changes following review

* Fixes for a few more issues

* Add Analyzers image build to Concourse CI

* Multiple small fixes
- fix text search in analysis list
- fix title of links in analysers info page
- handle slow connections by only polling analysis list when not already

* Fix kubeGuid for helm world

* Add AnalysisReportRunnerComponent
- Still need to add this to other places

* Delete reports when endpoint is unregistered

* Buf fixes. Use breadcrumbs in sub-nav

* Add run analysis button to workload analyis and graph tabs

* Fix select of overlay in workload graphs page

* Change default sort order of analysis list to age

* Ensure table cell links update on row change

* Align table action's icon better

* Use a side panel for analyzer info

* Add actions/effects for all used analyis actions
- Add new ResetPaginationOfType action, like ResetPagination but applies to all types
- Allows user to refresh reports list after kicking off new report on namespace & workload tabs
- Handle missing report param in reports returned from get all reports

* Remove some console.logs, converted some to console.info

* Update Kube Dashboard, allow download link to be configurable
- Default download link updated to v2.0.3
- Can configured link by setting env var `STRATOS_KUBERNETES_DASHBOARD_IMAGE`
- Can configure env var in helm via `console.kubeDashboardImage`
- Kube Dashboard now expanded by default (to show namespace drop down)

* Fix after merge

* Changes following review

* Fix expand of kube dashboard header by default

* Changes following review

* Fix json-viewer dark mode

* Fix profile page and side nav top position following header diet
- Fix side nav top position
- Update fix for profile page to also work in non-desktop mode

* WIP Wire in alerts to workload graph
- need to understand if namespace should be checked when matching node/resource to alert
- need to apply correct colour

* Fix workload security analysis overlay slide in

* Hide analysis headers info in tech preview & tie in tech preview check to analysisService.hideAnalysis$
- Q should the backend plugins be available in tech preview, see TODO

* Hide the Workload Graph view if in tech preview

* Fix disable of analysis plugin when tech preview is switched off

* Adderss PR feedback

* Minor tidy ups, fix analysis in graph
- apply typing to many places
- handle kube resources that we fail to fetch/parse
- wire in analysis overlay to graphs and resource slide in

* Remove debug code

Co-authored-by: Richard Cox <richard.cox@suse.com>

.gitignore

@@ -95,6 +95,7 @@ src/jetstream/console-database.db
 src/jetstream/config.properties
 src/jetstream/db/dbconf.yml
 src/jetstream/plugins/monocular/chart-repo/chartrepo
+src/jetstream/plugins/analysis/container/analyzers
 
 # Customisations - these can be removed in the future
 # Left in for now to prevent these files being checked-in, if they are still present

deploy/ci/suse-console-dev-releases.yml

@@ -66,12 +66,18 @@ resources:
     password: ((docker-password))
     repository: ((docker-repository))/stratos-chartsync
 - name: kube-terminal-image
-    type: docker-image
-    source:
-      username: ((docker-username))
-      password: ((docker-password))
-      repository: ((docker-repository))/stratos-kube-terminal
-
+  type: docker-image
+  source:
+    username: ((docker-username))
+    password: ((docker-password))
+    repository: ((docker-repository))/stratos-kube-terminal
+- name: analyzers-image
+  type: docker-image
+  source:
+    username: ((docker-username))
+    password: ((docker-password))
+    repository: ((docker-repository))/stratos-analyzers
+
 # Artifacts
 - name: image-tag
   type: s3
@@ -199,6 +205,14 @@ jobs:
           tag: image-tag/v2-alpha-tag
           patch_base_reg: ((patch-base-reg))
           patch_base_tag: ((patch-base-tag))
+      - put: analyzers-image
+        params:
+          dockerfile: stratos/src/jetstream/plugins/analysis/container/Dockerfile
+          build: stratos/src/jetstream/plugins/analysis/container/
+          tag: image-tag/v2-alpha-tag
+          patch_base_reg: ((patch-base-reg))
+          patch_base_tag: ((patch-base-tag))
+
 - name: create-chart
   plan:
   - get: stratos

deploy/kubernetes/console/templates/analyzers.yaml

@@ -0,0 +1,62 @@
+---
+{{- if semverCompare ">=1.16" (printf "%s.%s" .Capabilities.KubeVersion.Major (trimSuffix "+" .Capabilities.KubeVersion.Minor) )}}
+apiVersion: apps/v1
+{{- else }}
+apiVersion: extensions/v1beta1
+{{- end }}
+kind: Deployment
+metadata:
+  name: stratos-analyzers
+  labels:
+    app.kubernetes.io/name: "stratos"
+    app.kubernetes.io/instance: "{{ .Release.Name }}"
+    app.kubernetes.io/version: "{{ .Chart.AppVersion }}"
+    app.kubernetes.io/component: "stratos-analyzers"
+    helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: "stratos"
+      app.kubernetes.io/component: "stratos-analyzers"
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: "stratos"
+        app.kubernetes.io/instance: "{{ .Release.Name }}"
+        app.kubernetes.io/version: "{{ .Chart.AppVersion }}"
+        app.kubernetes.io/component: "stratos-analyzers"
+        helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
+        app: "{{ .Release.Name }}"
+    spec:
+      containers:
+      - name: analyzers
+        image: {{.Values.kube.registry.hostname}}/{{.Values.kube.organization}}/stratos-analyzers:{{.Values.consoleVersion}}
+        imagePullPolicy: {{.Values.imagePullPolicy}}
+        ports:
+        - name: api
+          containerPort: 8090
+        env:
+        - name: ANALYSIS_SCRIPTS_DIR
+          value: "/scripts"
+        - name: ANALYSIS_REPORTS_DIR
+          value: "/reports"
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: "{{ .Release.Name }}-analyzers"
+  labels:
+    app.kubernetes.io/name: "stratos"
+    app.kubernetes.io/instance: "{{ .Release.Name }}"
+    app.kubernetes.io/version: "{{ .Chart.AppVersion }}"
+    app.kubernetes.io/component: "stratos-analyzers-service"
+    helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
+spec:
+  type: ClusterIP
+  ports:
+  - name: analyzers
+    port: 8090
+    targetPort: 8090
+  selector:
+    app: "{{ .Release.Name }}"
+    app.kubernetes.io/component: "stratos-analyzers"

deploy/kubernetes/console/values.yaml

@@ -57,6 +57,7 @@ console:
       servicePort: 80
       # nodePort: 30001
 
+
   # Name of config map that provides the template files for user invitation emails
   templatesConfigMapName:
 
@@ -120,6 +121,7 @@ images:
   fdbserver: stratos-fdbserver
   fdbdoclayer: stratos-fdbdoclayer
   chartsync: stratos-chartsync
+  analyzers: stratos-analyzers
 
 # Specify which storage class should be used for PVCs
 #storageClass: default

deploy/kubernetes/custom/__stratos.tpl

@@ -12,6 +12,8 @@
   value: "mongodb://{{ .Release.Name }}-fdbdoclayer:27016"
 - name: SYNC_SERVER_URL
   value: "http://{{ .Release.Name }}-chartsync:8080"
+- name: ANALYSIS_SERVICES_API
+  value: "http://{{ .Release.Name }}-analyzers:8090"
 - name: STRATOS_KUBERNETES_NAMESPACE
   value: "{{ .Release.Namespace }}"
 - name: STRATOS_KUBERNETES_TERMINAL_IMAGE

deploy/kubernetes/custom/custom-build.sh

@@ -25,4 +25,9 @@ function custom_image_build() {
   # Build and push an image for the Kubernetes Terminal
   log "-- Building/publishing Kubernetes Terminal"
   patchAndPushImage stratos-kube-terminal Dockerfile.kubeterminal "${STRATOS_PATH}/deploy/containers/kube-terminal"
+
+  # Analzyers container
+  log "-- Building/publishing Stratos Analyzers"
+  patchAndPushImage stratos-analyzers Dockerfile "${STRATOS_PATH}/src/jetstream/plugins/analysis/container"
+
 }

docs/extensions.md

@@ -174,7 +174,7 @@ First, create the custom-src folder structure - from the top-level of the Strato
 
 ```
 mkdir -p custom-src/frontend/app/custom
-mkdir -p custom-src/frontend/assets/custom
+mkdir -p /frontend/assets/custom
 ```
 
 Next, run the customize task:

docs/suse/kube-terminal-dev.md → docs/suse/suse-extensions-dev.md

@@ -18,4 +18,24 @@ you will need to edit the `src/jetstream/config.properties` file and set these t
 
 The Jetstream backend should be configured.
 
-> Note: Ensure you set `ENABLE_TECH_PREVIEW=true` to enable the Kubernetes Terminal feature.
+> Note: Ensure you set `ENABLE_TECH_PREVIEW=true` to enable the Kubernetes Terminal feature.
+
+
+# Enabling Security Obvervability Analyzers in local development
+
+You need to build the docker image for the analyzers container.
+
+```
+cd src/jetstream/plugins/analysis/container
+docker build . -t stratos-analyzers
+```
+
+Now run this container - this will provide the analysis engines to Stratos:
+
+`docker run -d -p 8090:8090 stratos-analyzers`
+
+Edit your Jetstream `config.properties` file and add the following lines:
+
+```
+ANALYSIS_SERVICES_API=http://127.0.0.1:8090
+```