BYOC Documentation

.gitignore

@@ -1,5 +1,6 @@
 
 site/
+lib/
 .idea/
 .vscode/
 .DS_Store

docs/byoc/byoc-vs-cloud.md

@@ -0,0 +1,35 @@
+# When to choose Quix Brign Your Own Cloud Enterprise Edition?
+
+Our cloud offering is a great way to work with the Quix Platform for most use cases. We think of Quix in the cloud as the perfect way to use the platform if you prefer and require the managed experience. 
+
+BYOC goes one step further than cloud by offering all the powers of the Quix Platform, along with the ability to control and scale the underlying services, infrastructure to fit your needs. 
+
+When hosting your own Quix platform, you are in full control of how all its parts operate. We provide safe and sane defaults, but it is ready to be configured to fit your exact use case.
+
+| | Quix Cloud | Quix BYOC |
+|----|----|----|
+| Analyze data streams | Yes   | Yes |
+| Quickly deploy real-time apps | Yes   | Yes |
+| Iso 27001 certified | Yes   | No[^1] |
+| You store the data | No | Yes[^2] |
+| On-premise cost savings | No | Possible[^3] |
+| Consume cloud reservations | No | Possible[^3] |
+| Bespoke Networking | No | Yes |
+| Bespoke Storage | No | Yes |
+| Private CA certificates | No | Yes |
+| Can be scaled to your needs | Yes | Yes |
+
+
+
+[^1]:
+    The process of creating Quix Platform deliverables continues to be ISO 27001 certified. This means that the containers and services inside them that you receive as part of the BYOC offering are built to stringent security standards. You can expect them to follow best industry practices and safe defaults. 
+
+    Once you install them on your environment, they become part of your processes and fall under your data security policies. Therefore these software packages may or may not be ISO (and ISO27001 specifically) compliant when deployed outside of Quix Cloud.
+
+[^2]:
+    Exclusively. Quix BYOC Enterprise Editions are self-contained. _Only_ you store your data.
+
+[^3]:
+    Running your own infrastructure on your premises (or in a co-location) is often more cost-efficient than most cloud offerings. This is particularly true if you have specific hardware requirements that are not easily met in the cloud, such as GPUs, FPGAs, ultra-high-speed storage or networking. However, Quix Cloud remains the best managed experience we offer.
+
+    You may choose to run Quix BYOC in your own cloud environment, consuming reservations or quotas established in prior agreements with cloud providers. This has the potential to lower your operational costs, while maintaining a cloud presence with Quix. 

docs/byoc/installation.md

@@ -0,0 +1,114 @@
+# How to install BYOC
+
+## Overview
+
+The short story of deployment is as follows:
+
+1. Create a [Kubernetes cluster](#1-create-a-kubernetes-cluster)
+2. Obtain a Quix Container Registry [API key](#2-obtain-an-api-key-to-pull-quix-platform-containers) to pull Quix Platform containers
+3. Obtain a copy of the Quix Platform [BYOC installer](#3-obtain-a-copy-of-the-quix-platform-byoc-installer-ansible-recipe--docker-image) (Ansible Recipe + Docker Image)
+4. [Prepare Secrets](#4-prepare-secrets-and-platform-configuration) and platform configuration
+5. Run the [installer container](#5-run-the-installer-container)
+6. [Push Secrets](#6-push-secrets-and-platform-configuration-to-the-cluster) and platform configuration to the cluster
+7. Optional: [Initialize ArgoCD](#7-optional-initialize-argocd)
+8. [Install Quix Platform](#8-install-quix-platform) for the first time
+
+## Step By Step
+
+### 1. Create a Kubernetes cluster
+
+This step is where you have the most freedom of choice. The assumption this guide works with is that you have an operational Kubernetes cluster ready to schedule pods and create the requirement dependencies. Any Kubernetes installation should work as long as it clears the requirements in the [Requirements](requirements.md) section.
+
+In short, if your Kubernetes cluster configuration isn't _particularly_ niche, it should work without any extra configuration.
+
+We recommend a production ready Kubernetes cluster, with HA control plane and multiple worker nodes, as this makes maintenance and scaling easier.
+
+### 2. Obtain an API key to pull Quix Platform containers
+
+In this step you will be receiving a username / API token and docker registry URL. These will be provided to you by Quix before you start the installation process.
+
+### 3. Obtain a copy of the Quix Platform BYOC installer (Ansible Recipe + Docker Image)
+
+With the provided API key you can download the installer from our private Container Registry. The installer is a Docker image that contains an Ansible recipe and all the required dependencies to run it. This way you don't need to install Ansible (and its particular Python version dependencies), Helm and all the other programs and libraries required to run the installer. We have packaged it all to offer a turn-key, convenient experience.
+
+### 4. Prepare Secrets and platform configuration
+
+A successful installation of the Quix Platform requires a few secrets and variables to be set. Secrets are used to store sensitive information about your Platform Installation. API keys, usernames, passwords for all the services, dependencies and integrations are stored in Secrets which the Installer will use to configure the Platform. By storing secrets in a Kubernetes native way we ensure that any subsequent successful installations of the Quix Platform do not depend on the machine used for installation. After a successful initialisation of the Secrets and Platform configuration, anyone with the correct permissions should be able to maintain the platform.
+
+If ArgoCD is used -- of which we supply an opinionated installation, if you choose to use it --, it will be able to utilize these secrets and configuration to maintain the platform.
+
+### 5. Run the installer container
+
+The container is built and tested with Docker. On any system capable of running Docker, you can run the installer with the following command:
+
+```bash
+./quixplatform container quix -V 1.1.2
+```
+
+This will run the Quix Platform Installer using the Quix Platform CLI tool provided to you ahead of the installation process. 
+
+![Quixplatform CLI](../images/deploy/byoc/using-quixplatform-cli.jpg){width=80%}
+
+### 6. Push Secrets and platform configuration to the cluster
+
+The Quixplatform CLI tools help you push the Secrets and Platform configuration to the cluster.
+
+1. Ensure you have everything set correctly in `vars/platform-variables.yaml` and `vars/platform-secrets-human.yaml`
+
+Then, from inside the Installer container started in step 4, run the following command:
+
+
+```bash
+namespace=quix
+./secrets.sh init vars/platform-secrets-human.yaml.template vars/platform-secrets-human.yaml
+# Add your secrets to vars/platform-secrets-human.yaml
+./secrets.sh encode vars/platform-secrets-human.yaml vars/platform-secrets.yaml
+./secrets.sh push ${namespace} vars/platform-secrets.yaml
+./quixplatform config ${namespace} -f vars/platform-variables.yaml
+```
+As a general rule, commands and steps are designed to be idempotent wherever possible. This is also the case when manipulating secrets and platform configuration:
+any subsequent pushes that make no changes will look like this: 
+![Quixplatform CLI Config Push](../images/deploy/byoc/push-variables.png){width=80%}
+
+### 7. Optional: Initialize ArgoCD
+Run the following commands inside the installer container:
+
+Change values in `init-scripts/application.yaml.template` to meet your requirements.
+The following lines are of particular interest:
+
+```yaml
+    repoURL: ''
+    targetRevision: stable
+    path: releases/
+```
+These define what release 'stream' your environment is subscribed to. When running ArgoCD connected to our release branches, any updates we deploy to the repository your environment follows will be automatically applied, based on GitOps principles. There's a [little more to it](release-filtering.md), but generally speaking your environment will deploy the latest version of the platform, unless you specify otherwise.
+
+Any software package we release for the BYOC offering will have been put through our QA process and validated. That said, we operate multiple external branches to facilitate testing leading-edge features on your test environments before we add them to the stable branch.
+
+
+```bash
+argo_namespace=argo
+quix_namespace=quix
+init-scripts/init.sh ${argo_namespace} ${quix_namespace}
+```
+
+This will initialize ArgoCD and create the Quix application in the namespace you specify. If you do not specify a namespace, it will default to `argocd` and `quix`.
+
+!!! tip
+    You may host your own release repository to subscribe to with ArgoCD. This allows you to be in ultimate control of the entire release process.
+
+### 8. Install Quix Platform
+
+Inside the installer container, run the following command:
+
+```bash
+./quixplatform install
+```
+or if ArgoCD is installed, but you want to manually trigger the installation:
+```bash
+./quixplatform install --context=<your_kubectl_context_name>
+```
+
+This will install the Quix Platform on your Kubernetes cluster. The installation process will take 3-5 minutes. The progress of the installation will be displayed as Ansible roles initialize and set up the various components and then the Quix Platform itself.
+
+![Quixplatform Successful Installation](../images/deploy/byoc/byoc-successful-install.png){width=80%}

docs/byoc/overview.md

@@ -0,0 +1,18 @@
+# Bring Your Own Cluster
+
+BYOC is Quix, packaged to run on your own Kubernetes cluster. Bring Your Own Cluster, control its resources and make them work efficiently by deploying the Quix Platform to it.
+
+!!! Info "Glossary"
+    **Kubernetes** is a container orchestration platform tailored to offer great flexibility and power. You can run Kubernetes on a variety of different computers ranging from your laptop to hundreds of Virtual Machine nodes in data centres distributed around the world.
+
+The driving force behind the design of **Quix BYOC Enterprise Edition** was to enable the installation of the Quix Platform in as many different configurations of Kubernetes as technology allows. This enables us to offer Quix on a range of platforms, from minimal-viable clusters functioning as pilot projects on spare bare-metal hardware, to extensive production clusters managed within AKS, EKS, GCP, Proxmox and similar environments. These advanced setups can include hundreds of CPU cores and terabytes of RAM, utilize container engines off the beaten path and provide storage through a multitude of means.
+
+We have developed Quix BYOC to deliver a safe, opinionated default configuration, designed to ensure our platform operates effectively across a diverse range of Kubernetes cluster setups. Additionally, it is crafted for easy integration with existing components and technologies you currently employ and wish to continue using.
+
+We have also made a conscious effort to ensure Quix works well with clusters that already run workloads. We do not require a dedicated empty cluster for Quix to run on, although you may want to consider operating BYOC in its own environment for a variety of reasons.
+
+
+
+## Next steps
+
+See the requirements of the [Quix platform](requirements.md)

docs/byoc/release-filtering.md

@@ -0,0 +1,73 @@
+# What are Quix BYOC Release Filters?
+
+We recommend using ArgoCD or Flux to orchestrate releases and updates to your BYOC environment. We publish multiple branches, or "streams", of releases, but you may choose to only deploy a subset of these releases to your environment. This is where release filters become useful. 
+
+ArgoCD does not natively support picking and choosing the parts of an application you wish to deploy. So we built a tool for it ourselves.
+
+A filter is a powerful tool you can use to determine what versions of our services or chart to accept or reject, based on multiple possible criteria.
+
+Our BYOC distribution comes bundled with ArgoCD. If you decide to deploy it, you'll gain the ability to specify which releases are acceptable. This feature is particularly useful for testing or adhering to the version constraints of your underlying infrastructure.
+
+## How do I use filters?
+
+A Release filter is a Configmap containing a YAML file, which describes the conditions under which a release should be accepted or rejected.
+
+By default the filter is empty, and your platform will deploy any release to which your platform subscribes, to your environment automatically.
+
+The following example shows the various ways in which you may filter releases:
+```yaml
+---
+platform_release:
+  - type: versionBelow
+    key: version
+    value: "2.0.0"
+  - type: versionBelow
+    key: minSupportedKubernetes
+    value: "1.24.19"
+  - type: includeKeyword
+    key: requiredStorageBackend
+    value: 
+      - "nfsclient"
+auth:
+  - type: dateBefore
+    key: imagetag
+    value: "20231111"
+  - type: excludeKeyword
+    key: imagetag
+    value: 
+      - "cactus"
+      - "beta"
+      - "test"
+  - type: includeKeyword
+    key: imagetag
+    value: 
+      - "bamboo"
+  - type: exact
+    key: imagetag
+    value: "20231116.2-2023-11-03-promocode"
+  - type: highestMinorVersion
+    key: imagetag
+    value: "1"
+```
+
+Any release chart rejected because of the `platform_release` service matching a filter will cause the entire chart to be rejected. This happens even if the `auth` filter would have accepted the auth service otherwise. Platform_release supersedes individual service filters.
+
+A rejected service (or chart) will not be deployed to your environment either at the time of us pushing it to the branch your environment is subscribed to, or at the time of your environment syncing with the branch. If you wish to install a release anyway, you may at any time change your filter and manually sync your ArgoCD Quix application.
+
+!!! note
+    We recommend that if you set a filter, it only applies to the platform_release service. Any more granular filtering should be done at the recommendation of a Quix support engineer during a troubleshooting session. During normal operation, keeping the minSupportedKubernetes key filtered is a good idea, as it will prevent you from deploying a release that is not compatible with your Kubernetes cluster. Otherwise, we recommend you keep the filter, at least initially, empty.
+
+## How do I know what was filtered out?
+
+In your Kubernetes cluster, you can find the Configmap containing a chart (essentially a service version matrix) of all the services your platform was meant to deploy. In case of a filter-induced rejection, you will see a Configmap called `containerversions` receive a new annotation at `.metadata.annotations.rejectedServices` containing a json array of all the services and reasons they were rejected. Or it may be empty in case no such event occurred.
+
+```
+> kubectl -n quix get cm containerversions -o jsonpath='{.metadata.annotations.rejectedServices}' | base64 -d
+{}
+
+
+- or -
+
+{ "auth": { "type": "excludeKeyword", "key": "imagetag", "value": [ "cactus", "promocode" ] }, "platform_release": { "type": "versionBelow", "key": "version", "value": "1.0.0" } }
+```
+

docs/byoc/requirements.md

@@ -0,0 +1,54 @@
+# Requirements for BYOC
+
+The BYOC installation method assumes that some resources are available and meet version or quantity criteria. We have conducted extensive testing on a variety of Kubernetes cluster configurations and have validated the platform on a number of different setups. 
+
+The requirements are designed to ensure that the platform will run effectively and efficiently. Most default installations in managed kubernetes environments will meet these requirements, as well as any opinionated installation of Kubernetes on on-prem VMs running any modern linux operating system.
+
+
+## Essentials
+
+To ensure the success of the installation process, the following essential requirements must be fulfilled:
+
+- For running the installer: A computer capable of running Linux containers. It may be either a physical or a virtual machine. This computer should run Linux, MacOS, or Windows (using Docker Desktop or WSL), and it may have either an aarch64 (arm64, Apple Silicon in most cases) or amd64 architecture. This will only be used to run the installation scripts.
+- For running the installer: A kubectl client that has already been configured to communicate with this cluster. This client must be able to communicate with the cluster from the computer running the installation scripts. 
+- To run the platform: A Kubernetes cluster that has already been set up.
+- Sufficient permissions to create a service account, manage limited tokens, and create new namespaces and other resources.
+
+## Kubernetes cluster minimum requirements
+- AMD64 architecture.[^1]
+- Multiple nodes with 6 cpu cores and 48GB RAM total.[^2].
+- Kubernetes version 1.24.1 or later.
+- A container runtime capable of handling Linux containers, MongoDB and Kafka. (containerd, cri-o etc.)
+- A storage class capable of handling dynamic provisioning of Persistent Volumes. (nfs with nfs-subdir-external-provisioner, Ceph, Longhorn, EBS/EFS, azurefile, Google filestore, Isilon etc)
+- One standard RWX storage class and one standard RWO storage class.
+- The ability of exposing services outside of the Kubernetes cluster. (either LoadBalancer compatible Load Balancer or NodePorts)
+- Network ingress and egress permissive enough for the kubelet to pull platform containers from the Quix Container Registry.
+
+## Kubernetes cluster recommended requirements
+- Everything in the minimum requirements
+- Three separate control plane nodes (or managed control plane) for high availability and easy maintenance.
+- A nodepool sufficient for your requirements, but at least 40 CPU cores and 200GB memory total.
+- Kubernetes version 1.28 or later.
+- Optional: additional Premium RWO storage class
+- A Load Balancer capable of exposing a LoadBalancer type service (such as AWS ELB or MetalLB)
+
+
+!!! warning
+
+    You may re-use existing services, such as Kafka and MongoDB. If you do not, Quix BYOC will install its own versions of these services. Your underlying infrastructure must be able to run these services, which may require enabling AVX, AVX2 and XSAVE instructions on your CPU, or passed through for it in the Hypervisor. This is not the default in some hypervisors and may be the source of issues.
+
+## Support requirements
+
+For the purposes of installing Quix BYOC Enterprise Edition, an engineer comfortable with using kubectl (or Lens) and the linux terminal is required to attend the initial setup. We also provide live support during this.
+
+You are in full ownership and control of the underlying infrastructure running the platform, therefore scaling, patching and other administrative tasks will be performed by your team to your requirements and specifications. We are always happy to help with any issues you may encounter and will be delivering container and application level security patches to the Quix Platform.
+
+## Next steps
+Follow the [installation](installation.md) process to find out how to install the Quix Platform on your Kubernetes cluster.
+
+
+[^1]: 
+    We are excited about ARM64 CPUs as well! They are fast becoming a serious contender for building distributed computational platforms, especially on AWS. Unfortunately, some of our dependencies are not yet available for aarch64 and therefore prevent us from shipping Quix for aarch64 Kubernetes clusters. When we do, this document will change. You are welcome to run the installation scripts and the container delivering them on aarch64 computers from the native multi-arch image.
+
+[^2]:
+    We recommend nodes at least 16GB of RAM each node to err on the safe side. Quix has been tested on 3 x 8GB nodes and has been found to work well. This however leaves very little room for your other workloads and isn't a production experience we wish for you to have with Quix.