Rework CLI build pipeline tutorial

docs/cli/cli-add-destination.md

@@ -0,0 +1,65 @@
+# Add destination
+
+Now add a destination using the CLI:
+
+```
+quix local app create starter-destination
+```
+
+This creates a starter destination for you. Alternatively, you could type `quix local app create` and then interactively select the starter destination. You saw an example of this when adding the demo data source. You can call the destination `destination`, or any other name you like.
+
+## Review the destination code
+
+Replace the destination code completely with the following new code:
+
+``` python
+from quixstreams import Application
+import os
+
+from dotenv import load_dotenv
+load_dotenv()
+
+# called for every message
+def sink(message):
+    print("Average speed is: ", message['average-speed'])
+    print("Timestamp at end of window is: ", message['time'])    
+
+app = Application()
+input_topic = app.topic(os.environ["input"])
+sdf = app.dataframe(input_topic)
+sdf = sdf.update(sink)
+
+if __name__ == "__main__":
+    app.run(sdf)
+```
+
+The code just prints out the components of the message individually. You could perform any processing you want here, such as persisting the data, or displaying values on a real-time chart.
+
+## Edit requirements.txt
+
+Check the `requirements.txt` file. Make sure you are using Quix Streams greater than or equal to 2.4.1:
+
+```
+quixstreams>=2.4.1
+```
+
+## Test your destination code
+
+Start your source and transform if they are not already running locally. Now run your destination code. It outputs messages such as:
+
+```
+Average speed is:  282.27891156462584
+Timestamp at end of window is:  1713524910000
+```
+
+## Get ready to sync to Cloud
+
+You are now ready to synchronize everything with Quix Cloud, and run your complete pipeline as a set of dockerized services in a cluster managed by Kubernetes, with nothing more than a single command. This is described in the next step.
+
+!!! important
+
+    Before you proceed to the next step, make sure you stop all your local code running.
+
+## Next step
+
+* [Sync to Cloud](./cli-sync-to-cloud.md)

docs/cli/cli-add-source.md

@@ -0,0 +1,45 @@
+# Add a source
+
+Now create a sample application. In this case you'll first add a data source application. In your project directory enter the following command:
+
+```
+quix local app create 
+```
+
+You will be prompted to select a library item using the interactive menu. Select `Source` and then `Demo Data`. This will give you a source that generates F1 racing car data from a CSV file. Give the application a suitable name, such as `F1 demo data`.
+
+## Create your local variables
+
+To create your local variables for the source application, run the following command:
+
+```
+quix local variables export
+```
+
+This generates a `.env` file with several environment variables set for you, including the streaming token used for authenticating with your selected environment, and your input and output topics, for example:
+
+```
+Quix__Portal__Api=https://portal-api.platform.quix.io
+Quix__Organisation__Id=yourorg
+Quix__Workspace__Id=yourorg-tutorial-prod
+Quix__Sdk__Token=sdk-349...dd
+output=f1-data
+```
+
+## Run your code locally
+
+Run your source which will publish data to the `f1-data` topic. In the source application directory:
+
+```
+python3 main.py
+```
+
+!!! tip
+
+    The required topic(s) are created. If your program generates a timeout error, it means the topic(s) are still being created. Simply wait a few moments and then try running the program again.
+
+You can leave your code running, create a new shell tab, and proceed to the next step.
+
+## Next step
+
+* [Add a transform](./cli-add-transform.md)

docs/cli/cli-add-transform.md

@@ -0,0 +1,132 @@
+# Add a transform
+
+Now add a transform using the CLI. In you project directory run the command:
+
+```
+quix local app create starter-transformation
+```
+
+This creates a starter transformation for you. Alternatively, you could type `quix local app create` and then interactively select the starter transform. You saw an example of this interactivity when adding the demo data source. You can call the transform `transform`, or any other name you like.
+
+## Modify the transform code
+
+Now, you'll modify the starter transform code to do something more useful. Let's say you want to calculate the average speed from the race car. You could do that with a tumbling window with a time window of 30 seconds:
+
+``` python
+import os
+from quixstreams import Application
+from datetime import timedelta
+
+# for local dev, load env vars from a .env file
+from dotenv import load_dotenv
+load_dotenv()
+
+# create a Quix Streams application
+app = Application()
+
+# JSON deserializers/serializers used by default
+input_topic = app.topic(os.environ["input"])
+output_topic = app.topic(os.environ["output"])
+
+# consume from input topic
+sdf = app.dataframe(input_topic)
+
+# calculate average speed using a 30 second tumbling window
+sdf = sdf.apply(lambda row: row["Speed"]) \
+    .tumbling_window(timedelta(seconds=30)).mean().final() \
+        .apply(lambda value: {
+            'average-speed': value['value'],
+            'time': value['end']
+            })
+
+# print every row
+sdf = sdf.update(lambda row: print(row))
+
+# publish to output topic
+sdf = sdf.to_topic(output_topic)
+
+if __name__ == "__main__":
+    app.run(sdf)
+```
+
+This publishes a message to the output topic, with the following format, every 30 seconds:
+
+``` json
+{'average-speed': 249.04918032786884, 'time': 1713518340000}
+```
+
+## App.yaml file
+
+Now take a look at the `app.yaml` file for your transform:
+
+``` yaml
+name: transformer
+language: Python
+variables:
+  - name: input
+    inputType: InputTopic
+    description: Name of the input topic to listen to.
+    defaultValue: csv-data
+    required: false
+  - name: output
+    inputType: OutputTopic
+    description: Name of the output topic to write to.
+    defaultValue: transform
+    required: false
+dockerfile: dockerfile
+runEntryPoint: main.py
+defaultFile: main.py
+```
+
+This file defines the application, including its input and output topics.
+
+Note that the default input topic is `csv-data`, but you need your transform to subscribe to the `f1-data` topic. You'll fix this in the next section.
+
+## Local environment variables
+
+There are a couple of ways you can set the input topic of the transform, but a sensible way is to change the environment variable that sets the input topic. But as yet, there are no environment variables created. You can create them with the command you saw in the previous step, `quix local variables export`.
+
+You can then edit the `.env` file so that the input topic is `f1-data`.
+
+## Edit requirements.txt
+
+Check the `requirements.txt` file. Make sure you are using Quix Streams greater than or equal to 2.4.1:
+
+```
+quixstreams>=2.4.1
+```
+
+## Run your transform
+
+Now run your transform. In the transform application directory:
+
+```
+python3 main.py
+```
+
+## Testing
+
+There are a couple of ways you can test if this is working. One way is to switch to Quix Cloud, and navigate to the Topics section of the main menu - you will see active data on your topics. Another way is to run a command-line program to read the data on a topic. So, to read the `transform` topic, you could create some code `test.py` in your transform app directory:
+
+``` python
+from quixstreams import Application                                                                                
+from dotenv import load_dotenv
+import os
+
+load_dotenv()
+app = Application()
+topic = app.topic('transform')
+sdf = app.dataframe(topic)
+sdf = sdf.update(lambda row: print(row))
+app.run(sdf)
+```
+
+This reads the `transform` topic (the output of your transformer) and prints out the results. This proves data is being produced into the transform topic, by your transformer.
+
+!!! tip
+
+    In your test program, you could have loaded the topic to read from the local `.env` file, rather than hard coding it. You'd load it with code such as the following: `topic = app.topic(os.environ["output"])`. This would enable you perhaps use the same test code in multiple applications, without needed to edit the code to change the topic name.
+
+## Next step
+
+* [Add a destination](./cli-add-destination.md)

docs/cli/cli-build-pipeline.md

@@ -0,0 +1,14 @@
+# Build a pipeline using the Quix CLI
+
+In this tutorial you'll build a simple pipeline using the Quix CLI. The steps are:
+
+1. **Create a project** - you create your Git project.
+2. **Sign up to Cloud** - Sign up to Quix Cloud for free.
+3. **Add a source** - you add a demo data source.
+4. **Add a transform** - you add a transform and perform some simple processing.
+5. **Add a destination** - you add a simple destination.
+6. **Sync to Quix Cloud** - you synchronize you CLI project with Quix Cloud.
+
+## Next step
+
+* [Create a project](./cli-create-project.md)

docs/cli/cli-cloud-signup.md

@@ -0,0 +1,25 @@
+# Sign up to Quix Cloud for free
+
+Sign up [here](https://portal.platform.quix.io/self-sign-up){target=_blank}.
+
+You need to sign up to Quix Cloud at this stage as you're going to be testing some of your code locally, and rather than set up a local broker, you'll use a Quix-hosted broker for your Kafka topics.
+
+When you log into Quix Cloud, you'll be prompted to create a project.
+
+## Create a Quix project
+
+In this step you create a project in Quix Cloud based on your Git repository.
+
+1. Give you project a suitable name, such as `Simple Pipeline`.
+2. Select `Quix advanced configuration` to continue creation of your project.
+3. Select your Git provider (for example, GitHub).
+4. Link the Quix project to your Git repository using the **setup guide** provided for your chosen Git provider.
+5. Use PROD for your environment, and make sure the `main` branch is selected.
+
+If you need help on creating a project, you can [read the documentation](../create/create-project.md).
+
+When you've created the project, switch back to the command line.
+
+## Next step
+
+* [Add a source](./cli-add-source.md)

docs/cli/cli-create-project.md

@@ -0,0 +1,49 @@
+---
+title: Create a Git project
+description: You install the Quix CLI and create a simple project, with a pipeline consisting of one data source application, which you then sync up with Quix Cloud.
+---
+
+# Create a Git project
+
+In previous sections of the documentation you explored using Quix Streams. You now continue your journey on the command line by installing the Quix CLI, and then using it to connect with Quix Cloud. 
+
+From your Git repository, you create a simple project on the command line, add a data source application, along with a transform and destination, test it, and then sync it with your Quix Cloud pipeline view.
+
+## Step 1: Create a Git repository
+
+Create a Git repo where you can store your files, for example you could use GitHub. Create a repo initialized with a `README.md` file, so it can be cloned more easily. 
+
+## Step 2: Clone your Git repo into your local project directory
+
+For example, if your GitHub repo is named `cli-app`:
+
+```
+git clone <url-to-git>/cli-app
+cd cli-app
+```
+
+## Step 3: Install Quix CLI
+
+```
+curl -fsSL https://github.com/quixio/quix-cli/raw/main/install.sh | sudo bash
+```
+
+For further details on installation, including instructions for Microsoft Windows, see the [install guide](https://github.com/quixio/quix-cli?tab=readme-ov-file#installation-of-quix-cli){target=_blank}.
+
+!!! tip
+
+    To update Quix CLI just run `quix update` to get the latest version of Quix CLI.
+
+## Step 4: Initialize your project as a Quix project
+
+In your Git project directory, enter:
+
+```
+quix local init
+```
+
+This initializes your Quix project with a `quix.yaml` file, which describes your Quix project.
+
+## Next step
+
+* [Sign up to Cloud](./cli-cloud-signup.md)

docs/kb/cli.md → docs/cli/cli-reference.md

@@ -1,22 +1,11 @@
 ---
-title: Quix CLI
-status: new
-description: The Quix Command-Line Interface.
+title: Quix CLI Reference Guide
+description: The Quix Command-Line Interface reference guide.
 ---
 
-# Quix Command-Line Interface (CLI)
+# Quix CLI Reference Guide
 
-The [Quix CLI](https://github.com/quixio/quix-cli){target=_blank} is a powerful command-line companion for seamlessly managing and interacting with the features of Quix Cloud. While Quix offers a robust UI for a user-friendly experience, the CLI empowers you with efficiency and flexibility, enabling you to streamline your workflow, and take control from the command line.
-
-* Effortless Control: Execute commands effortlessly to manage various aspects of your Quix organization.
-
-* Script Automation: Integrate Quix operations into your scripts for automated workflows and enhanced productivity.
-
-* Accessibility: Access and manipulate Quix features directly from the command line, providing an alternative interface for users who prefer terminal-based interactions.
-
-* Scalability: Seamlessly scale your Quix operations, whether you are working on a single instance or orchestrating tasks across multiple environments.
-
-View information about the CLI in the [GitHub repository](https://github.com/quixio/quix-cli){target=_blank}.
+This is the reference guide for the Quix CLI.
 
 ## Installation