diff --git a/docs/cli/cli-add-destination.md b/docs/cli/cli-add-destination.md new file mode 100644 index 00000000..22556694 --- /dev/null +++ b/docs/cli/cli-add-destination.md @@ -0,0 +1,65 @@ +# Add destination + +Now add a destination using the CLI: + +``` +quix local app create starter-destination +``` + +This creates a starter destination for you. Alternatively, you could type `quix local app create` and then interactively select the starter destination. You saw an example of this when adding the demo data source. You can call the destination `destination`, or any other name you like. + +## Review the destination code + +Replace the destination code completely with the following new code: + +``` python +from quixstreams import Application +import os + +from dotenv import load_dotenv +load_dotenv() + +# called for every message +def sink(message): + print("Average speed is: ", message['average-speed']) + print("Timestamp at end of window is: ", message['time']) + +app = Application() +input_topic = app.topic(os.environ["input"]) +sdf = app.dataframe(input_topic) +sdf = sdf.update(sink) + +if __name__ == "__main__": + app.run(sdf) +``` + +The code just prints out the components of the message individually. You could perform any processing you want here, such as persisting the data, or displaying values on a real-time chart. + +## Edit requirements.txt + +Check the `requirements.txt` file. Make sure you are using Quix Streams greater than or equal to 2.4.1: + +``` +quixstreams>=2.4.1 +``` + +## Test your destination code + +Start your source and transform if they are not already running locally. Now run your destination code. It outputs messages such as: + +``` +Average speed is: 282.27891156462584 +Timestamp at end of window is: 1713524910000 +``` + +## Get ready to sync to Cloud + +You are now ready to synchronize everything with Quix Cloud, and run your complete pipeline as a set of dockerized services in a cluster managed by Kubernetes, with nothing more than a single command. This is described in the next step. + +!!! important + + Before you proceed to the next step, make sure you stop all your local code running. + +## Next step + +* [Sync to Cloud](./cli-sync-to-cloud.md) diff --git a/docs/cli/cli-add-source.md b/docs/cli/cli-add-source.md new file mode 100644 index 00000000..218d683e --- /dev/null +++ b/docs/cli/cli-add-source.md @@ -0,0 +1,45 @@ +# Add a source + +Now create a sample application. In this case you'll first add a data source application. In your project directory enter the following command: + +``` +quix local app create +``` + +You will be prompted to select a library item using the interactive menu. Select `Source` and then `Demo Data`. This will give you a source that generates F1 racing car data from a CSV file. Give the application a suitable name, such as `F1 demo data`. + +## Create your local variables + +To create your local variables for the source application, run the following command: + +``` +quix local variables export +``` + +This generates a `.env` file with several environment variables set for you, including the streaming token used for authenticating with your selected environment, and your input and output topics, for example: + +``` +Quix__Portal__Api=https://portal-api.platform.quix.io +Quix__Organisation__Id=yourorg +Quix__Workspace__Id=yourorg-tutorial-prod +Quix__Sdk__Token=sdk-349...dd +output=f1-data +``` + +## Run your code locally + +Run your source which will publish data to the `f1-data` topic. In the source application directory: + +``` +python3 main.py +``` + +!!! tip + + The required topic(s) are created. If your program generates a timeout error, it means the topic(s) are still being created. Simply wait a few moments and then try running the program again. + +You can leave your code running, create a new shell tab, and proceed to the next step. + +## Next step + +* [Add a transform](./cli-add-transform.md) diff --git a/docs/cli/cli-add-transform.md b/docs/cli/cli-add-transform.md new file mode 100644 index 00000000..4fd24c94 --- /dev/null +++ b/docs/cli/cli-add-transform.md @@ -0,0 +1,132 @@ +# Add a transform + +Now add a transform using the CLI. In you project directory run the command: + +``` +quix local app create starter-transformation +``` + +This creates a starter transformation for you. Alternatively, you could type `quix local app create` and then interactively select the starter transform. You saw an example of this interactivity when adding the demo data source. You can call the transform `transform`, or any other name you like. + +## Modify the transform code + +Now, you'll modify the starter transform code to do something more useful. Let's say you want to calculate the average speed from the race car. You could do that with a tumbling window with a time window of 30 seconds: + +``` python +import os +from quixstreams import Application +from datetime import timedelta + +# for local dev, load env vars from a .env file +from dotenv import load_dotenv +load_dotenv() + +# create a Quix Streams application +app = Application() + +# JSON deserializers/serializers used by default +input_topic = app.topic(os.environ["input"]) +output_topic = app.topic(os.environ["output"]) + +# consume from input topic +sdf = app.dataframe(input_topic) + +# calculate average speed using a 30 second tumbling window +sdf = sdf.apply(lambda row: row["Speed"]) \ + .tumbling_window(timedelta(seconds=30)).mean().final() \ + .apply(lambda value: { + 'average-speed': value['value'], + 'time': value['end'] + }) + +# print every row +sdf = sdf.update(lambda row: print(row)) + +# publish to output topic +sdf = sdf.to_topic(output_topic) + +if __name__ == "__main__": + app.run(sdf) +``` + +This publishes a message to the output topic, with the following format, every 30 seconds: + +``` json +{'average-speed': 249.04918032786884, 'time': 1713518340000} +``` + +## App.yaml file + +Now take a look at the `app.yaml` file for your transform: + +``` yaml +name: transformer +language: Python +variables: + - name: input + inputType: InputTopic + description: Name of the input topic to listen to. + defaultValue: csv-data + required: false + - name: output + inputType: OutputTopic + description: Name of the output topic to write to. + defaultValue: transform + required: false +dockerfile: dockerfile +runEntryPoint: main.py +defaultFile: main.py +``` + +This file defines the application, including its input and output topics. + +Note that the default input topic is `csv-data`, but you need your transform to subscribe to the `f1-data` topic. You'll fix this in the next section. + +## Local environment variables + +There are a couple of ways you can set the input topic of the transform, but a sensible way is to change the environment variable that sets the input topic. But as yet, there are no environment variables created. You can create them with the command you saw in the previous step, `quix local variables export`. + +You can then edit the `.env` file so that the input topic is `f1-data`. + +## Edit requirements.txt + +Check the `requirements.txt` file. Make sure you are using Quix Streams greater than or equal to 2.4.1: + +``` +quixstreams>=2.4.1 +``` + +## Run your transform + +Now run your transform. In the transform application directory: + +``` +python3 main.py +``` + +## Testing + +There are a couple of ways you can test if this is working. One way is to switch to Quix Cloud, and navigate to the Topics section of the main menu - you will see active data on your topics. Another way is to run a command-line program to read the data on a topic. So, to read the `transform` topic, you could create some code `test.py` in your transform app directory: + +``` python +from quixstreams import Application +from dotenv import load_dotenv +import os + +load_dotenv() +app = Application() +topic = app.topic('transform') +sdf = app.dataframe(topic) +sdf = sdf.update(lambda row: print(row)) +app.run(sdf) +``` + +This reads the `transform` topic (the output of your transformer) and prints out the results. This proves data is being produced into the transform topic, by your transformer. + +!!! tip + + In your test program, you could have loaded the topic to read from the local `.env` file, rather than hard coding it. You'd load it with code such as the following: `topic = app.topic(os.environ["output"])`. This would enable you perhaps use the same test code in multiple applications, without needed to edit the code to change the topic name. + +## Next step + +* [Add a destination](./cli-add-destination.md) diff --git a/docs/cli/cli-build-pipeline.md b/docs/cli/cli-build-pipeline.md new file mode 100644 index 00000000..c3a00c47 --- /dev/null +++ b/docs/cli/cli-build-pipeline.md @@ -0,0 +1,14 @@ +# Build a pipeline using the Quix CLI + +In this tutorial you'll build a simple pipeline using the Quix CLI. The steps are: + +1. **Create a project** - you create your Git project. +2. **Sign up to Cloud** - Sign up to Quix Cloud for free. +3. **Add a source** - you add a demo data source. +4. **Add a transform** - you add a transform and perform some simple processing. +5. **Add a destination** - you add a simple destination. +6. **Sync to Quix Cloud** - you synchronize you CLI project with Quix Cloud. + +## Next step + +* [Create a project](./cli-create-project.md) diff --git a/docs/cli/cli-cloud-signup.md b/docs/cli/cli-cloud-signup.md new file mode 100644 index 00000000..8283e96a --- /dev/null +++ b/docs/cli/cli-cloud-signup.md @@ -0,0 +1,25 @@ +# Sign up to Quix Cloud for free + +Sign up [here](https://portal.platform.quix.io/self-sign-up){target=_blank}. + +You need to sign up to Quix Cloud at this stage as you're going to be testing some of your code locally, and rather than set up a local broker, you'll use a Quix-hosted broker for your Kafka topics. + +When you log into Quix Cloud, you'll be prompted to create a project. + +## Create a Quix project + +In this step you create a project in Quix Cloud based on your Git repository. + +1. Give you project a suitable name, such as `Simple Pipeline`. +2. Select `Quix advanced configuration` to continue creation of your project. +3. Select your Git provider (for example, GitHub). +4. Link the Quix project to your Git repository using the **setup guide** provided for your chosen Git provider. +5. Use PROD for your environment, and make sure the `main` branch is selected. + +If you need help on creating a project, you can [read the documentation](../create/create-project.md). + +When you've created the project, switch back to the command line. + +## Next step + +* [Add a source](./cli-add-source.md) diff --git a/docs/cli/cli-create-project.md b/docs/cli/cli-create-project.md new file mode 100644 index 00000000..8d612f5e --- /dev/null +++ b/docs/cli/cli-create-project.md @@ -0,0 +1,49 @@ +--- +title: Create a Git project +description: You install the Quix CLI and create a simple project, with a pipeline consisting of one data source application, which you then sync up with Quix Cloud. +--- + +# Create a Git project + +In previous sections of the documentation you explored using Quix Streams. You now continue your journey on the command line by installing the Quix CLI, and then using it to connect with Quix Cloud. + +From your Git repository, you create a simple project on the command line, add a data source application, along with a transform and destination, test it, and then sync it with your Quix Cloud pipeline view. + +## Step 1: Create a Git repository + +Create a Git repo where you can store your files, for example you could use GitHub. Create a repo initialized with a `README.md` file, so it can be cloned more easily. + +## Step 2: Clone your Git repo into your local project directory + +For example, if your GitHub repo is named `cli-app`: + +``` +git clone /cli-app +cd cli-app +``` + +## Step 3: Install Quix CLI + +``` +curl -fsSL https://github.com/quixio/quix-cli/raw/main/install.sh | sudo bash +``` + +For further details on installation, including instructions for Microsoft Windows, see the [install guide](https://github.com/quixio/quix-cli?tab=readme-ov-file#installation-of-quix-cli){target=_blank}. + +!!! tip + + To update Quix CLI just run `quix update` to get the latest version of Quix CLI. + +## Step 4: Initialize your project as a Quix project + +In your Git project directory, enter: + +``` +quix local init +``` + +This initializes your Quix project with a `quix.yaml` file, which describes your Quix project. + +## Next step + +* [Sign up to Cloud](./cli-cloud-signup.md) diff --git a/docs/kb/cli.md b/docs/cli/cli-reference.md similarity index 86% rename from docs/kb/cli.md rename to docs/cli/cli-reference.md index e50204d3..e91a2bfd 100644 --- a/docs/kb/cli.md +++ b/docs/cli/cli-reference.md @@ -1,22 +1,11 @@ --- -title: Quix CLI -status: new -description: The Quix Command-Line Interface. +title: Quix CLI Reference Guide +description: The Quix Command-Line Interface reference guide. --- -# Quix Command-Line Interface (CLI) +# Quix CLI Reference Guide -The [Quix CLI](https://github.com/quixio/quix-cli){target=_blank} is a powerful command-line companion for seamlessly managing and interacting with the features of Quix Cloud. While Quix offers a robust UI for a user-friendly experience, the CLI empowers you with efficiency and flexibility, enabling you to streamline your workflow, and take control from the command line. - -* Effortless Control: Execute commands effortlessly to manage various aspects of your Quix organization. - -* Script Automation: Integrate Quix operations into your scripts for automated workflows and enhanced productivity. - -* Accessibility: Access and manipulate Quix features directly from the command line, providing an alternative interface for users who prefer terminal-based interactions. - -* Scalability: Seamlessly scale your Quix operations, whether you are working on a single instance or orchestrating tasks across multiple environments. - -View information about the CLI in the [GitHub repository](https://github.com/quixio/quix-cli){target=_blank}. +This is the reference guide for the Quix CLI. ## Installation diff --git a/docs/cli/cli-sync-to-cloud.md b/docs/cli/cli-sync-to-cloud.md new file mode 100644 index 00000000..a6fdc75a --- /dev/null +++ b/docs/cli/cli-sync-to-cloud.md @@ -0,0 +1,47 @@ +# Sync to Quix Cloud + +In this section you sync your local project with Quix Cloud. + +## Log in using the CLI + +Log into Quix Cloud using the CLI using the following command: + +``` +quix login +``` + +If you're not logged into Cloud, you'll be prompted to log in. + +## Check your context + +The context for the CLI is the combination of the portal URL being used, and the environment you're connecting to. You can check this information with the following command: + +``` +quix context list +``` + +You should be connected to the default context with the Portal URL `https://portal-api.platform.quix.io`. You can have multiple contexts to enable you to have connections for production and development brokers, and potentially local brokers. See the [reference guide](cli-reference.md) for further informatin on more complex setups. + +## Sync your application + +To sync your application, change into the `F1 demo data` directory and enter: + +``` +quix local deploy --push --sync +``` + +This updates your `quix.yaml` project file. The `--push` option pushes all changes to your Git repository (it is similar to git add, git commit, git push). You can also add a commit message with the `--commit-message` option if you want. The `--sync` option ensures your Quix environment is synched with the updated Git repository. + +!!! important + + Although you are running this command in an application directory, the command notes any changes outside of the directory, and interactively prompts you to include those changes in the push or not. + +## See your pipeline running + +In Quix Cloud, go to pipeline view, and see your pipeline running, with your F1 demo data source running. + +![CLI pipeline](../images/cli-pipeline.png) + +## Next step + +* [Read the CLI reference guide](./cli-reference.md) diff --git a/docs/cli/overview.md b/docs/cli/overview.md new file mode 100644 index 00000000..5f63c166 --- /dev/null +++ b/docs/cli/overview.md @@ -0,0 +1,23 @@ +--- +title: Quix CLI +description: The Quix Command-Line Interface. +--- + +# Quix Command-Line Interface (CLI) + +The [Quix CLI](https://github.com/quixio/quix-cli){target=_blank} is a powerful command-line companion for seamlessly managing and interacting with the features of Quix Cloud. While Quix offers a robust UI for a user-friendly experience, the CLI empowers you with efficiency and flexibility, enabling you to streamline your workflow, and take control from the command line. + +* Effortless Control: Execute commands effortlessly to manage various aspects of your Quix organization. + +* Script Automation: Integrate Quix operations into your scripts for automated workflows and enhanced productivity. + +* Accessibility: Access and manipulate Quix features directly from the command line, providing an alternative interface for users who prefer terminal-based interactions. + +* Scalability: Seamlessly scale your Quix operations, whether you are working on a single instance or orchestrating tasks across multiple environments. + +View information about the CLI in the [GitHub repository](https://github.com/quixio/quix-cli){target=_blank}. + +## Next steps + +* [Step through a tutorial](./cli-build-pipeline.md) +* [Read the reference guide](./cli-reference.md) diff --git a/docs/get-started/build-cli.md b/docs/get-started/cli-quickstart.md similarity index 95% rename from docs/get-started/build-cli.md rename to docs/get-started/cli-quickstart.md index e0b14cc7..39b3f3e5 100644 --- a/docs/get-started/build-cli.md +++ b/docs/get-started/cli-quickstart.md @@ -10,6 +10,10 @@ curl -fsSL https://github.com/quixio/quix-cli/raw/main/install.sh | sudo bash For further details on installation, including instructions for Microsoft Windows, see the [install guide](https://github.com/quixio/quix-cli?tab=readme-ov-file#installation-of-quix-cli){target=_blank}. +!!! tip + + To update Quix CLI just run `quix update` to get the latest version of Quix CLI. + ## Step 2: Sign up to Quix Cloud for free Sign up [here](https://portal.platform.quix.io/self-sign-up){target=_blank}. @@ -87,6 +91,6 @@ Go to pipeline view, and see your pipeline running, with your Starter transforma ![Pipeline running](../images/starter-transform.png) -## Next steps +## Next step -* [Read the Quix CLI documentation](../kb/cli.md). +* [See the CLI docs](../cli/overview.md) diff --git a/docs/get-started/process.md b/docs/get-started/process.md index 3f5f166a..3c12140c 100644 --- a/docs/get-started/process.md +++ b/docs/get-started/process.md @@ -83,4 +83,4 @@ You'll notice the messages in the topic are in the JSON format. ## Next step -* [Build a pipeline](./build-cli.md) - build a pipeline using the Quix CLI. +* [Build a pipeline](../cli/cli-build-pipeline.md) - build a pipeline using the Quix CLI. diff --git a/docs/get-started/welcome.md b/docs/get-started/welcome.md index 48ca54a5..f588abfd 100644 --- a/docs/get-started/welcome.md +++ b/docs/get-started/welcome.md @@ -21,7 +21,7 @@ description: Welcome to the Quix Developer documentation. This documentation inc --- - You can [sign up for free](https://portal.platform.quix.io/self-sign-up){target=_blank}, and then build a stream processing pipeline with Quix Streams and Quix Cloud in under ten minutes. + You can sign up to [Quix Cloud for free](https://portal.platform.quix.io/self-sign-up){target=_blank}, and then build a stream processing pipeline with Quix Streams and Quix Cloud in under ten minutes. [Quix Cloud Quickstart :octicons-arrow-right-24:](../quix-cloud/quickstart.md) diff --git a/docs/images/cli-pipeline.png b/docs/images/cli-pipeline.png new file mode 100644 index 00000000..c15a029c Binary files /dev/null and b/docs/images/cli-pipeline.png differ diff --git a/docs/kb/what-is-quix.md b/docs/kb/what-is-quix.md index ae0b383a..908bc384 100644 --- a/docs/kb/what-is-quix.md +++ b/docs/kb/what-is-quix.md @@ -214,7 +214,7 @@ Quix provides a suite of tools to enable you to monitor and manage your data. Th * Logs - Real-time logging information is displayed in a console tab. You also have the option of downloading your logs. * CPU monitor - you can monitor the CPU utilization of your deployment in real time. * Memory monitor - you can monitor the memory usage of the deployment in real time. -* CLI - a powerful command line interface (see the [CLI documentation](../kb/cli.md)). +* CLI - a powerful command line interface, see the [CLI documentation](../cli/overview.md). [See the Data Explorer in action](https://www.loom.com/share/0e3c24fb5f8c48038fe5cf02859b7ebc?sid=743fbdf7-fad5-4c26-831d-b6dad78b9b06). diff --git a/layouts/background.jpg b/layouts/background.jpg index 6dbc582c..11df2328 100644 Binary files a/layouts/background.jpg and b/layouts/background.jpg differ diff --git a/layouts/quix.yml b/layouts/quix.yml index ad93ab3c..8094de86 100644 --- a/layouts/quix.yml +++ b/layouts/quix.yml @@ -36,20 +36,20 @@ layers: image: layouts/background.jpg #Docs URL text - - size: { width: 150, height: 30 } - offset: { x: 144, y: 64 } + - size: { width: 160, height: 36 } + offset: { x: 200, y: 64 } typography: content: quix.io/docs - align: start + overflow: shrink + align: start center color: "#FFFFFF66" font: family: Inter style: SemiBold #Page title text -# - size: { width: 768, height: 336 } - - size: { width: 768, height: 190 } - offset: { x: 144, y: 126 } + - size: { width: 800, height: 240 } + offset: { x: 144, y: 136 } typography: content: *page_title overflow: shrink @@ -57,23 +57,22 @@ layers: color: white line: amount: 3 - height: 1.2 + height: 1.25 font: family: Inter style: SemiBold #Page description text - - size: { width: 768, height: 336 } -# offset: { x: 144, y: 235 } - offset: { x: 144, y: 245 } + - size: { width: 800, height: 144 } + offset: { x: 144, y: 422 } typography: content: *page_description - overflow: shrink - align: start - color: lightgrey + overflow: truncate + align: start bottom + color: white line: - amount: 5 - height: 2.5 + amount: 3 + height: 1.5 font: family: Inter - style: Normal + style: Regular diff --git a/mkdocs.yml b/mkdocs.yml index 23877ef3..0b90bdf6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -25,7 +25,19 @@ nav: - 'Produce data': 'get-started/produce.md' - 'Consume data': 'get-started/consume.md' - 'Process data': 'get-started/process.md' - - 'Build a pipeline': 'get-started/build-cli.md' + - 'CLI quickstart': 'get-started/cli-quickstart.md' + - 'Quix CLI': + - 'Overview': 'cli/overview.md' + - 'Build a pipeline': + - 'Overview': 'cli/cli-build-pipeline.md' + - 'Create a project': 'cli/cli-create-project.md' + - 'Sign up to Quix Cloud': 'cli/cli-cloud-signup.md' + - 'Add a source': 'cli/cli-add-source.md' + - 'Add a transform': 'cli/cli-add-transform.md' + - 'Add a destination': 'cli/cli-add-destination.md' + - 'Sync to Cloud': 'cli/cli-sync-to-cloud.md' + - 'Reference guide': 'cli/cli-reference.md' + # Mostly out of date # - 'Project templates': 'get-started/project-templates.md' - 'Quix Streams': '!import https://github.com/quixio/quix-streams?branch=main' @@ -154,7 +166,6 @@ nav: - 'Why stream processing?': 'kb/why-stream-processing.md' - 'What is Kafka?': 'kb/what-is-kafka.md' - 'Reference': - - 'Quix CLI': 'kb/cli.md' - 'Glossary': 'kb/glossary.md' - 'Contribute': 'kb/contribute.md' - 'Tutorials': @@ -328,7 +339,7 @@ plugins: 'get-started/what-is-kafka.md': 'kb/what-is-kafka.md' 'get-started/glossary.md': 'kb/glossary.md' 'get-started/contribute.md': 'kb/contribute.md' - + 'get-started/build-cli.md' : 'get-started/cli-quickstart.md' theme: name: 'material'