Skip to content

Update predictive maintenance tutorial #300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 22, 2024
Merged

Update predictive maintenance tutorial #300

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cd60732
[wip] - update tutorial to reflect updated template
Apr 16, 2024
5ff52ee
[fix] - put transcript inside tip
Apr 17, 2024
6389910
[chore] - template moving
Apr 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 37 additions & 37 deletions docs/tutorials/computer-vision/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,79 +88,79 @@ You also need to have a Git account. This could be GitHub, Bitbucket, GitLab, or

While this tutorial uses an external Git account, Quix can also provide a Quix-hosted Git solution using Gitea for your own projects. You can watch a video on [how to create a project using Quix-hosted Git](https://www.loom.com/share/b4488be244834333aec56e1a35faf4db?sid=a9aa124a-a2b0-45f1-a756-11b4395d0efc){target=_blank}.

??? "Transcript"
??? "Transcript"

0:01 Hi there, welcome to this video on creating a quick project. I should point out before I get into this video that I'm using the beta development environment at Quix and so what you see may be slightly different when you're testing Quicks.
0:01 Hi there, welcome to this video on creating a quick project. I should point out before I get into this video that I'm using the beta development environment at Quix and so what you see may be slightly different when you're testing Quicks.

0:21 Okay, so having said that I'll get straight into creating a new project. You can see this button over here. I'm going to click that to create a new project.
0:21 Okay, so having said that I'll get straight into creating a new project. You can see this button over here. I'm going to click that to create a new project.

0:33 The new workflow in Quix starts with creating a project. Everything is done inside a project. Generally speaking, I'm simplifying it quite a lot.
0:33 The new workflow in Quix starts with creating a project. Everything is done inside a project. Generally speaking, I'm simplifying it quite a lot.

0:44 A project corresponds to a Git repository. So everything that you create, including your pipeline configuration and so on, will be stored in a Git repository.
0:44 A project corresponds to a Git repository. So everything that you create, including your pipeline configuration and so on, will be stored in a Git repository.

0:59 So I'll give my project a name and I'll call it video project. And the first thing I need to do is specify where this Git repository is going to be.
0:59 So I'll give my project a name and I'll call it video project. And the first thing I need to do is specify where this Git repository is going to be.

1:12 Now, the simplest option is to get quick to do all the work for you and just create the Git repository.
1:12 Now, the simplest option is to get quick to do all the work for you and just create the Git repository.

1:21 And the second thing I need to do is a product called gitty to host the Git repository. So we can do that for you.
1:21 And the second thing I need to do is a product called gitty to host the Git repository. So we can do that for you.

1:27 It all happens in the background. And as you'll see in later videos, you have complete control about what happens in that repository.
1:27 It all happens in the background. And as you'll see in later videos, you have complete control about what happens in that repository.

1:37 You can review pull requests and do, merges and all of those normal things that you would do in a Git workflow.
1:37 You can review pull requests and do, merges and all of those normal things that you would do in a Git workflow.

1:47 The other option is to use an external provider. So for example, you could use GitLab, GitHub, Bitbucket and so on.
1:47 The other option is to use an external provider. So for example, you could use GitLab, GitHub, Bitbucket and so on.

1:57 You can use any provider that supports the use of an SS. H key. For the purposes of this video, I'm going to keep it simple for now and just use the Quix manage Git.
1:57 You can use any provider that supports the use of an SS. H key. For the purposes of this video, I'm going to keep it simple for now and just use the Quix manage Git.

2:11 So I'm now ready to create the project. Now every project will have at least one environment and usually several environments.
2:11 So I'm now ready to create the project. Now every project will have at least one environment and usually several environments.

2:28 So what is an environment? An environment roughly corresponds to a branch in your Git repository, but there's a specific for an environment.
2:28 So what is an environment? An environment roughly corresponds to a branch in your Git repository, but there's a specific for an environment.

2:41 As well as you'll see as we go through this dialogue. But the first thing we need to do is give our environment a name.
2:41 As well as you'll see as we go through this dialogue. But the first thing we need to do is give our environment a name.

2:52 So I'm going to name it. Based on the typical development workflow, so usually we'd have production. Branch or environment and you might have staging and development.
2:52 So I'm going to name it. Based on the typical development workflow, so usually we'd have production. Branch or environment and you might have staging and development.

3:09 So I'm going to create the production environment and as I was saying earlier, that's going to roughly correspond to a branch.
3:09 So I'm going to create the production environment and as I was saying earlier, that's going to roughly correspond to a branch.

3:17 So in this next section, we're going to specify the, ,branch that this environment corresponds to and I want production to correspond to the main branch.
3:17 So in this next section, we're going to specify the, ,branch that this environment corresponds to and I want production to correspond to the main branch.

3:30 And so there's nothing really I need to do here. I could create a new branch if I wanted to, ,but I want to use the default branch which is main.
3:30 And so there's nothing really I need to do here. I could create a new branch if I wanted to, ,but I want to use the default branch which is main.

3:41 The other, thing that I can do here is protect this branch and what that means is that prevents developers from merging or committing content, making changes directly to the main branch.
3:41 The other, thing that I can do here is protect this branch and what that means is that prevents developers from merging or committing content, making changes directly to the main branch.

3:58 In order to change the main branch you'll have to raise, a pull request, a git pull request and that would have to be reviewed and approved and then merged in the usual way.
3:58 In order to change the main branch you'll have to raise, a pull request, a git pull request and that would have to be reviewed and approved and then merged in the usual way.

4:13 So we definitely want that because for production we don't want changes being made directly. So I'm protecting that and you'll, We'll see later how we go about taking changes that we've made and say the develop branch or the dev branch and merge those into the main branch.
4:13 So we definitely want that because for production we don't want changes being made directly. So I'm protecting that and you'll, We'll see later how we go about taking changes that we've made and say the develop branch or the dev branch and merge those into the main branch.

4:32 I'll cover that in another video. So for now let's just click continue. Now for each environment that you create you can specify how you want to host Kafka.
4:32 I'll cover that in another video. So for now let's just click continue. Now for each environment that you create you can specify how you want to host Kafka.

4:45 Now as you probably know Quix uses Kafka as its broker. And you have several options here. You if you want the simplest option and most convenient option is to just let Quix do all the hard work for you.
4:45 Now as you probably know Quix uses Kafka as its broker. And you have several options here. You if you want the simplest option and most convenient option is to just let Quix do all the hard work for you.

5:01 We will create them. Kafka broker and you don't need to do any configuration. We just do it all for you.
5:01 We will create them. Kafka broker and you don't need to do any configuration. We just do it all for you.

5:08 We make sure that scales nicely as well. We use Kubernetes and you know as I say it's the least. It's it's the quickest option especially.
5:08 We make sure that scales nicely as well. We use Kubernetes and you know as I say it's the least. It's it's the quickest option especially.

5:22 If you're testing things out, but there are other options as you can see here. You can use your own self hosted Kafka or you can connect to a Confluent Cloud and we'll cover those options in later videos.
5:22 If you're testing things out, but there are other options as you can see here. You can use your own self hosted Kafka or you can connect to a Confluent Cloud and we'll cover those options in later videos.

5:39 So for now, I'm just going to go with the simplest option and click. Continue. The other thing that you'll need to do here is specify the storage option for the environment.
5:39 So for now, I'm just going to go with the simplest option and click. Continue. The other thing that you'll need to do here is specify the storage option for the environment.

5:51 So the key thing to point out at this point is we were talking about environments as corresponding to a branch in get.
5:51 So the key thing to point out at this point is we were talking about environments as corresponding to a branch in get.

6:02 That is true, but it's also these other things like the Kafka options that you select and also the storage options.
6:02 That is true, but it's also these other things like the Kafka options that you select and also the storage options.

6:11 Now the storage option that you're selecting here is for any data that you might possess. So in Quix it's possible to persist the data that's published to topics.
6:11 Now the storage option that you're selecting here is for any data that you might possess. So in Quix it's possible to persist the data that's published to topics.

6:27 You can store that using our something called the data catalog and if you store, if you persist your topics and store the messages that are in the topics.
6:27 You can store that using our something called the data catalog and if you store, if you persist your topics and store the messages that are in the topics.

6:43 In our storage facilities, then there's a small charge associated with that. However, you don't have to persist topics. You can use external database solutions for storage of your data.
6:43 In our storage facilities, then there's a small charge associated with that. However, you don't have to persist topics. You can use external database solutions for storage of your data.

7:01 So you can use. Some of our standard connectors or even write your own connector to connect to more or less any database technology that you want.
7:01 So you can use. Some of our standard connectors or even write your own connector to connect to more or less any database technology that you want.

7:11 The other thing that's stored here is metadata associated with messages. So for now, I'm just going to choose the standard option and create.
7:11 The other thing that's stored here is metadata associated with messages. So for now, I'm just going to choose the standard option and create.

7:23 The environment. Okay, that's it. We'll wait for the project and the environment to be created. And then we'll have a look at what's in there in subsequent videos.
7:23 The environment. Okay, that's it. We'll wait for the project and the environment to be created. And then we'll have a look at what's in there in subsequent videos.

7:42 Okay, thanks for watching. And see you in the next video.
7:42 Okay, thanks for watching. And see you in the next video.

If you want to use the Quix AWS S3 service (optional), you'll need to provide your credentials for accessing AWS S3.

Expand Down
14 changes: 8 additions & 6 deletions docs/tutorials/predictive-maintenance/alert-service.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Sends alerts to an output topic when the temperature is under or over the thresh

It receives data from two topics (3d printer data and forecast) and triggers an alert (to output topic `alerts`) if the temperature is under or over the threshold.

![pipline section](./images/alert-pipeline-segment.png)
![pipeline section](./images/alert-pipeline-segment.png)

The default thresholds are as shown in the following table:

Expand All @@ -17,7 +17,7 @@ The default thresholds are as shown in the following table:
| min_hotend_temperature | 245 |
| max_hotend_temperature | 255 |

These thresholds are used to determine if the temperature, or forecast temperature, are under or over the threshold values. If so these alerts are published to the `alerts` topic.
These thresholds are used to determine if the temperature, or forecast temperature, are under or over the threshold values. If so these alerts are published to the `json-alerts` topic.

Note there are different alert types, with the message format for the `no-alert` type alert:

Expand All @@ -26,7 +26,7 @@ Note there are different alert types, with the message format for the `no-alert`
"status": "no-alert",
"parameter_name": "hotend_temperature",
"message": "'Hotend temperature' is within normal parameters",
"alert_timestamp": 1701280033000000000,
"timestamp": 1701280033000000000,
"alert_temperature": 246.04148121958596
}
```
Expand All @@ -37,7 +37,7 @@ An example of the `under-now` alert message format:
{
"status": "under-now",
"parameter_name": "bed_temperature",
"alert_timestamp": 1701273328000000000,
"timestamp": 1701273328000000000,
"alert_temperature": 104.0852349596566,
"message": "'Bed temperature' is under the threshold (105ºC)"
}
Expand All @@ -50,7 +50,7 @@ Here's an `over-forecast` alert message format:
"status": "over-forecast",
"parameter_name": "forecast_fluctuated_ambient_temperature",
"alert_temperature": 55.014602460947586,
"alert_timestamp": 1701278280000000000,
"timestamp": 1701278280000000000,
"message": "'Ambient temperature' is forecasted to go over 55ºC in 1:36:29."
}
```
Expand All @@ -62,14 +62,16 @@ Here's the `under-forecast` alert message format:
"status": "under-forecast",
"parameter_name": "forecast_fluctuated_ambient_temperature",
"alert_temperature": 44.98135836928914,
"alert_timestamp": 1701277320000000000,
"timestamp": 1701277320000000000,
"message": "'Ambient temperature' is forecasted to fall below 45ºC in 1:20:28."
}
```

<!--
These alerts are subscribed to by the Printers dashboard service, and the alerts can be displayed in real time on the scrolling charts, as well as the scrolling alert display:

![Alerts](./images/alerts-display.png)
-->

## Check the log messages

Expand Down
46 changes: 12 additions & 34 deletions docs/tutorials/predictive-maintenance/data-generator.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The [forecasting algorithm](./forecast-service.md) that attempts to estimate whe

## Data published

The generated data is published to the `3d-printer-data` topic:
The generated data is published to the `json-3d-printer-data` topic:

* Ambient temperature
* Ambient temperature with fluctuations
Expand All @@ -33,35 +33,13 @@ If you look at the messages in the `Messages` view, you'll see data has the foll

``` json
{
"Epoch": 0,
"Timestamps": [
1701277527000000000
],
"NumericValues": {
"hotend_temperature": [
250.8167407832582
],
"bed_temperature": [
106.9299672495977
],
"ambient_temperature": [
36.92387946005222
],
"fluctuated_ambient_temperature": [
36.92387946005222
]
},
"StringValues": {
"original_timestamp": [
"2023-11-29 17:05:27"
]
},
"BinaryValues": {},
"TagValues": {
"printer": [
"Printer 72"
]
}
"hotend_temperature": 249.52922614294954,
"bed_temperature": 110.12854118355098,
"ambient_temperature": 38.70099292962708,
"fluctuated_ambient_temperature": 38.70099292962708,
"timestamp": "2024-04-16T17:07:03.717628",
"original_timestamp": "2024-04-16T17:07:03.717628",
"printer": "Printer 33"
}
```

Expand Down Expand Up @@ -91,15 +69,15 @@ Review the code, you'll see that data is generated for each printer, and each pr
tasks = []
printer_data = generate_data()

# Distribute all printers over the data length
delay_seconds = int(os.environ['datalength']) / replay_speed / number_of_printers
# Distribute all printers over the data length (defaults to 60 seconds)
delay_seconds = get_data_length() / replay_speed / number_of_printers

for i in range(number_of_printers):
# Set stream ID or leave parameters empty to get stream ID generated.
# Set MessageKey/StreamID or leave parameters empty to get a generated message key.
name = f"Printer {i + 1}" # We don't want a Printer 0, so start at 1

# Start sending data, each printer will start with some delay after the previous one
tasks.append(asyncio.create_task(generate_data_and_close_stream_async(topic_producer, name, printer_data.copy(), delay_seconds * i)))
tasks.append(asyncio.create_task(generate_data_async(topic, producer, name, printer_data.copy(), int(delay_seconds * i))))

await asyncio.gather(*tasks)
```
Expand Down
Loading