Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(docs): update pokeshop and kafka trigger docs #3129

Merged
merged 3 commits into from
Sep 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Binary file added docs/docs/img/choose-trigger-0.13.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/docs/img/create-test-0.13.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
31 changes: 19 additions & 12 deletions docs/docs/live-examples/pokeshop/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

As a testing ground, the team at Tracetest has implemented a sample instrumented API around the [PokeAPI](https://pokeapi.co/).

The idea is to have a microservice-divided system that behaves like a typical scenario by having async processes ([RabbitMQ](https://www.rabbitmq.com/)), cache layers ([Redis](https://redis.io/)), database storage ([Postgres](https://www.postgresql.org/)) and simple CRUD interfaces for Pokemons.
The idea is to have a microservice-divided system that behaves like a typical scenario by having async processes ([RabbitMQ](https://www.rabbitmq.com/) and [Kafka](https://kafka.apache.org/)), cache layers ([Redis](https://redis.io/)), database storage ([Postgres](https://www.postgresql.org/)) and simple CRUD interfaces for Pokemons.

With this, users can get familiar with the Tracetest tool by focusing on creating assertions, visualizing the trace and identifying the different data that comes from the Collector ([Jaeger](https://www.jaegertracing.io/)). Users will learn about basic instrumentation practices: what tools to use, what data to send, when, and what suggested standards need to be followed.

Expand All @@ -18,15 +18,17 @@ We have three use cases that use each component of this structure and that can b
- [Get Pokemon by ID](./use-cases/get-pokemon-by-id.md): Given a Pokemon ID, this endpoint returns the data of a Pokemon. If the same Pokemon was queried, the API will use its cache to return it.
- [List Pokemon](./use-cases/list-pokemon.md): Lists all Pokemons registered into Pokeshop.
- [Import Pokemon](./use-cases/import-pokemon.md): Given a Pokemon ID, this endpoint does an async process, going to PokeAPI to get Pokemon data and adding it to the database.
- [Import Pokemon from Stream](./use-cases/import-pokemon-from-stream.md): Listening to a Stream, this use case also does an async process, going to PokeAPI to get Pokemon data and adding it to the database.

## System Architecture

The system is divided into two components:

- an **API** that serves client requests,
- a **Worker** who deals with background processes.
- a **Queue Worker** who deals with background processes, receiving data from the API
- a **Stream Worker** who handles import events sent from a stream

The communication between the API and Worker is made using a `RabbitMQ` queue, and both services emit telemetry data to Jaeger and communicate with a Postgres database.
The communication between the API and Queue Worker is made using a `RabbitMQ` queue, and both services emit telemetry data to Jaeger and communicate with a Postgres database. Additionally, a Stream Worker listens to a `Kafka` stream to see if there is any import event sent on it to execute.

A diagram of the system structure can be seen here:

Expand All @@ -36,16 +38,21 @@ flowchart TD
B[(Postgres)]
C(Node.js API)
D(RabbitMQ)
E(Worker)
E(Queue Worker)
F(Jaeger)

C --> |IORedis| A
C --> |Sequelize| B
C --> |ampqlib| D
D --> |ampqlib| E
E --> |Sequelize| B
C --> |OpenTelemetry Node.js SDK| F
E --> |OpenTelemetry Node.js SDK| F
G(Kafka)
H(Stream Worker)

C --> A
C --> B
C --> D
D --> E
E --> B
C --> F
E --> F
G --> H
H --> B
H --> F
```

In our live tests, we are deploying into a single Kubernetes namespace, deployed via a [Helm chart](https://github.com/kubeshop/pokeshop/blob/master/docs/installing.md#run-on-a-kubernetes-cluster).
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# Pokeshop API - Import Pokemon from Stream

This use case showcases a more complex scenario involving an async process. Usually, when working with microservices, there are use cases where some of the processing needs to happen asynchronously. For example, when triggering a user notification, generating reports or processing a payment order. With this endpoint, we provide an example of how users can implement trace-based testing for such scenarios.

Here the process listens to a stream, and whenever an event is read from it, the following process is triggered:
```mermaid
sequenceDiagram
participant Stream as Kafka
participant Worker as Stream Worker
participant ExternalAPI as PokeAPI
participant Database as Postgres

Stream->>Worker: read "import" message

Worker->>ExternalAPI: get pokemon info
ExternalAPI-->>Worker: pokemon info

Worker->>Database: save pokemon
Database-->>Worker: pokemon saved
```

You can trigger this use case by sending a message to Kafka on the `pokemon` topic with the following message value:
```json
{
"id": 143
}
```

## Building a Test for This Scenario

Using Tracetest, we can [create a test](../../../web-ui/creating-tests.md) that will produce a message to Kafka on `pokemon` topic and validate the following properties:
- The worker should read the import task.
- PokeAPI should return a valid response.
- The database should respond with low latency (< 200ms).

### Traces

Running these tests for the first time will create a distributed trace like the image below, where you can see spans for the stream messaging, the PokeAPI (external API) call and database calls.

![](../images/import-pokemon-from-stream-trace.png)

### Assertions

With this trace, we can build [assertions](../../../concepts/assertions.md) with Tracetest and validate the API and Worker behavior:

- **A message was received from Kafka stream:**
![](../images/import-pokemon-from-stream-message-received.png)

- **Import Pokemon use case was triggered**:
- ![](../images/import-pokemon-from-stream-use-case-executed.png)

- **PokeAPI should return a valid response:**
![](../images/import-pokemon-from-stream-get-pokeapi.png)

- **The database should respond with low latency (< 200ms):**
![](../images/import-pokemon-from-stream-database-latency.png)

Now you can validate this entire use case.

### Test Definition

If you want to replicate this entire test with Tracetest, you can replicate these steps in the Web UI or using the CLI, saving the following test definition as the file `test-definition.yml` and running:

```sh
tracetest run test -f test-definition.yml
```

```yaml
type: Test
spec:
id: a97syfdkjad
name: Import a Pokemon reading a Stream
description: Import a Pokemon via Stream
trigger:
type: kafka
kafka:
brokerUrls:
- stream:9092
topic: pokemon
headers: []
messageKey: snorlax-key
messageValue: "{\"id\":143}"
specs:
- selector: span[tracetest.span.type="messaging" name="pokemon process" messaging.system="kafka" messaging.destination="pokemon" messaging.destination_kind="topic" messaging.operation="process"]
name: A message was received from Kafka stream
assertions:
- attr:messaging.system = "kafka"
- selector: span[tracetest.span.type="general" name="import pokemon"]
name: Import Pokemon use case was triggered
assertions:
- attr:name = "import pokemon"
- selector: span[tracetest.span.type="http" name="GET" http.method="GET"]
name: PokeAPI should return a valid response
assertions:
- attr:http.response.body = '{"name":"snorlax"}'
- attr:http.status_code = 200
- selector: span[tracetest.span.type="database"]
name: The database should respond with low latency (< 200ms)
assertions:
- attr:tracetest.span.duration <= 200ms

```
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ sequenceDiagram
```mermaid
sequenceDiagram
participant Queue as RabbitMQ
participant Worker as Worker
participant Worker as Queue Worker
participant ExternalAPI as PokeAPI
participant Database as Postgres

Expand Down
7 changes: 4 additions & 3 deletions docs/docs/web-ui/creating-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,20 @@ Click the **Create** button and select **Create New Test** in the drop down:

The "Create New Test" dialog appears:

![Create a Test](../img/create-test-0.11.png)
![Create a Test](../img/create-test-0.13.png)

The option to choose the kind of trigger to initiate the trace is presented:

- HTTP Request - Create a basic HTTP request.
- GRPC Request - Test and debug your GRPC request.
- cURL Command - Define your HTTP test via a cURL command.
- Postman Collection - Define your HTTP request via a Postman collection.
- TraceID - Define you test via a TraceID.
- TraceID - Define your test via a TraceID.
- Kafka - Test consumers with Kafka messages

Choose the trigger and click **Next**:

![Choose Trigger](../img/choose-trigger-0.11.png)
![Choose Trigger](../img/choose-trigger-0.13.png)

In this example, HTTP Request has been chosen.

Expand Down
5 changes: 5 additions & 0 deletions docs/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,11 @@ const sidebars = {
id: "live-examples/pokeshop/use-cases/import-pokemon",
label: "Import Pokemon",
},
{
type: "doc",
id: "live-examples/pokeshop/use-cases/import-pokemon-from-stream",
label: "Import Pokemon from Stream",
},
],
},
],
Expand Down