Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nodejs #27

Merged
merged 10 commits into from
Apr 3, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 8 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
# Node
node_modules
dist

.DS_Store
.idea
.vscode

# Compiled class file
*.class

Expand Down
32 changes: 18 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,53 +1,57 @@
# Flowing Retail

This sample application demonstrates a simple order fulfillment system decomposed into multiple independant components (e.g. microservices).
This sample application demonstrates a simple order fulfillment system, decomposed into multiple independent components (that is to say: _microservices_).

There is code for multiple implementation alternatives to allow a broad audience to undestand the code or to compare alternatives. The table below lists these alternatives.
The repository contains code for multiple implementation alternatives to allow a broad audience to understand the code and to compare alternatives. The [table below](#alternatives) lists these alternatives.

The example respects learnings from **Domain Driven Design (DDD)**, Event Driven Architecture (EDA) and **Microservices (µS)** and should give you a very hands-on access to certain topics.
The example respects learnings from **Domain Driven Design (DDD)**, Event Driven Architecture (EDA) and **Microservices (µS)** and is designed to give you hands-on access to these topics.

**Note:** The code was written in order to be explained. Hence I favored simplified code or copy & paste over production-ready code with generic solutions. **Don't consider the coding style best practice! It serves the purpose to have easily explainable code**.
**Note:** The code was written in order to be explained. Hence, I favored simplified code or copy & paste over production-ready code with generic solutions. **Don't consider the coding style best practice! It is purpose-written to be easily explainable code**.

Flowing retail simulates a very easy order fulfillment system:

![Events and Commands](docs/workflow-in-service.png)

## Architecture and implemenation alternatives
<a name = "alternatives"></a>

## Architecture and implementation alternatives

The most fundamental choice is to select the **communication mechanism**:

* **[Apache Kafka](kafka/)** as event bus (could be easily changed to messaging, e.g. RabbitMQ): [](docs/architecture.png)
* **[REST](rest/)** communication between Services
* This example also shows how to do **stateful resilience patterns** like **stateful retries** leveraging a workflow engine
* **[Zeebe](zeebe/)** broker doing work distribution
* **[REST](rest/)** communication between Services.
* This example also shows how to do **stateful resilience patterns** like **stateful retries** leveraging a workflow engine.
* **[Zeebe](zeebe/)** broker doing work distribution.

Having chosen that your probably can choose the **workflow engine**:
After the communication mechanism, the next choice is the **workflow engine**:

* **Camunda BPM 7**
* **Zeebe** (if you are interessted why Zeebe is listed in the communication mechanism as well as workflow engine please look into the [Zeebe example readme](zeebe/))
* **Zeebe** (if you are interested why Zeebe is listed in the communication mechanism as well as workflow engine please look into the [Zeebe example readme](zeebe/))
* **Zeebe hosted on Camunda Cloud**

and the **programming language**:

* **Java**
* **Go**
* **JavaScript / TypeScript**

## Storyline

Flowing retail simulates a very easy order fulfillment system. The business logic is separated into the services shown above (shown as [context map](https://www.infoq.com/articles/ddd-contextmapping)).
Flowing retail simulates a very easy order fulfillment system. The business logic is separated into the services shown above (shown as a [context map](https://www.infoq.com/articles/ddd-contextmapping)).

### Long running services and orchestration

Some services are **long running** in nature, as e.g. the payment service ask customers to update expired credit cards. Hence a workflow engine is used to persist and control these long running interactions.
Some services are **long running** in nature - for example: the payment service asks customers to update expired credit cards. A workflow engine is used to persist and control these long running interactions.

### Workflows live within service boundaries

An important thought is, that this state machine (or workflow engine in this case) is a library used **within** one service. If different services need a workflow engine they potentally run multiple engines. This way it is an autonomous team decision if they want to use some framework and which one:
Note that the state machine (_or workflow engine in this case_) is a library used **within** one service. If different services need a workflow engine they can run whatever engine they want. This way it is an autonomous team decision if they want to use a framework, and which one:

![Events and Commands](docs/workflow-in-service.png)

### Resilience patterns for synchronous communication

Depending on the means of communication (e.g. messaging vs. blocking synchronous REST calls) you also have to tackle quite basic communication problems, e.g. by stateful retries.
You also have to deal with basic communication problems, the specifics of which depend on the means of communication (for example: _asynchronous messaging_ vs. _blocking synchronous REST calls_). You might use stateful retries for this.

![V1](docs/resilience-patterns/v1.png)

Expand Down
File renamed without changes.
53 changes: 31 additions & 22 deletions rest/README.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,22 @@
# Flowing Retail / REST

This folder contains services that connect via REST. Currently this is reduced to showcasing resilience patterns.
This folder contains services that connect via REST. Currently, this is reduced to showcasing resilience patterns.

# Sample service demonstrating stateful resilience patterns in a REST environment

This sample (micro-)service can retrieve payments and therefor needs to be called via REST. It requires an upstream REST service to charge credit cards.
This sample REST (micro-)service effects payments in response to a PUT call. It requires an upstream REST service that charges credit cards.

![REST callstack](../docs/resilience-patterns/situation.png)

This simple call-chain is great to demonstrate various important resilience patterns.
This simple call-chain is perfect for demonstrating important resilience patterns.

There are the following technology choices available for the code demos:
The following technology choices are available for the code demos:

* [**Java**](java/payment) + Spring, Hystrix, Camunda
* [**C#**](csharp/payment) + Polly, Camunda
* [**Node.js**](nodejs/payment) + [Brakes](https://github.com/awolden/brakes), Zeebe on Camunda Cloud

There is a stripped down version available for:
There is a stripped-down version available for:

* [**GoLang**](go/payment-zeebe), Zeebe

Expand All @@ -24,44 +25,50 @@ There is a stripped down version available for:

See **Fail fast is not enough**: https://blog.bernd-ruecker.com/fail-fast-is-not-enough-84645d6864d3

Assume the credit card service goes nuts, meaning it still responds, but very slow. Having no pattern in place this is the worst thing that can happen - as now the payment service will call the credit card service and block until he gets a response. As this take a long time all threads from the payment service are hold hostile and the payment service will eventually time out for its clients. Tiny failures somewhere in your system might blow up your whole system:
Let's assume a scenario where the upstream credit card service still responds, but its very slow. With no resilience pattern in place, this is the worst thing that can happen - as now the payment service will call the credit card service and block until it gets a response. As this take a long time, all threads from the payment service are held hostage, and the payment service will eventually time out for its clients. Tiny failures somewhere in your system might blow up your whole system:

![V1](../docs/resilience-patterns/v1.png)

* Java: [PaymentRestHacksControllerV1.java](java/payment/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV1.java)
* C#: [PaymentControllerV1](csharp/payment/Controllers/PaymentController.cs#L16)
* Node.js: [controller-v1.ts](nodejs/payment-zeebe/routes/controller-v1.ts)

## Fail fast

The least you have to do is to apply a **fail fast** pattern like [**circuit breaker**](https://martinfowler.com/bliki/CircuitBreaker.html). In this example I use [Netflix Hystrix](https://github.com/Netflix/Hystrix). If a service responds to slow the circuit breaker interrupts and the payment service gets a failure right away. This way you make sure the overall system is still responding, even if functionality degrades (I cannot charge credit cards).
A simple mitigation is to apply a **fail fast** pattern like [**circuit breaker**](https://martinfowler.com/bliki/CircuitBreaker.html). In this example I use [Netflix Hystrix](https://github.com/Netflix/Hystrix) (_and [Polly](https://github.com/App-vNext/Polly) for C# / [Brakes](https://github.com/awolden/brakes) for Node.js, which provide equivalent functionality_). If a service responds too slowly, the circuit breaker interrupts and the payment service gets a failure right away. This way you make sure the overall system is still responding, even if functionality degrades (meaning: we cannot charge credit cards).

![V2](../docs/resilience-patterns/v2.png)

* Java: [PaymentRestHacksControllerV2.java](java/payment/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV2.java#L41)
* C#: [PaymentControllerV2](csharp/payment/Controllers/PaymentController.cs#L74)
* Node.js: [controller-v2.ts](nodejs/payment-zeebe/routes/controller-v2.ts#13)


## Fail fast is not enough

But failing fast is not enough. Very often a retry after the credit card service has been fixed resolves the situation. This retry needs to be stateful to not only retry right away but again in a couple of minutes, hours or even days. Having this stateful retrying as possibility keeps the failure handling local to payment making the whole architecture less complex.
Failing fast is good, but it is not enough. Frequently, a retry after the credit card service has been fixed resolves the situation (if the service was in a hard failure mode) - or a retry may discover that the earlier attempt succeeded, but took an abnormal amount of time (if the service is in a degraded performance mode). This retry needs to be stateful to not only retry right away but again in a couple of minutes, hours or even days. Keeping this stateful retry local to the payment service reduces overall architectural complexity.

![V3](../docs/resilience-patterns/v3.png)

In the example I use the [Camunda workflow engine](http://camunda.com/) to do the stateful retry reliably.
In the example, I use the [Camunda workflow engine](http://camunda.com/) (or Zeebe in [Camunda Cloud](https://camunda.io)) to do the stateful retry reliably.

* Java
* [PaymentRestHacksControllerV3.java](java/payment/src/main/java/io/flowing/retail/payment/port/resthacks/PaymentRestHacksControllerV3.java#L45)
* The workflow is created by Java DSL
* C#
* [PaymentControllerV3](csharp/payment/Controllers/PaymentController.cs#L110)
* [PaymentV3.bpmn](csharp/payment/Models/PaymentV3.bpmn)
* Node.js
* [controller-v3.ts](nodejs/payment-zeebe/routes/controller-v3.ts#25)
* [paymentV3.bpmn](nodejs/payment-zeebe/bpmn/paymentV3.bpmn)

## Keep synchronous responses

The processing just got asynchronous, which is often not wanted. In this scenario you could very well return a synchronous response whenever the credit card service is available, but switch to asynchronicity only if it is not.
The processing just got asynchronous, which is often not wanted. In this scenario you could very well return a synchronous response whenever the credit card service is available, but switch to asynchronicity when it is not.

![V4](../docs/resilience-patterns/v4.png)

HTTP supports this by differntiating the return code (200 OK means all OK, 202 ACCEPTED means I call you back later).
HTTP supports this via return codes: `200 OK` means "_all OK_", `202 ACCEPTED` means "_I'll call you back later_".

![Sync vs. async](../docs/resilience-patterns/syncAsync.png)

Expand All @@ -71,14 +78,16 @@ HTTP supports this by differntiating the return code (200 OK means all OK, 202 A
* C#
* [PaymentControllerV4](csharp/payment/Controllers/PaymentController.cs#L159).
* [PaymentV4.bpmn](csharp/payment/Models/PaymentV4.bpmn)

* Nodejs
* [controller-v4.ts](nodejs/payment-zeebe/routes/controller-v3.ts#25)
* [paymentV4.bpmn](nodejs/payment-zeebe/bpmn/paymentV3.bpmn)


## Asynchronous work distribution without messaging

An alternative to synchronously call an upstream service is to communicate asynchronously. The default would be messaging.
An alternative to synchronously calling an upstream service is to communicate asynchronously. The default would be messaging.

In this example I show one alternative which a lot of customers use very sucessfully: Using the workflow engine as work distribution, behaving like a queue. Therefor I leveraged a concept called [External Tasks](https://docs.camunda.org/manual/latest/user-guide/process-engine/external-tasks/).
This example shows a successful approach taken by many customers: using the workflow engine as work distribution, behaving like a queue. This leverages the [External Tasks](https://docs.camunda.org/manual/latest/user-guide/process-engine/external-tasks/) pattern.

![Microservices](../docs/resilience-patterns/v5.png)

Expand All @@ -91,25 +100,24 @@ In this example I show one alternative which a lot of customers use very sucessf
* [PaymentControllerV6](csharp/payment/Controllers/PaymentController.cs#L74)



## Business transactions using compensation

The last part of the example adds compensation to the game. In distributed systems ACID transactions are not applicable (or at least do not scale well). Using compensation is the alternative - meaning that you reliably undo already executed work if something later on fails.
The last part of the example adds compensation to the game. In distributed systems, ACID transactions are not applicable (or at least do not scale well). Using compensation is the alternative - meaning that you reliably undo already executed work if something later on fails.

![Microservices](../docs/resilience-patterns/v6.png)

See [payment6.bpmn / Java](java/payment/src/main/resources/payment6.bpmn) and [PaymentV6.bpmn / C#](csharp/payment/Models/PaymentV6.bpmn) for the workflow


# How-to run

You have to startup both services:
* Stripe Fake Server
* Payment Service

This varies:
* [How to run in Java](java/payment)
* [How to run in .NET](csharp/payment)
* [How to run in Java](java/payment-camunda/README.md)
* [How to run in .NET](csharp/payment/README.md)
* [How to run in Node.js](nodejs/payment-zeebe/README.md)

Now the different versions of the payment service are available:

Expand All @@ -127,9 +135,6 @@ curl \
http://localhost:8100/api/payment/v1
```




## Hint on using Camunda Enterprise Edition

For Camunda there is an enterprise edition available with [https://camunda.com/products/cockpit/#/features](additional features in Cockpit) (the monitoring tool). It is quite handy to use this when playing around with the example. You can easily switch to use enterprise edition:
Expand All @@ -139,3 +144,7 @@ For Camunda there is an enterprise edition available with [https://camunda.com/p
* .NET: Download the Camunda enterprise version

Note that you do not need the enterprise edition to run the examples, the community edition will also do fine. But because of less features you do not see historical workflow instances - and that means you do not see that much in Camunda Cockpit if everything runs smoothly.

## Using Camunda Cloud

You can get a free hosted instance of Zeebe in Camunda Cloud at [https://camunda.io](https://camunda.io).
6 changes: 1 addition & 5 deletions rest/csharp/payment/NodeJs-CustomeCreditWorker/index.js
Original file line number Diff line number Diff line change
@@ -1,14 +1,10 @@
var request = require('request');
const { Client, Variables } = require("camunda-external-task-client-js");

const config = { baseUrl: "http://localhost:8080/engine-rest/engine/default/", interval: 50};
const client = new Client(config);

client.subscribe("customer-credit", async function({ task, taskService }) {
var remainingAmount = 0;
if (Math.random() > 0.3) {
remainingAmount = 15;
}
const remainingAmount = (Math.random() > 0.3) ? 15 : 0;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh no - I never understand these stuff - I deliberately invest more lines to get code I unserstand without thinking ;-)


const processVariables = new Variables();
processVariables.set("remainingAmount", remainingAmount);
Expand Down
34 changes: 34 additions & 0 deletions rest/nodejs-nest/payment-zeebe/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# compiled output
/dist
/node_modules

# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
lerna-debug.log*

# OS
.DS_Store

# Tests
/coverage
/.nyc_output

# IDEs and editors
/.idea
.project
.classpath
.c9/
*.launch
.settings/
*.sublime-workspace

# IDE - VSCode
.vscode/*
!.vscode/settings.json
!.vscode/tasks.json
!.vscode/launch.json
!.vscode/extensions.json
4 changes: 4 additions & 0 deletions rest/nodejs-nest/payment-zeebe/.prettierrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"singleQuote": true,
"trailingComma": "all"
}
33 changes: 33 additions & 0 deletions rest/nodejs-nest/payment-zeebe/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
<p align="center">
<a href="http://nestjs.com/" target="blank"><img src="https://nestjs.com/img/logo_text.svg" width="320" alt="Nest Logo" /></a>
</p>

[travis-image]: https://api.travis-ci.org/nestjs/nest.svg?branch=master
[travis-url]: https://travis-ci.org/nestjs/nest
[linux-image]: https://img.shields.io/travis/nestjs/nest/master.svg?label=linux
[linux-url]: https://travis-ci.org/nestjs/nest

<p align="center">A progressive <a href="http://nodejs.org" target="blank">Node.js</a> framework for building efficient and scalable server-side applications, heavily inspired by <a href="https://angular.io" target="blank">Angular</a>.</p>
<p align="center">

## Description

A [Nest](https://github.com/nestjs/nest) implementation of the Flowing Retail demo, using the [NestJS Zeebe integration](https://www.npmjs.com/package/@payk/nestjs-zeebe).

## Installation

```bash
$ npm install
```

## Running the app

```bash
# development
$ npm run start

# watch mode
$ npm run start:dev
```