ms-session-counter

This is a Redis + Nodejs backend microservice used for counting sessions from another apps.

It can keep the session count by application, node and day (and hours). Very useful for realtime dashboards.

It is a REST API which has 3 methods:

1. /counter (POST) method. It receives the application, node and timestamp of a new session and increases the hourly and daily counts.
2. /kpi (GET) method. It retrieves the hourly and daily counters of a particular application, node and date.
3. /home (GET) method. It checks the microservice and Redis status doing a PING command and sending back an OK response.

Session counter architecture

Redis log

How it works

How the data is stored:

Counters are stored in Redis Hashes. It has the benefit of saving a whole day of session counts, having a small footprint and at the same time be able to get a whole day of data in one single record.

Session counter (Hash) definition

The key has the format:

kpi:sessions:app:{application}:node:{nodeNumber}:day:{YYYYMMDD}

Key composition follows Redis suggestions.

Prefix	Value
kpi	sessions - string (fixed value)
app	Application name - string
node	Node number - integer (starts at 0)
day	Sessions date- string in format: YYYYMMDD

Hash values definition:

Key	Value	Nullable
day	Sessions count during the day	No
0	Sessions count during 0 hour	Yes
1	Sessions count during 0 hour	Yes
N	Sessions count during N hour	Yes
23	Sessions count during 23 hour	Yes

The application uses transactions to ensure unique counts and avoid race conditions.

How the data is accessed:

At writting

1. Nodejs calculates the key and hour, using the function _getKeyAndHour. It receives the epoch timestamp in UTC timezone so it can be used anywhere in the world.
2. With key and hour defined, app increases day and hour keys in that application, node and day record in one single transaction using the function incrementKpi. Redis commands used:
   • WATCH. To block that particular hash and avoid race conditions.
   • MULTI. It start a transaction.
   • HINCRBY. To increment day and hour counts.
   • EXEC. Executes the transaction.

At reading

• HGETALL. Gets in one single round the stats for a whole day.

At checking Redis state

• PING. Gets an confimation answer from Redis.

API calls examples.

http://localhost:3000/counter endpoint.

This endpoint increments the session counter for an application, node, day and hour, posting the next JSON data.

Request body.

{
    "app": "local",
    "node": 0, 
    "timestamp": 1660054277326
}

The http method is POST.

curl -X POST \
    -H 'Content-Type:application/json' \
    http://localhost:3000/counter \
    -d '{"app":"local", "node":0, "timestamp":1660054277326}' 

NOTE: It's very important to include the header: Content-Type:application/json

Service response.

{
    "type":"WebResponse",
    "isError":false,
    "description":"OK.",
    "data": {
        "type":"CounterResponse","key":"kpi:sessions:app:local:node:0:day:20220809",
        "day":2,
        "hour":2
    }
}

http://localhost:3000/kpi endpoint.

This endpoint get the session counter for an application, node and day. Using the key definition. The http method is GET and uses the query param key.

curl -X GET http://localhost:3000/kpi\?key\=kpi:sessions:app:local:node:0:day:20220819

Service response.

{
    "type": "WebResponse",
    "isError": false,
    "description": "OK.",
    "data": {
        "0": "3000",
        "1": "3200",
        "day": "6200"
    }
}

http://localhost:3000/home endpoint.

This endpoint makes a PING to Redis database and returns a positive answer when database is up and runnning, otherwise in returns an error.

The http method is GET.

curl -X GET http://localhost:3000/home 

Service response.

{
    "type":"WebResponse",
    "isError":false,
    "description":"Service OK.",
    "data":null
}

Performance Benchmarks

I used Apache JMeter to run a performance test.

Here is the test file.

Settings of JMeter test:

Results

Test was executed at a Macbook pro with the next characteristics:

Feature	Value
Modelo	MacBook Pro
Identifier	MacBookPro11,2
Processor	Intel Core i7 (4 Cores)
Processor speed	2 GHz
Processor number	1
Cores	4
Cache level 2 (per core)	256 KB
Caché level 3	6 MB
Hyper-Threading	Activated
RAM	8 GB

The service can handle 252.4 requests per second over on single counter.

That response time may go down when using different applications and nodes, because fo the blocking of different records (keys).

Application may also implement Lua scripting to reduce the response times.

How to run it locally?

Prerequisites

• Redis version 6.2.6+
• Nodejs version 12.22.5+

Local installation

Instructions for MAC or Linux.

Install Redis server.

• Check your Redis installation executing

  redis-server --version

• Create folder /etc/opt/redis/config

  sudo mkdir -p /etc/opt/redis

• Create data folder /etc/opt/redis/data

  sudo mkdir -p /etc/opt/redis/data

• Add write permissions to data folder /etc/opt/redis/data

  sudo chmod a+wr /etc/opt/redis/data

• Copy ALL files contained in redis_config folder in folder /etc/opt/redis/config

  sudo cp -R config /etc/opt/redis/

Install Nodejs. You also can use these instructions.

• Check your Nodejs installation executing

  node --version

Redis ACL.

This service uses Redis ACLs for improving security. You can check the Redis security documentation.

The current ACL has 3 users:

• administrator. This is the Redis database administrator user. It has all the permissions available in Redis.
• service. This is the service user which has enabled only operative Redis commands.
• replicant. This role user thought for being used in Redis cluster instances. NOT IMPLEMENTED YET.

These are the defaulted passwords:

User	Password
administrator	Redis_Admin!
service	Redis_Service!
replicant	Redis_Replica!

ms-session-counter application is configured to work using the default service user and password.

Redis database can be managed using the administrator user, password and redis-client cli. But also can be accessed by the other two users. You can use the AUTH command to authenticate into the server.

IMPORTANT NOTE!

It's highly recommended to change the default passwords in production environments. You can do it, calculating the SHA256 hash of the password using the SHA256 hash calculator and replace the password hash values in the file redis_config/users.acl.

Local configuration

1. Open a terminal and Start Redis server locally using the next command.

redis-server /etc/opt/redis/config/redis_local.conf

2. In a new terminal. Create a .env file at the root folder with your own parameters. You can copy the .env-template file to .env file and use it as a starter point.

cp .env-template .env

If in production, change .env file parameters as needed. Save the file.

3. Set your local environment parameters, executing the next command:

source .env

4. Install nodejs modules. Execute the next command:

npm install

5. With Redis set and running. Start the Session counter app. Using the command:

node app.js

6. Start doing API requests to the Session counter app, as noted in API calls examples section.

Deployment

To make deploys work, you need to create free account on Redis Cloud

Google Cloud Run

Heroku

Vercel

More Information about Redis Stack

Here some resources to help you quickly get started using Redis Stack. If you still have questions, feel free to ask them in the Redis Discord or on Twitter.

Getting Started

1. Sign up for a free Redis Cloud account using this link and use the Redis Stack database in the cloud.
2. Based on the language/framework you want to use, you will find the following client libraries:
   • Redis OM .NET (C#)
     • Watch this getting started video
     • Follow this getting started guide
   • Redis OM Node (JS)
     • Watch this getting started video
     • Follow this getting started guide
   • Redis OM Python
     • Watch this getting started video
     • Follow this getting started guide
   • Redis OM Spring (Java)
     • Watch this getting started video
     • Follow this getting started guide

The above videos and guides should be enough to get you started in your desired language/framework. From there you can expand and develop your app. Use the resources below to help guide you further:

1. Developer Hub - The main developer page for Redis, where you can find information on building using Redis with sample projects, guides, and tutorials.
2. Redis Stack getting started page - Lists all the Redis Stack features. From there you can find relevant docs and tutorials for all the capabilities of Redis Stack.
3. Redis Rediscover - Provides use-cases for Redis as well as real-world examples and educational material
4. RedisInsight - Desktop GUI tool - Use this to connect to Redis to visually see the data. It also has a CLI inside it that lets you send Redis CLI commands. It also has a profiler so you can see commands that are run on your Redis instance in real-time
5. Youtube Videos
   • Official Redis Youtube channel
   • Redis Stack videos - Help you get started modeling data, using Redis OM, and exploring Redis Stack
   • Redis Stack Real-Time Stock App from Ahmad Bazzi
   • Build a Fullstack Next.js app with Fireship.io
   • Microservices with Redis Course by Scalable Scripts on freeCodeCamp