Skip to content

Commit 724081b

Browse files
committed
Fixed PR
1 parent 08753c7 commit 724081b

File tree

11 files changed

+454
-35
lines changed

11 files changed

+454
-35
lines changed
290 KB
Loading
355 KB
Loading
845 KB
Loading
122 KB
Loading
Lines changed: 352 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,352 @@
1+
---
2+
id: index-analytics-using-aws
3+
title: How to Build and Deploy Your Own Analytics Dashboard on AWS using NodeJS and Redis
4+
sidebar_label: Building Analytics Dashboard on using NodeJS and then deploy it on AWS
5+
slug: /create/aws/analytics-using-aws
6+
authors: [ajeet]
7+
---
8+
9+
An interactive analytics dashboard serves several purposes. They allow you to share data and provide you with all those vital information to make game-changing decisions at a faster pace. Building a real-time dynamic dashboard using a traditional relational database might require a complex set of queries. By using a NoSQL database like Redis, you can build a powerful interactive and dynamic dashboard with a small number of Redis commands.
10+
11+
Let’s take a look at how this was achieved.
12+
13+
14+
- What will you build?
15+
- What will you need?
16+
- Getting started
17+
- How does it work?
18+
- How data is stored
19+
- Navigating the application
20+
21+
22+
### 1. What will you build?
23+
24+
You’ll build an analytics dashboard app that uses Redis Bitmap written in NodeJS (JavaScript) and then deploy it to AWS
25+
26+
Ready to get started? Ok, let’s dive straight in.
27+
28+
29+
### 2. What will you need?
30+
31+
- [NodeJS](https://developer.redis.com/develop/node): used as an open-source, cross-platform, backend JavaScript runtime environment that executes Javascript code outside a web browser.
32+
- [Redis Enterprise Cloud](https://developer.redis.com/create/rediscloud): used as a real-time database, cache, and message broker.i
33+
- [NPM](https://www.npmjs.com/): used as a package manager. It allows you to build node apps.
34+
35+
### 3. Getting Started
36+
37+
### Prepare the environment
38+
39+
- Install Node - v12.19.0
40+
- Install NPM - v6.14.8
41+
42+
43+
### Step 1. Sign up for a Free Redis Enterprise Cloud Account
44+
45+
[Follow this tutorial](https://developer.redis.com/create/aws/redis-on-aws) to sign up for a free Redis Enterprise Cloud account.
46+
47+
![image](analytics3.png)
48+
49+
Choose AWS as a Cloud vendor while creating your new subscription. At the end of the database creation process, you will get a Redis Enterprise CLoud database endpoint and password. You can save it for later use.
50+
51+
![image](analytics4.png)
52+
53+
54+
55+
56+
### Step 2. Clone the repository
57+
58+
```bash
59+
git clone https://github.com/redis-developer/basic-analytics-dashboard-redis-bitmaps-nodejs
60+
```
61+
62+
63+
### Step 3. Setting up backend
64+
65+
66+
First we will be settin up environmental variables
67+
68+
Go to /server folder (cd ./server) and then execute the below command:
69+
70+
```bash
71+
cp .env.example .env
72+
```
73+
74+
Open .env file and add Redis Enterprise Cloud Database Endpoint URL, port and password as shown below:
75+
76+
```
77+
78+
PORT=3000
79+
80+
# Host and a port. Can be with `redis://` or without.
81+
# Host and a port encoded in redis uri take precedence over other environment variable.
82+
# preferable
83+
REDIS_ENDPOINT_URI=redis://redis-XXXX.c212.ap-south-1-1.ec2.cloud.redislabs.com:15564
84+
85+
# Or you can set it here (ie. for docker development)
86+
REDIS_HOST=redis-XXXX.c212.ap-south-1-1.ec2.cloud.redislabs.com
87+
REDIS_PORT=XXXX
88+
89+
# You can set password here
90+
REDIS_PASSWORD=reXXX
91+
92+
COMPOSE_PROJECT_NAME=redis-analytics-bitmaps
93+
```
94+
95+
96+
### Step 4. Install dependencies
97+
98+
```bash
99+
npm install
100+
```
101+
102+
103+
### Step 5. Run the backend
104+
105+
```bash
106+
npm run dev
107+
```
108+
109+
### Step 6. Setting up the frontend
110+
111+
Go to /client folder (cd ./client) and then:
112+
113+
```bash
114+
cp .env.example .env
115+
```
116+
117+
Add the exact URL path and port number of your choice for VUE_APP_API_URL parameter as shown below:
118+
119+
```
120+
VUE_APP_API_URL=http://localhost:3000
121+
```
122+
123+
124+
### Step 7. Install dependencies
125+
126+
```bash
127+
npm install
128+
```
129+
130+
### Step 8. Run the frontend
131+
132+
```bash
133+
npm run serve
134+
```
135+
136+
![analytics](analytics_traffic.png)
137+
138+
### How does it work?
139+
140+
#### How the data is stored:
141+
142+
The event data is stored in various keys and various data types.
143+
144+
For each of time spans:
145+
146+
- year: like 2021
147+
- month: like 2021-03 (means March of 2021)
148+
- day: like 2021-03-03 (means 3rd March of 2021)
149+
- weekOfMonth: like 2021-03/4 (means 4th week of March 2021)
150+
- anytime
151+
152+
and for each of scopes:
153+
154+
- source
155+
- action
156+
- source + action
157+
- action + page
158+
- userId + action
159+
- global
160+
161+
and for each of data types (types):
162+
163+
- count (Integer stored as String)
164+
- bitmap
165+
- set
166+
167+
Is generated key like:
168+
169+
170+
```bash
171+
rab:{type}[:custom:{customName}][:user:{userId}][:source:{source}][:action:{action}][:page:{page}]:timeSpan:{timeSpan}
172+
```
173+
174+
where values in [] are optional.
175+
176+
- For each generated key like rab:count:*, data is stored like: INCR {key}
177+
Example:
178+
179+
```bash
180+
INCR rab:count:action:addToCart:timeSpan:2015-12/3
181+
```
182+
- For each generated key like: rab:set:*, data is stored like: SADD {key} {userId}
183+
Example:
184+
185+
```bash
186+
SADD rab:set:action:addToCart:timeSpan:2015-12/3 8
187+
```
188+
- For each generated key like rab:bitmap:*, data is stored like: SETBIT {key} {userId} 1.
189+
Example:
190+
191+
```bash
192+
SETBIT rab:bitmap:action:addToCart:timeSpan:2015-12/3 8 1
193+
```
194+
### Cohort data
195+
196+
- We store users who register and then bought some products (action order matters).
197+
- For each buy action in December we check if user performed register action before (register counter must be greater than zero).
198+
- If so, we set user bit to 1 like: SETBIT rab:bitmap:custom:cohort-buy:timeSpan:{timeSpan} {userId} 1
199+
200+
- E.g User Id 2 bought 2 products on 2015-12-17. It won't be stored.
201+
- E.g User Id 10 bought 1 product on 2015-12-17 and registered on 2015-12-16.
202+
It will be stored like: SETBIT rab:bitmap:custom:cohort-buy:timeSpan:2015-12 10 1.
203+
- We assume that user cannot buy without register.
204+
205+
#### Retention data
206+
207+
- Retention means users who bought on two different dates
208+
- For each buy action we check if user bought more products anytime than bought on particular day (current purchase not included).
209+
- If so, we add user id to set like: SADD rab:set:custom:retention-buy:timeSpan:{timeSpan} {userId}
210+
- E.g User Id 5 bought 3 products on 2015-12-15. His retention won't be stored (products bought on particular day: 2, products bought anytime: 0).
211+
- E.g User Id 3 bought 1 product on 2015-12-15 and before - 1 product on 2015-12-13. His retention will be stored (products bought on particular day: 0, products bought anytime: 1) like: SADD rab:set:custom:retention-buy:timeSpan:2015-12 3.
212+
213+
#### How the data is accessed:
214+
215+
- Total Traffic:
216+
217+
December: ```BITCOUNT rab:bitmap:custom:global:timeSpan:2015-12```
218+
X week of December: ```BITCOUNT rab:bitmap:custom:global:timeSpan:2015-12/{X}```
219+
Example:
220+
221+
```bash
222+
BITCOUNT rab:bitmap:custom:global:timeSpan:2015-12/3
223+
```
224+
225+
- Traffic per Page ({page} is one of: homepage, product1, product2, product3):
226+
227+
```
228+
December: BITCOUNT rab:bitmap:action:visit:page:{page}:timeSpan:2015-12
229+
```
230+
231+
Example:
232+
233+
```bash
234+
BITCOUNT rab:bitmap:action:visit:page:homepage:timeSpan:2015-12
235+
```
236+
237+
- X week of December:
238+
239+
```
240+
BITCOUNT rab:bitmap:action:visit:page:{page}:timeSpan:2015-12/{X}
241+
```
242+
243+
Example:
244+
245+
```bash
246+
BITCOUNT rab:bitmap:action:visit:page:product1:timeSpan:2015-12/2
247+
```
248+
249+
- Traffic per Source ({source} is one of: google, Facebook, email, direct, referral, none):
250+
251+
December:
252+
253+
```
254+
BITCOUNT rab:bitmap:source:{source}:timeSpan:2015-12
255+
```
256+
257+
Example:
258+
259+
```bash
260+
BITCOUNT rab:bitmap:source:referral:timeSpan:2015-12
261+
```
262+
263+
- X week of December: ```BITCOUNT rab:bitmap:source:{source}:timeSpan:2015-12/{X}```
264+
265+
Example:
266+
267+
```bash
268+
BITCOUNT rab:bitmap:source:google:timeSpan:2015-12/1
269+
```
270+
271+
- Trend traffic ({page} is one of: homepage, product1, product2, product3):
272+
273+
- December: from ```BITCOUNT rab:bitmap:action:visit:{page}:timeSpan:2015-12-01``` to ```BITCOUNT rab:bitmap:action:visit:{page}:timeSpan:2015-12-31```
274+
- 1 Week of December: Similar as above, but from 2015-12-01 to 2015-12-07
275+
- 2 Week of December: Similar as above, but from 2015-12-08 to 2015-12-14
276+
- 3 Week of December: Similar as above, but from 2015-12-15 to 2015-12-21
277+
- 4 Week of December: Similar as above, but from 2015-12-22 to 2015-12-28
278+
- 5 Week of December: Similar as above, but from 2015-12-29 to 2015-12-31
279+
- Example:
280+
281+
```bash
282+
BITCOUNT rab:bitmap:action:visit:homepage:timeSpan:2015-12-29 => BITCOUNT rab:bitmap:action:visit:homepage:timeSpan:2015-12-30 => BITCOUNT rab:bitmap:action:visit:homepage:timeSpan:2015-12-31
283+
```
284+
285+
- Total products bought:
286+
287+
- December: ```GET rab:count:action:buy:timeSpan:2015-12```
288+
- X week of December: ```GET rab:count:action:buy:timeSpan:2015-12/{X}```
289+
Example:
290+
291+
```bash
292+
GET rab:count:action:buy:timeSpan:2015-12/1
293+
```
294+
295+
- Total products added to cart:
296+
297+
December: ```GET rab:count:action:addToCart:timeSpan:2015-12```
298+
X week of December: ```GET rab:count:action:addToCart:timeSpan:2015-12/{X}```
299+
Example:
300+
301+
```bash
302+
GET rab:count:action:addToCart:timeSpan:2015-12/1
303+
```
304+
305+
- Shares of products bought ({productPage} is on of product1, product2, product3):
306+
307+
December: ```GET rab:count:action:buy:page:{productPage}:timeSpan:2015-12```
308+
Example:
309+
310+
```bash
311+
GET rab:count:action:buy:page:product3:timeSpan:2015-12
312+
```
313+
314+
- X week of December: ```GET rab:count:action:buy:page:{productPage}:timeSpan:2015-12/{X}```
315+
Example:
316+
317+
```bash
318+
GET rab:count:action:buy:page:product1:timeSpan:2015-12/2
319+
```
320+
321+
#### Customer and Cohort Analysis
322+
323+
- People who registered: BITCOUNT rab:bitmap:action:register:timeSpan:2015-12
324+
- People who register then bought (order matters): BITCOUNT rab:bitmap:custom:cohort-buy:timeSpan:2015-12
325+
- Dropoff: (People who register then bought / People who register) * 100 [%]
326+
- Customers who bought only specified product ({productPage} is one of: product1, product2, product3):
327+
328+
```
329+
SMEMBERS rab:set:action:buy:page:{productPage}:timeSpan:2015-12
330+
```
331+
332+
Example:
333+
334+
```bash
335+
SMEMBERS rab:set:action:buy:page:product2:timeSpan:2015-12
336+
```
337+
338+
- Customers who bought Product1 and Product2:
339+
340+
```
341+
SINTER rab:set:action:buy:page:product1:timeSpan:anytime rab:set:action:buy:page:product2:timeSpan:anytime
342+
```
343+
344+
- Customer Retention (customers who bought on the different dates): SMEMBERS rab:set:custom:retention-buy:timeSpan:anytime
345+
346+
347+
### References
348+
349+
- [Project Source Code](https://github.com/redis-developer/basic-analytics-dashboard-redis-bitmaps-nodejs)
350+
- [Use cases of Bitmaps](https://redis.io/topics/data-types-intro)
351+
- [How to Build a Slack Bot to Retrieve Lost Files Using AWS S3 and RediSearch](/create/aws/slackbot)
352+
- [How to Deploy and Manage Redis Database on AWS Using Terraform](/create/aws/terraform)

0 commit comments

Comments
 (0)