Manual Setup - one-click deployment is WIP

Introduction

Docker-deploy as the name suggests is a docker-based deployment script that can be used to set up the UCI for usage.

Why Docker?

We are using Docker to enable a smooth, single command experience for all the adopters of UCI to get it up and running. Docker is an open-source containerization platform. It enables developers to package applications into containers—standardized executable components combining application source code with the operating system (OS) libraries and dependencies required to run that code in any environment.

Intended Users

Any user/ organization that wants to use setup UCI on their own server.

Pre-requisites for Setting up UCI

Install Docker if not already installed. Click Here
Infra:
- 8 CPU Cores
- 16 GB RAM
- 20 GB Storage
- Server Fixed/Static IP
- Preffered OS - Linux
Basic familiarity required with:
- Terminal & its commands
- Docker
- Server
- Rest apis
Please make sure all of the ports mentioned used in the file are open & are not being used by any other service on server.

Manual Setup - one-click deployment is WIP

Add the desired version of uci-apis which you want to use to .env. Run setup-manual.sh and follow the prompts.
Once all the containers are up, you will notice that there will be an minio-fusionauth setup out of the box. Go to minio console at port 9001, navigate to Identity -> Open ID, click on edit and turn Claim User Info on.
Some of the components are required to be created manually, they will be automated once once-click-deployment is completed
- Hashicorp Vault has to initiated for the first time and relevant keys should be setup.
- A kv v1 engine has to be created with firebase secrets in hashicorp vault.
- Org and ownerIDs to should be created
- Manual adapters are need to be created - https://uci.sunbird.org/use/developer/uci-basics
- Transformer ( broadcast & generic) and ODK has to created
On the first run cassandra migrations are applied. If you see any migration error then set it to 0 to run all migrations if not already applied, else set to 6 if all migrations are applied. In most cases after the first run, setting the migrations count in .env to 6 will work. Migrations get applied from 0 to 6 in a sequential manner.
NOTE:
- You can restart the services by using docker-compose commands
- It is a good practice to modify the volume mounts within the docker-compose file to ensure that all data is mounted on a disk that is entirely separate from the operating system (OS) disk.
- ODK's default password is not picked on runtime, if you are using something other than default password, you have to manually reset the password for administrator user.
- In server deployments, it is necessary to expose the fusionauth service on a specific port. It is recommended to associate a domain name with this service (for example: auth.yourdomain.com). Subsequently, you should update the MINIO_IDENTITY_OPENID_CONFIG_URL configuration setting by replacing fusionauth with either the server's IP address along with the designated port or the domain name you have directed it to. For running it on your local machine, the provided configurations should suffice.

Docker Setup Steps

Cloning and Run

Take clone of this repository.

git clone https://github.com/samagra-comms/docker-deploy.git
Go to folder

cd docker-deploy
Run below command to download & start the services using docker.

bash installv2.sh
- If script is fail on runtime then you should run below commands for continue run this script
  - Stop the all services
    
    docker-compose down
  - Start the all services
    
    docker-compose up -d
While the script is running, You will be asked to enter encryption key, contact the administrator and get this.
If you are asked to enter Netcore Whatsapp Auth Token, Source, URI, enter the details if you have any else press enter. Click here to check the whatsapp configuration for netcore.

Note: Please note this installation is just the first step. If your needs are not fulfilled with the current installation, please start scaling the individual services by using them in docker stack.

After a Successful Docker Setup Completion

Script execution time - Around 1 hours (its depends on your server)
On a successful run of the script you will get below services:
- Inbound Service: http://localhost:9080/health
- Orchestrator Service: http://localhost:8686/health
- Transformer Service: http://localhost:9091/health
- Outbound Service: http://localhost:9090/health
- Broadcast Transformer Service: http://localhost:9093/health
- Campaign Service: http://localhost:9999
- Kafka UI: http://localhost:18080/ui/docker-kafka-server/topic
- Bot DB Hasura UI: http://localhost:15003/console/login
- Fusion Auth: http://localhost:9011
- ODK: http://localhost:8080/Aggregate.html#management/forms
- Chat Frontend: http://localhost:9098
- Admin Console: http://localhost:9097
Note: If above mentioned url's are not working then restart the all docker services or check the docker container logs.
If you want to check logs for a specific service, follow below flow.
- Show all docker containers
  
  docker ps -a
- Copy the container id from the list for the service you want to see the logs for, and use it in below command.
  
  docker --follow --tail 10 container_id

Note: If the services are updated on any server, use server's ip instead of localhost. Eg. http://143.112.x.x:9080

Steps After Docker Setup

Tracking Tables/Views : When the script is executed, tables will be created for bot schema via migrations. These tables can be assessed in Hasura. We need to track these tables to expose these tables to Hasura GraphQL. Click to view tracking overview.
- Go to the url http://localhost:15003/console/data/default/schema/public
- The admin secret can be controlled using this link
- Go to http://localhost:15003/console/data/default/schema/public and track all of the items one by one.

After the tables are created, some default data for adapter, transformers etc should be added. These will be used later while creating a bot. Please follow below step:

In the sidebar click on the SQL button and add the following commands and run.

INSERT INTO service ("id", "type", "config")
VALUES ('94b7c56a-6537-49e3-88e5-4ea548b2f075', 'odk', '{"cadence": { "retries": 0, "timeout": 60, "concurrent": true, "retries-interval": 10 }, "credentials": { "vault": "samagra", "variable": "samagraMainODK" } }');
INSERT INTO adapter ("id", "provider", "channel", "config", "name") 
VALUES ('44a9df72-3d7a-4ece-94c5-98cf26307324', 'gupshup', 'WhatsApp', '{ "2WAY": "2000193033", "phone": "9876543210", "HSM_ID": "2000193031", "credentials": { "vault": "samagra", "variable": "gupshupSamagraProd" } }', 'SamagraProd');
INSERT INTO adapter ("id", "provider", "channel", "config", "name") 
VALUES ('44a9df72-3d7a-4ece-94c5-98cf26307323', 'Netcore', 'WhatsApp', '{ "phone": "912249757677", "credentials": { "vault": "samagra", "variable": "netcoreUAT" } }', 'SamagraNetcoreUAT');
INSERT INTO adapter ("id", "provider", "channel", "config", "name") 
VALUES ('64036edb-e763-44b1-99b8-37b6c7b292c5', 'gupshup', 'sms', '{"credentials": {"vault": "samagra","variable": "gupshupSamagraSMS"}}', 'SamagraGupshupSms');
INSERT INTO adapter ("id", "provider", "channel", "config", "name") 
VALUES ('4e0c568c-7c42-4f88-b1d6-392ad16b8546', 'cdac', 'sms', '{"2WAY":"2000193033","phone":"9876543210","HSM_ID":"2000193031","credentials":{"vault":"samagra","variable":"gupshupSamagraProd"}}', 'SamagraCdacSms');
INSERT INTO adapter ("id", "provider", "channel", "config", "name") 
VALUES ('2a704e82-132e-41f2-9746-83e74550d2ea', 'firebase', 'web', '{ "credentials": { "vault": "samagra", "variable": "uci-firebase-notification" } }', 'SamagraFirebaseWeb');
INSERT INTO transformer ("name", "tags", "config", "id", "service_id") 
VALUES ('SamagraODKAgg', array['ODK'], '{}', 'bbf56981-b8c9-40e9-8067-468c2c753659', '94b7c56a-6537-49e3-88e5-4ea548b2f075');
INSERT INTO transformer ("name", "tags", "config", "id", "service_id") 
VALUES ('SamagraBroadcast', array['broadcast'], '{}', '774cd134-6657-4688-85f6-6338e2323dde', '94b7c56a-6537-49e3-88e5-4ea548b2f075');
 INSERT INTO transformer ("name", "tags", "config", "id", "service_id") 
VALUES ('SamagraGeneric', array['generic'], '{}', '0832ca13-c698-4234-8070-b5f708bc0b1a', '94b7c56a-6537-49e3-88e5-4ea548b2f075');

Now we can start Sent/Receive messages using uci web channel http://localhost:9098/ but first you should create a bot for conversation after that you will send starting message.
You can start using FusionAuth Console using http://localhost:9011/ and create an Account, for managing users and what resources they are authorized to access.
For managing all the assesment data go on URL : http://localhost:15002/ and track all the tables and relation using token.

Gitpod Setup Steps

Introduction

Gitpod continuously builds your git branches like a CI server so that you can start coding right away - no more waiting for dependencies to download and builds to finish.

Gitpod = server-side-dev-envs + dev-env-as-code + prebuilds + IDE + collaboration.

How to Run Script on Gitpod

Click below button for gitpod workspace
After click above button we will show below screen
While the script is running, You will be asked to enter encryption key, contact the administrator and get this.
If you are asked to enter Netcore Whatsapp Auth Token, Source, URI, enter the details if you have any else press enter. Click here to check the whatsapp configuration for netcore.
After configuration, it will take time for cloning and container creation Around 45 Minutes
If you see below screen it means your all services are up

Note: This script may fail during runtime, in this case run the below commands to restart services.
- Stop all the services
  
  docker-compose down
- Start all the services
  
  docker-compose up -d
How to get list of open ports
How to get gitpod url for individual service
- List of running services port. Click the browser icon
- After click, It will open in new tab
  
  Gitpod Sample URL : https://15003-samagracomm-dockerdeplo-93qdi1tvb7x.ws-us47.gitpod.io/console/login

After Setup Completion

On a successful run of the script you will get below services:
- Inbound Service: https://9080-gitpodurl/health
- Orchestrator Service: https://8686-gitpodurl/health
- Transformer Service: https://9091-gitpodurl/health
- Outbound Service: https://9090-gitpodurl/health
- Broadcast Transformer Service: https://9093-gitpodurl/health
- Campaign Service: https://9999-gitpodurl
- Kafka UI: https://18080-gitpodurl/ui/docker-kafka-server/topic
- Bot DB Hasura UI: https://15003-gitpodurl/console/login
- Fusion Auth: https://9011-gitpodurl
- ODK: http://8080-gitpodurl/Aggregate.html#management/forms
- Chat Frontend: https://9098-gitpodurl
- Admin Console: https://9097-gitpodurl
If you want to check logs for a specific service, follow below flow.
- Show all docker containers
  
  docker ps -a
- Copy the container id from the list for the service you want to see the logs for, and use it in below command.
  
  docker --follow --tail 10 container_id

Steps After All Services Running

When all services are up, we will have to follow the same routine we do in docker setup after steps. Click here to view the steps.
Note: we will have to change the url in the steps to that of gitpod ones.

Common Errors and resolution

On Sending the starting message to UCI front , if you do not receive any reply in 1 minute. Follow below steps
- Check the logs of inbound service using below command
  - Check the container id
    
    docker ps -a
  - Check the logs for orchestrator container
    
    docker logs --follow --tail 100 container_id
  - If log shows pushed to inbound-processed topic, then follow the same flow for orchestrator, transformer or outbound
- One of the service must not be receiving data from topic, restart that service using below commands
  - Check the container id
    
    docker ps -a
  - Stop the service
    
    docker stop container_id
  - Start the service
    
    docker-compose up -d service_name
When we run script on gitpod sometimes we are facing below error
- Stop the all services
  
  docker-compose down
- Start the all services
  
  docker-compose up -d

Setting up your first bot

UCI Admin
1. Go to Link
2. Click on Add new button
3. Fill the form with a unique starting message, start date equals to current & end date more than current date.
4. Click on next button
5. Click on Add logic button
6. Fill the form & upload a xml form. Eg. Sample ODK Excel Form
7. Add this & submit this form.
8. The bot will be added and we can start using this on UCI front.

APIs

Convert a ODK Excel form to XML form using Link.

Sample ODK Excel Form

Upload this XML from using this api.

curl --location --request POST 'http://localhost:9999/admin/v1/forms/upload' \
--header 'admin-token: EXnYOvDx4KFqcQkdXqI38MHgFvnJcxMS' \
--form 'form=@"{PATH_OF_ODK_FORM}"'

Sample ODK XML Form

Response: The api will return a form id, use this form id in create conversation logic api. Form id Eg. List-Button-test-v1

{
    "ts": "2022-05-24T13:46:06.640Z",
    "params": {
        "resmsgid": "dc586de0-db67-11ec-ae84-fbd67a9c1174",
        "msgid": null,
        "status": "successful",
        "err": null,
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {
        "data": "List-Button-test-v1"
    }
}

Create a Conversation Logic

curl --location --request POST 'http://localhost:9999/admin/v1/conversationLogic/create' \
--header 'admin-token: EXnYOvDx4KFqcQkdXqI38MHgFvnJcxMS' \
--header 'Content-Type: application/json' \
--data-raw '{
    "data": {
        "name": "UCI demo bot logic",
        "transformers": [
            {
                "id": "bbf56981-b8c9-40e9-8067-468c2c753659",
                "meta": {
                    "form": "https://hosted.my.form.here.com",
                    "formID": "List-Button-test-v1"
                }
            }
        ],
        "adapter": "44a9df72-3d7a-4ece-94c5-98cf26307323"
    }
}'

Response: It will return a conversation logic id, use it in create bot api. Eg. 92f7b965-4118-4ddc-9c7d-0bc0f77092db

{
    "ts": "2022-05-24T13:48:06.407Z",
    "params": {
        "resmsgid": "23b94970-db68-11ec-ae84-fbd67a9c1174",
        "msgid": null,
        "status": "successful",
        "err": null,
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {
        "data": {
            "transformers": "[{"id":"bbf56981-b8c9-40e9-8067-468c2c753659","meta":{"form":"https://hosted.my.form.here.com/%22,/%22formID/%22:/%22List-Button-test-v1/%22%7D%7D]",
            "adapter": "44a9df72-3d7a-4ece-94c5-98cf26307323",
            "name": "UCI demo bot logic",
            "id": "92f7b965-4118-4ddc-9c7d-0bc0f77092db"
        }
    }
}

Create a bot

curl --location --request POST 'http://localhost:9999/admin/v1/bot/create' \
--header 'admin-token: EXnYOvDx4KFqcQkdXqI38MHgFvnJcxMS' \
--header 'Content-Type: application/json' \
--data-raw '{
    "data": {
        "startingMessage": "Hi Test Bot",
        "name": "Test Bot",
        "users": [],
        "logic": [
            "c556dfc8-5dd3-477c-83bb-65d234c4d223" // Get this id from Create a conversation logic api.
        ],
        "status": "enabled",
        "startDate": "2022-05-24",
        "endDate": "2023-05-24"
    }
}'

Response: This api will return a bot id & other bot information. Use the starting message (Eg. Hi Test Bot) from here to start conversation with a bot.

{
    "ts": "2022-05-24T13:49:15.292Z",
    "params": {
        "resmsgid": "4cc874d0-db68-11ec-ae84-fbd67a9c1174",
        "msgid": null,
        "status": "successful",
        "err": null,
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {
        "data": {
            "startingMessage": "Hi Test Bot",
            "name": "Test Bot",
            "users": [],
            "status": "enabled",
            "startDate": "2022-05-24",
            "endDate": "2023-05-24",
            "logicIDs": [
                "92f7b965-4118-4ddc-9c7d-0bc0f77092db"
            ],
            "id": "9f0b1401-44d2-46be-83bd-7cbd5014f899"
        }
    }
}

Note: If you want to use the gupshup adapter, please contact the administrator for credentials.

Start using bot

Using Web Channel

Once the bot is created, we can start using it. Send the starting message added in the Create a bot api.

Example : Hi Test Bot

Using Whatsapp

Once the bot is created, we can start using it. If you have set up gupshup/netcore provider for whatsapp, Send the starting message added in the Create a bot api to the whatsapp number.

Whatsapp Flow

To get the messages from any service provider say netcore/gupshup, contact their support team, and ask them to add your ip with netcore/gupshup adapter url For Netcore: ip:inbound_extrenal_port/netcore/whatsApp (Eg. - 143.112.x.x:9080/netcore/whatsApp) For Gupshup: ip:inbound_external_port/gupshup/whatsApp (Eg. - 143.112.x.x:9080/gupshup/whatsApp)
To sent messages via outbound, we'll use API from service provider (Netcore/Gupshup). For Netcore, Contact their support team to grant below credentials :
```
    NETCORE_WHATSAPP_AUTH_TOKEN = # Authentication Token 

    NETCORE_WHATSAPP_SOURCE = # Source ID for sending messages to Netcore 

    NETCORE_WHATSAPP_URI = # Netcore API Base URL
```
Update these details in .env file and restart inbound & outbound services to reflect the changes.

Contact Administrator:

Please write to the Maintainer - Chakshu (chakshu@samagragovernance.in), and cc - Saket (saket@samagragovernance.in), Sukhpreet (sukhpreet@samagragovernance.in)

TODO

DB for UCI APIs doesn't get auto populated for the default ODK transformers.
Auto Tracking of tables not supported - this can be automated by writing a script to do POST requests like mentioned here.
Adding docker stack commands to scale up.
Add benchmarking.
Add CI to verify setup.

FYI

If asked Would you like to share anonymous usage data about this project with the Angular Team at Google under Google’s Privacy Policy at https://policies.google.com/privacy? For more details and how to change this setting, see http://angular.io/analytics., Press y.
If you change anything in .env file, you will have to stop the services, then restart them.
- Stop all services:
  
  docker-compose -f docker-compose.yml down
- Start all services:
  
  docker-compose -f docker-compose.yml up -d

Name		Name	Last commit message	Last commit date
Latest commit History 129 Commits
.github/ISSUE_TEMPLATE		.github/ISSUE_TEMPLATE
docs		docs
env.dev		env.dev
etc		etc
init-scripts		init-scripts
kickstart		kickstart
media		media
vault		vault
.env		.env
.env-cdac		.env-cdac
.gitignore		.gitignore
.gitpod.yml		.gitpod.yml
cassandra.yaml		cassandra.yaml
docker-compose.yml		docker-compose.yml
install.gitpod.sh		install.gitpod.sh
install.py		install.py
readme.md		readme.md
requirements.txt		requirements.txt

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Introduction

Why Docker?

Intended Users

Pre-requisites for Setting up UCI

Manual Setup - one-click deployment is WIP

Docker Setup Steps

Cloning and Run

After a Successful Docker Setup Completion

Steps After Docker Setup

Gitpod Setup Steps

Introduction

How to Run Script on Gitpod

After Setup Completion

Steps After All Services Running

Common Errors and resolution

Setting up your first bot

Start using bot

Using Web Channel

Using Whatsapp

Whatsapp Flow

Contact Administrator:

TODO

FYI

About

Releases

Packages

Contributors 10

Languages

samagra-comms/docker-deploy

Folders and files

Latest commit

History

Repository files navigation

Introduction

Why Docker?

Intended Users

Pre-requisites for Setting up UCI

Manual Setup - one-click deployment is WIP

Docker Setup Steps

Cloning and Run

After a Successful Docker Setup Completion

Steps After Docker Setup

Gitpod Setup Steps

Introduction

How to Run Script on Gitpod

After Setup Completion

Steps After All Services Running

Common Errors and resolution

Setting up your first bot

Start using bot

Using Web Channel

Using Whatsapp

Whatsapp Flow

Contact Administrator:

TODO

FYI

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Contributors 10

Languages

Packages