Clojure and Datalevin framework for making complex Telegram Bots/Applications
- Idea
- Installation
- Usage
- Testing
- Logging
- Configuration
- Deploy
- Further development and bugfixing
- Roadmap
- License
Cloregram has several main ideas:
- Main and temporal messages - there is one main actual message that mostly reacts and interacting with user. For certain cases can be used such a "temporal" messages with button to delete them. Temporal messages appear with notification, when main one is changing smoothly.
- Nearly stateless approach - Cloregram does not save exact user state. Instead, each button points to unique Callback entity with user ref, function symbol and EDN-serialised arguments saved in Datalevin. Only interaction with user input (text/media etc) needs to change user state. This approach allows to describe more robust and predictible bot behaviour.
- Virtual users testing - Cloregram comes with ready-made integration testing framework. It mocks Telegram API server to allow developer describe users behaviour and test bot outcome in very convenient and flexible way.
It is highly recommended to use Java 21+ to benefit from JVM virtual threads.
The simplest way to start project on Cloregram framework is to use Leiningen template:
lein new algoflora/cloregram my-cloregram-bot
To check it use following commands:
cd my-cloregram-bot
lein eftest
I am trying to support actual template version.
Every user interacting with the bot is recorded in the database. User entity has following fields:
Key | Description | Required? | Unique? |
---|---|---|---|
:user/username |
username of user in Telegram | ☑️ (if exists) | |
:user/id |
ID of user similar to chat_id | ☑️ | ☑️ |
:user/first-name |
first name of user in Telegram | ☑️ | |
:user/last-name |
last name of user in Telegram | ||
:user/language-code |
code of user's language in Telegram | ☑️ | |
:user/msg-id |
Id of main message. Mostly for internal usage! | ☑️ | |
:user/handler |
Current handler for Message with args. Mostly for internal usage! | ☑️ |
In most cases current interacting with bot User is binded to *current-user*
dynamic Var. Read further for more details and examples.
Public API functions to interact with user are located in cloregram.api
namespace. For now, support of many features like locations etc is missing. Support of media is very limited. Framework is still in active development.
Example usage:
(cloregram.api/send-message *current-user* (format "*Hello!* Number is %d.\n\nWhat we will do?" n)
[[["+" 'my-cloregram-bot.handlers/increment {:n n}]["-" 'my-cloregram-bot.handlers/decrement {:n n}]]]
:markdown)
user
- the user who will receive the message(format <...>)
- text of the message- vector of vectors of buttons - keyboard as collection of rows of buttons
:markdown
- option to use MarkdownV2 in message.
All available functions and options look in API documentation.
Inline keyboard for mesage is presented as vector of button rows. Each row is a vector of buttons.
Each button can be presented as map. This is good for WebApp buttons, or buttons pointing on URLs.
In most cases we need button that performing certain action when clicked. For such buttons is convenient to use vector notation:
- first element is button text string
- second element is symbol of handler function which will be called on click
- optional third element is arguments map for handler function
One of key points in Cloregram are handler functions.
Handler function takes a map of parameters. Always there will be key :user
containing User map.
All handling logic will have in scope dynamic Var *current-user*
binded to User from whom update was accepted. If current update carring callback query, then dynamic Var *from-message-id*
wouldbe binded to ID of messasge where button with callback query was clicked. This is useful e.g. if you need to delete temporal message after some action from it. To use this vars you have to require cloregram.dynamic
namespace: (require '[cloregram.dynamic :refer :all])
.
If handler function is supposed to handle Message, then it will have Message map in parameters map on key :message
.
The main entry point is my-cloregram-bot.handlers/main
. It will be called on start or on any Message input from user if this behaviour wasn't changed with calling cloregram.users/set-handler
.
Following example of common handler will greet user by first name and repeat his text message:
(ns my-cloregram-bot.handlers
(:require [cloregram.api :as api]
[cloregram.dynamic :refer :all]))
(defn common
[{:keys [user message]}]
(api/send-message user
(format "Hi, %s!\n\nYou have said \"%s\", haven't you?"
(:user/first-name user)
(:text message))
[]))
If handler function is suposed to handle Callback Query (inline keyboard buttons clicks), then in parameter map will be :user
key and other keys if any passed from button.
Look at extended example that will increment or decrement number value depending of button clicked:
(ns my-cloregram-bot.handlers
(:require [cloregram.api :as api]
[cloregram.dynamic :refer :all]))
(defn main
[_]
(api/send-message *current-user* (format "Hello, %s! Initial number is 0." (:user/first-name *current-user*))
[[["+" 'my-cloregram-bot.handlers/increment {:n 0}]["-" 'my-cloregram-bot.handlers/decrement {:n 0}]]]))
(defn increment
[{:keys [n]}]
(let [n (inc n)]
(api/send-message *current-user* (format "Number was incremented: %d" n)
[[["+" 'my-cloregram-bot.handlers/increment {:n n}]["-" 'my-cloregram-bot.handlers/decrement {:n n}]]])))
(defn decrement
[{:keys [n]}]
(let [n (dec n)]
(api/send-message *current-user* (format "Number was decremented: %d" n)
[[["+" 'my-cloregram-bot.handlers/increment {:n n}]["-" 'my-cloregram-bot.handlers/decrement {:n n}]]])))
Note that in this example any input except for button clicks will call common
handler and reset number value to null!
To make user pay for something use API function cloregram.api/send-invoice
. When user succesfully paid, payment handler is called. Payment handler have to be located in my-cloregram-bot.handlers/payment
function. This function take parameters map with keys :user and :payment. Use user data and :invoice_payload
field in payment map to determine further behaviour.
Namespace cloregram.database
provides functions for working with database:
Call | Description | Comment |
---|---|---|
(cloregram.database/conn) |
Returns Datalevin connection | Look at Datalevin Datalog store example for details |
(cloregram.database/db) |
Returns database structure for queries, pulls etc | Look at Datalevin Datalog store example for details |
Also cloregram.db
namespace has two useful functions to keep the schema up to date and to load initial data:
Call | Description | Comment |
---|---|---|
(cloregram.db/update-schema) |
Wriring to database internal Cloregram schema (User and Callback entities) and all entities from project's resources/schema folder. Schema entities data have to be located in .edn files. For conviency good to have different files for each entity. Nested folders are supported. |
Take attention that this function is automatically launching every application startup. So kindly use resources/schema folder only for schema entities, but not for data. Otherwise data will be rewrited every startup even if it was changed by application. One more problem is that for now update-schema not removing entities attributes that are not in schema any more - (Issue #6). |
(cloregram.db/load-data) |
Writing to database data from project's resources/data folder. Data have to be in .edn files. For conviency good to have different files for each entity. Nested folders are supported. |
Take attention that this function is not called anywhere in code except cloregram.test.fixtures/load-initial-data test fixture. So you have to manage manual calling of load-data after project startup at first launch in production. Don't forget remove this call later or all involved data will be always overwritten from scratch. This behaviour expected to be changed in Issue #6. |
For internationalisation can be used built-in cloregram.texts
namespace. At first you have to create folder 'texts' in project's resource path. In that folder you an put any number of .edn files, each representing clojure nested map. These maps will be deeply merged on startup. On the deepest level all values must be a map with language codes as keys and corresponding strings as values.
For example you have file resources/texts/main.edn
:
{:main {:en "Select action:"
:fr "Sélectionnez l'action:"}
:greeting {:en "Hello, %s!"
:fr "Bonjour, %s!"}}
as well as resources/texts/buttons.edn
:
{:buttons {:greet-en {:en "Greet in English"
:fr "Saluer en anglais"}
:greet-fr {:en "Greet in French"
:fr "Saluer en français"}
:back {:en "Back"
:fr "Dos"}}}
Now you can use cloregram.texts/txt
and cloregram.texts/txti
functions. Difference is that txti
function accepts explicit language code as fifst argument, while txt
function uses :language-code
field of *current-user*
. Both functions accept various arguments possibly used in string formatting.
(ns my-cloregram-bot.handlers
(:require [cloregram.api :as api]
[cloregram.dynamic :refer :all]
[cloregram.texts :refer [txt txti]]))
(defn main
[_]
(api/send-message *current-user*
(txt :main)
[[[(txt [:buttons :greet-en]) 'my-cloregram-bot.handlers/greet {:lang :en}]]
[[(txt [:buttons :greet-fr]) 'my-cloregram-bot.handlers/greet {:lang :fr}]]]))
(defn greet
[{:keys [lang]}]
(api/send-message *current-user*
(txti lang :greeting (:user/first-name *current-user*))
[[[(txt [:buttons :back]) 'my-cloregram.handlers/common]]]))
Cloregram has powerful integration testing suite in cloregram.validation
namespace. Main idea is to simulate behaviour of users with virtual ones and check bot output. This approach in the ideal case allows to test all scenarios, involving any number of virtual users.
Framework has useful fixtures to prepare testing environment and load initial data:
(ns my-cloregram-bot.core-test
(:require [clojure.test :refer :all]
[cloregram.validation.fixtures :as fix]))
(use-fixtures :once fix/use-test-environment fix/load-initial-data)
The cloregram.validation.users
namespace is responsible for working with virtual users:
(require '[cloregram.validation.users :as u])
(u/add :user-1) ; creates virtual user with username "user-1" and default language code "en"
(u/add :user-2 "fr") ; creates virtual user with username "user-2" and language code "fr""
(u/main-message :user-1) ; => nil
(u/last-temp-mesage :user-2) ; => nil
(u/count-temp-messages :user-2) ; => 0
The cloregram.validation.client
namespace contains functions for interaction with bot by virtual users:
(require '[cloregram.validation.client :as c])
(c/send-text :user-1 "Hello, bot!") ; sends to bot the text message "Hello, bot!" by virtual user "user-1"
Also in this namespace are functions:
click-btn
to simulate clicking button in incoming messagepay-invoice
to simulate clicking Pay button in incoming invoicesend-photo
to send to bot image from resourcessend-message
to send more generic messages, not for common use cases
The cloregram.test.infrastructure.inspector
namespace contains functions to check contents of incoming messages:
(require '[cloregram.test.infrastructure.inspector :as i])
(def msg ....) ; message structure
(i/check-text msg "Hello from bot!") ; asserts message's text
(i/check-btns msg [["To Main Menu"]] ; asserts keyboard layout
(i/check-document msg "Caption" contents) ; asserts incoming document caption and contents
(i/check-photo msg "Photo caption" resource) ; asserts incoming photo caption and equality of image to certain resource
(i/check-invoice msg expected-invoice-data) ; asserts incoming invoice
Common test workflow can be like following:
(ns my-cloregram-bot.core-test
(:require [clojure.test :refer :all]
[cloregram.validation.fixtures :as fix]
[cloregram.validation.users :as u]
[cloregram.validation.client :as c]
[cloregram.validation.inspector :as i]))
(use-fixtures :once fix/use-test-environment)
(deftest core-test
(testing "Core Test"
(u/add :testuser-1)
(c/send-text :testuser-1 "/start")
(-> (u/main-message :testuser-1)
(i/check-text "Hello, testuser-1! Initial number is 0.")
(i/check-btns [["+" "-"]])
(c/click-btn :testuser-1 "+"))
(-> (u/main-message :testuser-1)
(i/check-text "Number was incremented: 1")
(i/check-btns [["+" "-"]])
(c/click-btn :testuser-1 "+"))
(-> (u/main-message :testuser-1)
(i/check-text "Number was incremented: 2")
(i/check-btns [["+" "-"]])
(c/click-btn :testuser-1 "-"))
(-> (u/main-message :testuser-1)
(i/check-text "Number was decremented: 1"))))
Cloregram is already configurated to use for testing purposes weavejester/eftest Leiningen plugin.
For logging Cloregram using μ/log library.
Out of the box it write all logs to logs/logs.mulog file. Also:
- logs/last.mulog - all logs of last run. Cleans at startup.
- logs/events.mulog - logs of last run events. No traces. Cleans at startup.
- logs/errors.mulog - errors of last run. Cleans on startup.
- logs/publishers-errors.mulog - errors in μ/log publishers. Cleans at startup.
You have to create in resources
folder file config.prod.edn
for production deploy as well as confid.dev.edn
and config.test.edn
if needed. These files will be used with corresponding Leiningen profiles. More configs and profiles can be added manually. To reference use file resouces/config.example.edn
and following table:
Key | Description | Default | Comment |
---|---|---|---|
:bot/token |
Bot's token obtained from BotFather | "0000000000:XXX..." | Default value works in tests |
:bot/ip |
Ip address of Bot application | "0.0.0.0" | |
:bot/port |
Port Bot application to listen | 8443 | |
:bot/https? |
Will bot use HTTPS? | false |
In production this value must be true |
:bot/admins |
List of administrators usernames | Legacy option. Will be revised as part of Issue #5 | |
:bot/api-url |
Telegram API URL if not default is used | nil |
nil means common Telegram Bot API. Different value is used in tests and can be useful if you are deployed local bot API server |
:bot/server -> :options -> :keystore |
Keystore path for SSL | "./ssl/keystore.jks" | Look Obtaining certificates for details |
:bot/server -> :options -> :keystore-password |
Password for SSL keystore | "cloregram.keystorepass" | Look Obtaining certificates for details |
:bot/instance -> :certificate |
Path to PEM certificate | "./ssl/cert.pem" | Look Obtaining certificates for details |
:db/connection -> :uri |
Datalevin database URI | In tests /tmp/cloregram-datalevin is used. |
|
:db/connection -> :clear? |
Clear the database on startup? | false |
In tests value is true to clear temporal database for each run. |
:project/config |
Map of project specific config | {} |
Values from this map could be accessed with hekp of function (cloregram.system.state/config :key :nested-key ...) |
If you (my-username
) want to deploy the bot (my-cloregram-bot
) in home directory on Ubuntu server on address 127.1.2.3
, use following instructions:
On your server install required JAVA package:
sudo apt update && sudo apt install -y default-jre-headless
Then create there folder structure for project:
cd ~
mkdir my-cloregram-bot
cd my-cloregram-bot
mkdir logs
On local machine (or in CI script) in project sources root folder run (pay attention to your current project version):
lein uberjar
rsync target/uberjar/my-cloregram-bot-0.1.0-standalone.jar my-username@127.1.2.3:/home/my-username/my-cloregram-bot/my-cloregram-bot-standalone.jar
On server in project folder run the bot:
java -jar my-cloregram-bot-standalone.jar
You can use cli option -XX:-OmitStackTraceInFastThrow
for better errors display, but this option may slightly reduce performance.
Or you can start the bot as systemd service
Use editor to create service file /etc/systemd/system/my-cloregram-bot.service
:
[Unit]
Description=My Cloregram Bot Service
After=network.target
[Service]
User=my-username
ExecStart=java -XX:-OmitStackTraceInFastThrow -jar /home/my-username/my-cloregram-bot/my-cloregram-bot-standalone.jar
WorkingDirectory=/home/my-username/my-cloregram-bot
Restart=always
[Install]
WantedBy=multi-user.target
Then start service, enable it on startup and check health:
sudo systemctl start my-cloregram-bot.service
sudo systemctl enable my-cloregram-bot.service
sudo systemctl status my-cloregram-bot.service
In cause of updated .jar file restart service:
sudo systemctl restart my-cloregram-bot.service
To ensure the correct operation of bot webhooks, you must prepare your SSL certificates::
- In deploy folder create ssl folder:
mkdir ssl
- Jump inside it:
cd ssl
- Create certificate and private key. Fill in Country, State/Province, Locality and Organisation as you see fit. CN field is for IP-address or domain where bot is deploying.
openssl req -newkey rsa:2048 -sha256 -nodes -keyout private.key -x509 -days 365 -out cert.pem -subj "/C=LK/ST=Southern Province/L=Kathaluwa/O=Algoflora/CN=127.1.2.3"
- Obtain PKCS12 certificate from our keys:
openssl pkcs12 -export -in cert.pem -inkey private.key -out certificate.p12 -name "certificate"
It will ask you to come up with a password. It will be needed in next step - Create JKS keystore:
keytool -importkeystore -srckeystore certificate.p12 -srcstoretype pkcs12 -destkeystore keystore.jks
At first you will be asked about new keystore password. By default Cloregram using cloregram.keystorepass. Next step you have to enter password from previous step - Now you can delete certificate.p12 and private.key:
rm certificate.p12 private.key
Files cert.pem and keystore.jks as well as ssl folder can be different. Just then you need to specify in the config fields :bot/server :options :keystore (default to ssl/keystore.jks) and :bot/instance :certificate (default to ssl/cert.pem). As well you can set keystore password different from cloregram.keystorepass by specifying field :bot/server :options :keystore-password.
The process of developing the framework depends on the needs of specific projects and/or the author's inspiration. Take a look on Issues page and feel free to suggest something there.
- write spec and unit tests
- add opportunity to select database in config (Datalevin/Datomic)
- enhance and expand media support
- add flexible logging configuration
- enhance payment support
Copyright © 2023-2024
This program and the accompanying materials are made available under the terms of the Eclipse Public License 2.0 which is available at http://www.eclipse.org/legal/epl-2.0.
This Source Code may also be made available under the following Secondary Licenses when the conditions for such availability set forth in the Eclipse Public License, v. 2.0 are satisfied: GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or (at your option) any later version, with the GNU Classpath Exception which is available at https://www.gnu.org/software/classpath/license.html.