Flow

[tucnak]

Introduction

There are multiple missing pieces from the current Telebot implementation.

One of these pieces is a feature, known as the context. I don't believe that it should be included in the framework itself, as the state machine and the logic associated with it—is largely dependant on the implemention, therefore it must be implemented in the user code.

That said, we repeatedly end up implementing the context one way, or another. From the perspective of the code, it's trivial to do so. What is much more complicated is implementing complex multi-step interactions. This requires a large number of states, which adds to unnecessary verbosity and complicates the logic even further. To make things easier, I propose to introduce the concept of the flow.

What is the flow?

The flow is simply a high-level algorithm, a series of actions and branches, which naturally occur in the logic of any sophisticated bot.

Please, consider the following algorithm:

1. /start A greeting message is shown, accompanied with a list of commands and an inline keyboard with key actions.
   • At this point, the state is empty and there is no current interaction.
   • One of the many stateful interactions may be triggered.
2. Either a command, or an inline button—invokes an interaction.
   • Generally, it doesn't matter which one it was, what matters is the flow.
   • The user state is now set to this particular interaction.
3. The bot requires a piece of media.
   • Depending on whether if it's a photo, a video or an audio recording, this step should leave to three further interactions.
   • Important: to remember the path of the flow, i.e. all the previous steps, including the outer interaction.
   • The user might want to go back and choose a different file without cancelling the whole interaction. This can be done by deleting the sent media, or simply invoking a back action.
4. One of the three interactions get invoked, when a series of data is requested (a form.)
   • Each step of the way, data is harvested, validated separetely, and once form is validated as the whole.
   • At any given point, user is ought to be able to either go back in the form, or go back to the previous interaction.
   • Important: to note that the form itself is a distinct iteration with its own state.
5. Once the interaction is complete, the user can be brought back to a certain waiting state, which may or may (step 1) not be completely stateless.
   • Depending on the particular waiting state, a different set of interactions may be available.

In order to implement the aforementioned algorithm, currently you would have to create a state machine of your own, and laboriously spell out each and every state, alongside with the numerous transition rules. Principally speaking, this is trivial, but as the interactions require multiple kinds of media and have many intermittent requirements, the implementation would have to be spread out across different handlers. The code will quickly grow uncontrollably.

Proposal

The approach spelled out below is only but a first impression, much of it is open to discussion. I should fix a few principles to consider, when discussing it: (a) the state machine must not be naively exposed from the point of the flow, (b) the interaction must be functionally described by its steps, not the other way around, (c) interactions are always happening one-on-one between the bot and the user, (d) the flow is controlled via errors, handled by the interactions from the inner to the outer. Keep this in mind.

I will now walk though the key building blocks.

State

State of the interaction is implementation-dependant.

type State struct {
	Data   interface{}
	Parent int // # of path position
}

Path

Path is an interaction queue.

type Path struct {
	History []Interaction
	Global  State
	Local   []State
}

func (*Path) Interaction() Interaction
	{} // current interaction
func (*Path) Global() State
	{} // global state
func (*Path) Local() State
	{} // current interaction's state
func (*Path) Forward(Interaction, State)
	{} // pushes tail forward
func (*Path) Back() (Interaction, State)
	{} // pushes tail back and returns it

This struct maintains the snapshots of all states at any given interaction, so the rollback can be done from any point. The state is each and different for any given

Interaction

This is the logical building block, a state machine interface.

type Interaction interface {
	Type() string        // unique interaction name
	Step() string        // interaction step, if any
	Enter(*Path) error   // if validated, the interaction becomes the driver
	Back(*Path) error
	Push(*Path, *Update) error
	Cancel(*Path) error
}

Interaction implements the state of its own, while the path maintains the global state. Before the first message is pushed, Enter is called to validate the enter condition; if nil is returned, the interaction is added to the tail of the path along with the path state at the time of entrance.

When Back is called, it can either return nil, in which case the tail is simply rolled back to the last occurance of the interaction in the path, or a Rollback, which may specify the exact position and/or override the the state of the interaction at that point.

Interactions must be implemented to be valid Telebot handlers.

Flow

Flow is an interaction that outlines the high-level series of actions (interactions) in the builder fashion.

type stepFn func (*Path, *Message) error

type Flow interface {
	Interaction

	Text(step string, stepFn) Flow
	Photo(step string, stepFn) Flow
	Video(step string, stepFn) Flow
	Poll(step string, stepFn) Flow
	Dice(step string, stepFn) Flow
	// ...

	Then(step string, Interaction) Flow
	Or(step string, Interaction...) Flow

	Check(func(*Path) error) Flow
}

(I'm not sure whether if it should be a struct or interface, but it probably should be an interface.)

Flow can be used to set up a series of data fetches and validations for all the supported types of updates. Essentially, it's the high-level representation of the algorithm, where everything is supposed to come together using custom fetches, Or and Then to compose, Check to validate in-between steps.

Flows implement regular interactions and can be created with Begin(step string).

Rollback

Rollback is an error that can override the current position and the state of the path.

type Rollback struct {
	Err error
	Position int
	State State
}

func (*Rollback) Error() error
	{} // a well-formatted rollback error
func (*Rollback) Override(int, State)
	{} // position and the state of the override

Conclusion

This is it, for now. Hopefully, the API mentioned above is sufficient to build most, if not any of the bot interactions. Take a look at the actual code that is roughly implementing the algorithm outlined in the introduction.

alogorithm := tele.Begin("start").
	Text("get username", func(path *tele.Path, msg *tele.Message) error {
		
	}).
	Or("get photo or video", getPhoto, getVideo).
	Check("validate input", validation).
        Then("complex interaction", &complexInteraction{})

bot.Handle("/start", algorithm)

Please feel free to ask questions, as well as point out features and imperfections.

Cheers,
Ian

[kifirkin]

Please make it happen 🙏

[Menahem-Mendel]

this would be helpful

[0xff00ff]

You can use redis for that. For that functionality, you definitely need persistent storage. What will you do on: reboot, scaling?
You can use redis for example with key as userID and keep there your state.

[makarychev13]

Nice issue. It is looks like scene in telegraf (node.js lib for telegram bots) - telegraf/telegraf#705. This would be helpful

[wildsurfer]

Any progress on this?

[and3rson]

This would be really cool. I really like the implementation of conversations in python-telegram-bot using finite-state machines:

https://github.com/python-telegram-bot/python-telegram-bot/blob/master/examples/conversationbot.py
https://github.com/python-telegram-bot/python-telegram-bot/blob/master/examples/conversationbot2.py
https://github.com/python-telegram-bot/python-telegram-bot/blob/master/examples/conversationbot2.png

[tucnak]

@wildsurfer Hang in there.

@and3rson Yeah you could call these things FSMs, but there isn't much sense in doing so when you're not exactly reasoning about these machines. For all intents and purposes, the state mechanics of a chatbot must be overall correct (e.g. not being able to produce invalid states) as well as fluid (being able to deduce state transitions from natural input) and what you have to see is that this particular set of constraints is much more nuanced than whatever fits the bill of an FSM in your book, probably.

I say this because most of the time, out–of–the–box stock framework implementation for this sort of thing is completely unsound shit.

This is part of the reason, too, why I insisted that flow–like functionality must be implemented in the executable.

I've changed my mind though, but I don't intend to spew suboptimal solutions any time soon.

Hang in there. (Flow will make v3.)

[nii236]

@tucnak Did the concept of flows end up making it into V3?

[tucnak]

@nii236 No, not yet unfortunately it didn't. I'm overloaded with a bunch of stuff like The Stone Cross Foundation of Ukraine and the other FOSS projects on my neck. However, I've said this numerous times, I would happily lead whomever young engineering wanting to prove themselves with something like this.

My approach to Flow had evolved significantly since the time I wrote this issue.

My bad the evolution wasn't reflected here. Basically, instead of straight-up building the logic around something like looplab/fsm I suggest we do a more generic interface, because depending on the style the bot is implemented in, and the extent of i.e. possible back-and-forth logic to it, you might want something different to a plain generic FSM implementation, which has to be on the table one way or another. What if I want to collect metrics? And this is something truly interesting: imagine being able to record spans for your interactions like you do with prometheus. This would be a game-changer, but this would require if not some form of middleware, then a special implementation of the state machine.

This is the API that I currently have in mind:

// package tele
// var StateFunc func(*tele.State) error

// package yourbot
inb4 := func (prompt string) tele.StateFunc {
	return func(state *tele.State) error {
		return state.Bot.Send(prompt)
	}
})
bot.Handle("/hello", tele.Flow(). // sets up Interaction interface-chain
	Text(inb4("Enter username:"), getUsername).
	Text(inb4("Enter password:"), getPassword).
	Photo(inb4("Your avatar photo:"), getAvatar).
	Then(createUser))

The idea is that there's a Flow() chain-constructor for these interactions, and in each step of the way you would have a beforeFn and afterFn both accepting state and returning error, if any. These errors can be used to manipulate flow of the interaction, to.. For example, we may have a LeapError type having a number of steps, if negative— to go back, and if positive— to skip head of the way. So for the state it would have helper functions enabling you to write in the state function something along the lines of return state.Back() or return state.Back(2) to go back whole two steps.

Another open question remains whether if the state itself should be some fixed struct or an interface. I worry making it a full-blown interface would make it a rather big one, and extending it would be problematic. Otherwise, how do you have the end-users implementing serialisation and tracing, which in case of tracing would for the most part be prometheus, but this is not a given— I've seen people implementing tracing manually in-memory with Bloom filters and the like. Hence if serialisation and the other specialities are not part of the state struct/interface, they must be "registered" with the bot instance somehow. But at this point it might as well be of benefit to make it interface, so the end-users would embed and override the default state interface with their custom implementations like they do in labstack/echo.*Context

[BrianMwangi21]

I am really looking forward to this FSM support. I have a bot that needs this as soon as we can get it but I'm sure we will get there. The implementation outlined above is just perfect.

[shindakioku]

I'd like to try to do it, but I have my own idea of how it should be done.
I want to start by saying that I don't like relying on magic, and I believe developers should always maintain control over the program. However, I acknowledge that there are times when magic can be beneficial.

So, let's start from the beginning:
I agree with the author's post about the last idea, and it looks good to me. Let's introduce two new entities: FlowBus and Flow.

FlowBus is simply a handler for Telegram actions. It will receive actions from a user and direct them to the flow or elsewhere.
Flow represents the original idea from the post.

In detail, if we want to utilize the flow, the first step is to create an instance of FlowBus.

flowBus := telebot.NewFlowBus(...)

After that, we need to register all global handlers that we use in our flow process. This step might seem like overhead, but it's necessary to ensure that the flow can appropriately handle various actions.

However, consider a situation where a user needs a handler for any text input. How can they solve this problem if the flow process also requires that handler? It can be challenging to manage.

To address this issue, I've introduced the FlowBus entity. Typically, user code might look like this:

bot.Handle(telebot.OnText, flowBus.Handle)
bot.Handle(telebot.OnMedia, flowBus.Handle)

But in cases where a user needs a custom handler like ours, they can do the following:

bot.Handle(telebot.OnText, flowBus.ProcessUserToFlowOrCustom(func (c telebot.Context) error {
  // Called only if the user hasn't begun the flow.
 
  return nil
}))

Let's mention the Flow. I believe we need to have the following capabilities:

• Success in the flow (when the user successfully completes all the steps).
• Failure in the flow (a custom failure trigger. Sometimes you need to reject the user's flow process; perhaps your server is too busy, and you cannot proceed, but you don't want to make any changes in your code).
• Validation of the user prompt for every step.
• Possibilities for some decorations on steps (for logging, for example). This means we need to call a custom callback after a step has been successfully passed (we don't need another custom callback for the beginning of the process because we already have it).
• So on...

type NonEmptyValidator struct{}
func (NonEmptyValidator) Validate(state State, c *telebot.Context) error { return nil }

type IsEmailValidator struct{}
func (IsEmailValidator) Validate(state State, c *telebot.Context) error { return nil }

func Example() {
	flowBus := Handler{}
	bot.Handle(telebot.OnText, flowBus.Handle)
	
	var loggingMessage string
	loggingDecorator := func(state *State) {
		if state.Machine.Step() == 0 {
			loggingMessage += fmt.Sprintf("[FLOW] [USER: (%d)", state.context.Message().Sender.ID)

			return
		}

		loggingMessage += fmt.Sprintf("[STEP (%d)] [DATA: (%s)]", state.Machine.Step(), state.context.Message().Text)
	}
	
	sendUserMessage := func(message string) func(*State) {
		return func(state *State) {
			state.context.Reply(message)
		}
	}
	
	// Our greeting serves as the initial step
	startStep := func(state *State) {
		state.context.Reply("Hello there!")
	}

	var startDto struct {
		email    string
		password string
	}

	flowBus.Flow("/start").
		Step(BeginStep().Start(startStep).OnSuccess(loggingDecorator)).
		Step(BeginStep().
			Start(sendUserMessage("Enter email:")).
			Validate(NonEmptyValidator{}, IsEmailValidator{}).
			Assign(func(state *State) {
				startDto.email = state.context.Message().Text
			}).
			OnSuccess(loggingDecorator),
		).Step(BeginStep().
		Start(sendUserMessage("Enter password:")).
		// TextAssigner - is already an implemented function that sets a text from a user to the variable
		Assign(TextAssigner(&startDto.password)),
	).Step(BeginStep().
		Start(func(state *State) {
			state.Machine.Back()
			state.Machine.ToStep(2)
			// Global fail
			state.Machine.Fail()
		}),
	).Success(createUser).Fail(failHandler)
}

But please, don't worry about a lot of code. I'm confident that it will typically look something like this:

func Example() {
	flowBus := Handler{}
	bot.Handle(telebot.OnText, flowBus.Handle)
	sendUserMessage := func(message string) func(*State) {
		return func(state *State) {
			state.context.Reply(message)
		}
	}

	var email, password string
	flowBus.Flow("/start").
		Step(BeginStep().Start(sendUserMessage("Enter email:")).Assign(TextAssigner(&email))).
		Step(BeginStep().Start(sendUserMessage("Enter password:")).Assign(TextAssigner(&password))).
		Success(createUser)
}

[shindakioku]

The API above isn't the final idea. I'm going to think about it, and I'm pretty sure I can make it simpler and shorter.

[shindakioku]

I will make a pull request tomorrow. The code may not be production-ready, of course, and without tests, but it should be sufficient for manual testing. Unfortunately, I haven't worked with a Telegram bot api for the last two years, so I may not be able to cover all cases. I'm asking for help from anyone who wants to contribute. Just pull my branch, try to describe your own flows, and provide feedback.

[shindakioku]

#657 Let's discuss that