Implement an Asynchronous Task Management system using OOP principles

[Mega-JC]

Is your feature request related to a problem? Please describe.
Our current way of running asynchronous tasks that run in the background (e.g. bot reminders) is very limited. For instance, it isn't currently possible to dispatch a discord API event to several task functions at once. We also have no control of them at runtime, which could be very important for scenarios where a task function should end prematurely, either due to bugs or other server-specific reasons.

Describe the solution you'd like
A jobs module that defines a set of base classes which can be subclassed in order to implement specific types of job classes (interval based, event based, etc.) in the server for the bot to run. These classes would produce independent job instances as objects, which run based on specific data given as input. All active jobs would be managed by a job manager object which keeps track of them and can view, modify, pause, stop, restart or kill jobs at runtime. It should also be responsible for dispatching discord API events to the job types that support them. Job objects should also be able to interact with other jobs in different ways, like allowing other jobs to be notified when one job is finished/killed, or when it has produced a specific type of output. There would also need to be a permission system that prevents some tasks from stopping/killing other ones at runtime.

Describe alternatives you've considered
In the end, one could argue that all of this can be implemented using a simple discord.ext.tasks.loop decorator bound to a function and other data structures, but this alone is very inflexible, and misses out on the many possibilities that an OOP-based task system offers.

[Mega-JC]

All progress on this addition are in the async_task_system branch.

[Mega-JC]

I've made some large revisions on the async_task_system branch on the local side, therefore I'll hide and rewrite some of the previous comments.

[Mega-JC]

Implementation

In order to implement this module, we will need these things:

1. A submodule that implements the backbone of this task system using job base classes, from which a set of utility job subclasses can be made for any functionality to be implemented within the bot.

2. Another submodule that implements a class for managing instances of those job classes at runtime, involving their creation, introspection, modification and termination.

3. A small module defining base classes for event objects, which the job manager class can dispatch to all jobs upon request.

4. Another small module that implements wrapper classes for Gateway events received by the bot client using those event object base classes.

[Mega-JC]

1. Job Submodule

This module defines the basis for all jobs. The pattern of inheritance for jobs is as follows:

jobs/base_jobs.py
┣ JobBase    // Base class of all jobs that defines essential job attributes and methods
┃ ┣ IntervalJobBase    // A job that runs at a certain interval
┃ ┗ EventJobBase    // A job that runs in reaction to event objects dispatched by the job manager
┃   ┗ ClientEventJobBase  // A job that runs in reaction to gateway events on Discord received by the bot's client by default, which are wrapped in event objects
┃
┗ JobProxy // A proxy job class that is returned to a caller when jobs are instantiated, that only exposes the required functions other jobs should get access to
...
jobs/utils/__init__.py
┣ ClientEventJobBase  // A subclass of  job that runs in reaction to gateway events on Discord received by the bot's client by default
┣ SingleRunJob    // A job that only runs once before completing
┃ ┗ RegisterDelayedJob    // A job that registers the jobs given to it to the job manager after a certain period of time
┃ 
┗ MethodCallJob // a job that can be used to schedule a method call
...

JobBase is an internal class that simply defines everything needed for all jobs to work.

IntervalJobBase is a base class for interval based jobs, and subclasses can be configured by overloading class attributes to control at which interval they run, how often they run and more. These attributes get passed to the discord.ext.tasks.Loop() instance given to every job object to handle their code. Configuration should also be possible at the instance level while being instantiated.

EventJobBase is a base class for jobs which run in reaction to event objects dispatched by the job manager. A subclass can override the EVENT_TYPES tuple class variable to only contain the event classes whose instances they want to receive.

There are multiple utility jobs based upon these base job classes to ease things like Discord messaging, scheduling a method call, and more.

Various methods inherited from JobBase can be overloaded to achieve extra functionality, like job initialisation, error handling, cleanup, etc. The .data attribute of a job object is a namespace which they can use to store arbitrary data and control their state while running.

Every job object can use its .manager attribute to get access to methods which can be used to e.g. listen for events from Discord or manipulate other jobs using methods for instantiation, initialization, stopping, killing and restarting. This allows for interception of job-to-job manipulations for logging purposes. This is achieved using a proxy object to the main job manager.

JobProxy is a proxy object that limits external access to a job for encapsulation purposes, thereby only exposing the required functions other jobs should get access to.

JobBase subclasses are meant to overload special methods like on_init(), on_start(), on_run(), on_run_error(), on_stop(), etc. They are used to run all of the code of a job. on_start(), on_run(), on_run_error() and on_stop() rely on discord.ext.tasks.Loop() for calling them from inside its task loop. When a Loop instance is created for a job object, Loop.before_loop() receives on_start, Loop.after_loop() receives on_stop, Loop(coro, ...) receives on_run(), and so on.

[Mega-JC]

2. Job Manager Submodule

This module defines the job manager.

manager.py
┣ JobManager   // A class for managing all jobs created at runtime
┗ JobManagerProxy   // A proxy job class that is instantiated with every job object created, that only exposes the required functions jobs should get access to

JobManager is a class that manages all job objects at runtime. When a job object is added to it, that job receives a JobManagerProxy object to the manager in its .manager attribute and can then use that to access methods for registering other jobs, or to wait for a specific event type to be dispatched, or even to dispatch custom event types.

Job objects are unable to run properly if they are not added to a JobManager instance, and will be removed from it when killed, or when they have completed.

[Mega-JC]

3 & 4. Event Classes Submodule

These modules define base classes for event objects, which are used to propagate event occurences to all jobs that are listening for them. They can also store data in attributes to pass on to listening jobs.

events/
┣ base_events.py   // Module for base classes for event objects 
┃  ┣ BaseEvent      // Base class for all event objects
┃  ┗ CustomEvent    // Base class for any custom event object that can be dispatched.
┃
┗ client_events.py   // Module for BaseEvent subclasses that are used for propagating Discord gateway events
      ┗ ClientEvent      // A subclass of BaseEvent used for propagating Discord gateway events
            ┣ OnMessageBase  // Base class for all Discord message-related gateway events
            ┃ ┣ OnMessage    // Class for propagating the event of a message being sent
            ┃ ┣ OnMessageEdit  // Class for propagating the event of a message being edited
            ┃ ...
            ┣ OnRawMessageBase // Base class for all raw Discord message-related gateway events
    ...

BaseEvent is the base class for the entire event class hierarchy.

ClientEvent is a subclass of BaseEvent used for propagating Discord gateway events.

EventJobBase subclasses can specify which events they should recieve at runtime using an overloadable EVENT_TYPES class variable, which holds a tuple containing all the class objects for the events they would like to recieve. This system makes use of the event class hierarchy, and therefore also support subclasses. By default, this tuple only contains BaseEvent in EventJobBase, meaning that any event object that gets dispatched will be registered into the event queue of the EventJobBase instance. In ClientEventJobBase, this tuple holds ClientEvent instead.

[Mega-JC]

Examples

This sample code shows how the main file for job-class-based program is meant to be structured. Here, the Main job class is used as entry point into the code, by being imported and registered into a running job manager. Only one single job instance of the class Main should be registered at runtime.

job_main.py

class GreetingTest(core.ClientEventJobBase, permission_level=JobPermissionLevels.MEDIUM): # MEDIUM is the default
    """
   A job that waits for a user to type a message starting with 'hi', before responding with 'Hi, what's your name?'.
   This job will then wait until it receives another `OnMessage` event, before saying 'Hi, {event_content}'
   """
    EVENT_TYPES = (events.OnMessage,)

    def __init__(self, target_channel: Optional[discord.TextChannel] = None):
        super().__init__() # very important
        self.data.target_channel = target_channel

    async def on_init(self):
        if self.data.target_channel is None:
            self.data.target_channel = common.guild.get_channel(822650791303053342)

    def check_event(self, event: events.OnMessage):   # additionally validate any dispatched events
        return event.message.channel.id == self.data.target_channel.id

    async def on_run(self, event: events.OnMessage):
        if event.message.content.lower().startswith("hi"):
            with self.queue_blocker():    # block the event queue of this job while talking to a user, so that other events are ignored if intended
                await self.data.target_channel.send("Hi, what's your name?")

                author = event.message.author

                check = (
                    lambda x: x.message.author == author
                    and x.message.channel == self.data.target_channel
                    and x.message.content
                )

                name_event = await self.wait_for(self.manager.wait_for_event(    # self.wait_for signals that the job is awaiting something to other jobs and the job manager
                    events.OnMessage, check=check
                ))
                user_name = name_event.message.content

                await self.data.target_channel.send(f"Hi, {user_name}")
            
            
class Main(core.SingleRunJob, permission_level=JobPermissionLevel.HIGHEST): # prevent most jobs from killing the `Main` job
    """The main job class that serves as an entry into a program that uses jobs.
    """
    async def on_run(self):
        await self.manager.create_and_register_job(GreetingTest)

__all__ = [
    "Main",
]

[Mega-JC]

This issue will now be closed, as the code for implementing this functionality has been made bot-agnostic and is awaiting addition to the early snakecore module. A new issue has been created at pygame-community/snakecore#9.