Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement an Asynchronous Task system using OOP principles (ported from PygameCommunityBot) #9

Closed
Mega-JC opened this issue Mar 31, 2022 · 8 comments
Assignees
Labels
Module.Jobs Something related to the `jobs/jobutils` modules

Comments

@Mega-JC
Copy link
Member

Mega-JC commented Mar 31, 2022

This is a port of an issue from the PygameCommunityBot project.

@Mega-JC Mega-JC changed the title Implement an Asynchronous Task Management system using OOP principles (ported from [PygameCommunityBot](https://github.com/PygameCommunityDiscord/PygameCommunityBot/issues/371)) Implement an Asynchronous Task Management system using OOP principles (ported from PygameCommunityBot) Mar 31, 2022
@Mega-JC Mega-JC self-assigned this Mar 31, 2022
@Mega-JC
Copy link
Member Author

Mega-JC commented Mar 31, 2022

Is your feature request related to a problem? Please describe.
Our last ways of running asynchronous tasks that run in the background (e.g. bot reminders) were very limited. For example, it wasn't possible to dispatch a discord API event to several task functions at once. We also had no control of them at runtime, which could be very important for scenarios where a task function should end prematurely, either due to bugs or other server-specific reasons.

Describe the solution you'd like
A jobs module that defines a set of base classes which can be subclassed in order to implement specific types of job classes (interval based, event based, etc.) for a discord bot to run. These classes would produce independent job instances as objects, which run based on specific data given as input. All active jobs would be managed by a job manager object which keeps track of them and can view, modify, pause, stop, restart or kill jobs at runtime. It should also be responsible for dispatching discord API events to the job types that support them. Job objects should also be able to interact with other jobs in different ways, like allowing other jobs to be notified when one job is finished/killed, or when it has produced a specific type of output. There would also need to be a permission system that prevents some tasks from stopping/killing other ones at runtime.

Describe alternatives you've considered
In the end, one could argue that all of this can be implemented using a simple discord.ext.tasks.loop decorator bound to a function and other data structures, but this alone is very inflexible, and misses out on the many possibilities that an OOP-based task system offers.

@Mega-JC
Copy link
Member Author

Mega-JC commented Mar 31, 2022

Implementation

In order to implement this module, we will need these things:

  1. A submodule that implements the backbone of this job system using job base classes, from which a set of utility job subclasses can be made for any functionality to be implemented within the bot.

  2. Another submodule that implements a class for managing instances of those job classes at runtime, involving their creation, introspection, modification and termination.

  3. A small module defining base classes for event objects, which the job manager class can dispatch to all jobs upon request. (already implemented)

  4. Another small module that implements wrapper classes for Gateway events received by the bot client using those event object base classes. (already implemented)

  5. Utility classes that make it easier to work with job objects and create functionality with them.

  6. A small standard library of job classes that implement common & frequently used functionality.

@Mega-JC
Copy link
Member Author

Mega-JC commented Mar 31, 2022

1. Job Submodule

This module defines the basis for all jobs. The pattern of inheritance for jobs is as follows:

jobs/
┣ jobs.py
┃  ┗  JobBase    // Base class of all jobs that defines essential job attributes and methods
┃       ┣ IntervalJobBase    // A job that runs at a certain interval
┃       ┗ EventJobBase    // A job that runs in reaction to event objects dispatched by the job manager
┃           ┗ ClientEventJobBase  // A job that runs in reaction to gateway events on Discord received by the bot's client by default, which are wrapped in event objects
┗ proxies.py
    ┗ JobProxy // A proxy job class that is returned to a caller when jobs are instantiated, that only exposes the required functions other jobs should get access to
...
jobs/
    utils/
    ┣ __init__.py
    ┃    ┣ ClientEventJobBase  // A subclass of  job that runs in reaction to gateway events on Discord received by the bot's client by default
    ┃    ┣ SingleRunJob    // A job that only runs once before completing
    ┃    ┃ ┗ RegisterDelayedJob    // A job that registers the jobs given to it to the job manager after a certain period of time
    ┃    ┃ 
    ┃    ┗ MethodCallJob // a job that can be used to schedule a method call
    ┃
    ┗ messaging.py
        ┣ MessageSend // send a message with the specified arguments
        ┣ MessageEdit // edit a message...
         ... 

JobBase is an internal class that simply defines everything needed for all jobs to work.

IntervalJobBase is a base class for interval based jobs, and subclasses can be configured by overloading class attributes to control at which interval they run, how often they run and more. These attributes get passed to the discord.ext.tasks.Loop() instance given to every job object to handle their code. Configuration should also be possible at the instance level while being instantiated.

EventJobBase is a base class for jobs which run in reaction to event objects dispatched by the job manager. A subclass can override the EVENT_TYPES tuple class variable to only contain the event classes whose instances they want to receive.

There are multiple utility jobs based upon these base job classes to ease things like Discord messaging, scheduling a method call, and more.

Various methods inherited from JobBase can be overloaded to achieve extra functionality, like job initialisation, error handling, cleanup, etc. The .data attribute of a job object is a namespace which they can use to store arbitrary data and control their state while running.

Every job object can use its .manager attribute to get access to methods which can be used to e.g. listen for events from Discord or manipulate other jobs using methods for instantiation, initialization, stopping, killing and restarting. This allows for interception of job-to-job manipulations for logging purposes. This is achieved using a proxy object to the main job manager.

JobProxy is a proxy object that limits external access to a job for encapsulation purposes, thereby only exposing the required functions other jobs should get access to.

JobBase subclasses are meant to overload special methods like on_init(), on_start(), on_run(), on_run_error(), on_stop(), etc. They are used to run all of the code of a job. on_start(), on_run(), on_run_error() and on_stop() rely on discord.ext.tasks.Loop() for calling them from inside its task loop. When a Loop instance is created for a job object, Loop.before_loop() receives on_start, Loop.after_loop() receives on_stop, Loop(coro, ...) receives on_run(), and so on.

@Mega-JC
Copy link
Member Author

Mega-JC commented Mar 31, 2022

2. Job Manager Submodule

This module defines the job manager.

jobs/
┗ manager.py
    ┣ JobManager    // A class for managing all jobs created at runtime.
    ┃
    proxies.py
    ┗ JobManagerProxy   // A proxy job class that is instantiated with every job object created, that only exposes the required functions jobs should get access to
...

JobManager is a class that manages all job objects at runtime. When a job object is added to it, that job receives a JobManagerProxy object to the manager in its .manager attribute and can then use that to access methods for registering other jobs, or to wait for a specific event type to be dispatched, or even to dispatch custom event types.

Job objects are unable to run properly if they are not added to a JobManager instance, and will be removed from it when killed, or when they have completed.

Jobs can use their job manager proxy to access permission-restricted functions, which allow them to interact with other jobs in a controlled manner. With the help of their manager proxy, jobs can find, schedule, start, stop, restart, guard (to prevent unintended state modifications from other jobs), unguard, or kill other jobs, assuming that they have been given the permissions to do so. Without the required permissions, these operations will be blocked with a JobPermissionError exception.

An enum class is used to implement and document the different job permission levels that exist. These must be set on a class level prior to runtime and cannot be modified.

Permission Levels

    LOWEST = 1
    """The lowest permission level.
    An Isolated job which has no information about other jobs being executed.
    Permissions:
        - Can manage its own execution at will.
    """

    LOW = 1 << 1
    """A low permission level.

    Permissions:
        - Can manage its own execution at will.
        - Can discover and view all alive jobs, and request data from them.
    """

    MEDIUM = 1 << 2
    """The default permission level, with simple job management permissions.

    Permissions:
        - Can manage its own execution at will.
        - Can discover and view all alive jobs, and request data from them.
        - Can instantiate, register, start and schedule jobs of a lower permission level.
        - Can stop, restart, or kill jobs instantiated by itself or unschedule its scheduled jobs.
        - Can unschedule jobs that don't have an alive job as a scheduler.
    """

    HIGH = 1 << 3
    """An elevated permission level, with additional control over jobs
    of a lower permission level.

    Permissions:
        - Can manage its own execution at will.
        - Can discover and view all alive jobs, and request data from them.
        - Can instantiate, register, start and schedule jobs of a lower permission level.
        - Can stop, restart, or kill jobs instantiated by itself or unschedule its scheduled jobs.
        - Can unschedule jobs that don't have an alive job as a scheduler.
        - Can stop, restart, kill or unschedule any job of a lower permission level.
        - Can guard and unguard jobs of a lower permission level instantiated by itself.
        - Can dispatch custom events to other jobs (`CustomEvent` subclasses).
    """

    HIGHEST = 1 << 4
    """The highest usable permission level, with additional control over jobs
    of a lower permission level. Lower permissions additionally apply to this level.

    Permissions:
        - Can manage its own execution at will.
        - Can discover and view all alive jobs, and request data from them.
        - Can instantiate, register, start and schedule jobs of a lower permission level.
        - Can stop, restart, or kill jobs instantiated by itself or unschedule its scheduled jobs.
        - Can unschedule jobs that don't have an alive job as a scheduler.
        - Can stop, restart, kill or unschedule any job of a lower permission level.
        - Can guard and unguard jobs of a lower permission level instantiated by itself.
        - Can guard and unguard jobs of the same permission level instantiated by itself.
        - Can dispatch custom events to other jobs (`CustomEvent` subclasses).
        - Can dispatch any event to other jobs (`BaseEvent` subclasses).
        - Can instantiate, register, start and schedule jobs of the same permission level.
        - Can stop, restart, kill or unschedule any job of the same permission level.
    """

    SYSTEM = 1 << 5
    """The highest possible permission level reserved for system-level jobs. Cannot be used directly.
    Lower permissions additionally apply to this level.

    Permissions:
        - Can manage its own execution at will.
        - Can discover and view all alive jobs, and request data from them.
        - Can instantiate, register, start and schedule jobs of a lower permission level.
        - Can stop, restart, or kill jobs instantiated by itself or unschedule its scheduled jobs.
        - Can unschedule jobs that don't have an alive job as a scheduler.
        - Can stop, restart, kill or unschedule any job of a lower permission level.
        - Can guard and unguard jobs of a lower permission level instantiated by itself.
        - Can dispatch custom events to other jobs (`CustomEvent` subclasses).
        - Can dispatch any event to other jobs (`BaseEvent` subclasses).
        - Can instantiate, register, start and schedule jobs of the same permission level.
        - Can stop, restart, kill or unschedule any job of the same permission level.
        - Can guard or unguard any job.
    """

@Mega-JC
Copy link
Member Author

Mega-JC commented Mar 31, 2022

Job Scheduling

Job scheduling works by scheduling the instantiation of a specific job class with a timestamp. For this to work, each job class must be given a permanent and unique identifier to recognize it after it was scheduled with. All arguments to pass to the job constructor must additionally be serializable using pickle, it is advised to keep arguments as simple as possible to make arguments more portable. A fixed or unlimited amounts of automatic rescheduling at a specified interval is also possible.

@Mega-JC
Copy link
Member Author

Mega-JC commented Mar 31, 2022

3 & 4. Event Classes Submodule

These modules define base classes for event objects, which are used to propagate event occurences to all jobs that are listening for them. They can also store data in attributes to pass on to listening jobs.

events/
┣ base_events.py   // Module for base classes for event objects 
┃  ┣ BaseEvent      // Base class for all event objects
┃  ┗ CustomEvent    // Base class for any custom event object that can be dispatched.
┃
┗ client_events.py   // Module for BaseEvent subclasses that are used for propagating Discord gateway events
      ┗ ClientEvent      // A subclass of BaseEvent used for propagating Discord gateway events
            ┣ OnMessageBase  // Base class for all Discord message-related gateway events
            ┃ ┣ OnMessage    // Class for propagating the event of a message being sent
            ┃ ┣ OnMessageEdit  // Class for propagating the event of a message being edited
            ┃ ...
            ┣ OnRawMessageBase // Base class for all raw Discord message-related gateway events
    ...

BaseEvent is the base class for the entire event class hierarchy.

ClientEvent is a subclass of BaseEvent used for propagating Discord gateway events.

EventJobBase subclasses can specify which events they should recieve at runtime using an overloadable EVENT_TYPES class variable, which holds a tuple containing all the class objects for the events they would like to recieve. This system makes use of the event class hierarchy, and therefore also support subclasses. By default, this tuple only contains BaseEvent in EventJobBase, meaning that any event object that gets dispatched will be registered into the event queue of the EventJobBase instance. In ClientEventJobBase, this tuple holds ClientEvent instead.

@Mega-JC
Copy link
Member Author

Mega-JC commented Mar 31, 2022

Examples

This sample code shows how the main file for job-class-based program is meant to be structured. Here, the Main job class is used as entry point into the code, by being imported and registered into a running job manager. Only one single job instance of the class Main should be registered at runtime.

job_main.py

class GreetingTest(core.ClientEventJobBase, permission_level=JobPermissionLevels.MEDIUM): # MEDIUM is the default
    """
   A job that waits for a user to type a message starting with 'hi', before responding with 'Hi, what's your name?'.
   This job will then wait until it receives another `OnMessage` event, before saying 'Hi, {event_content}'
   """
    EVENT_TYPES = (events.OnMessage,)

    def __init__(self, target_channel: Optional[discord.TextChannel] = None):
        super().__init__() # very important
        self.data.target_channel = target_channel

    async def on_init(self):
        if self.data.target_channel is None:
            self.data.target_channel = common.guild.get_channel(822650791303053342)

    def check_event(self, event: events.OnMessage):   # additionally validate any dispatched events
        return event.message.channel.id == self.data.target_channel.id

    async def on_run(self, event: events.OnMessage):
        if event.message.content.lower().startswith("hi"):
            with self.queue_blocker():    # block the event queue of this job while talking to a user, so that other events are ignored if intended
                await self.data.target_channel.send("Hi, what's your name?")

                author = event.message.author

                check = (
                    lambda x: x.message.author == author
                    and x.message.channel == self.data.target_channel
                    and x.message.content
                )

                name_event = await self.wait_for(self.manager.wait_for_event(    # self.wait_for signals that the job is awaiting something to other jobs and the job manager
                    events.OnMessage, check=check
                ))
                user_name = name_event.message.content

                await self.data.target_channel.send(f"Hi, {user_name}")
            
            
class Main(core.SingleRunJob, permission_level=JobPermissionLevel.HIGHEST): # prevent most jobs from killing the `Main` job
    """The main job class that serves as an entry into a program that uses jobs.
    """
    async def on_run(self):
        await self.manager.create_and_register_job(GreetingTest)

__all__ = [
    "Main",
]

@Mega-JC Mega-JC changed the title Implement an Asynchronous Task Management system using OOP principles (ported from PygameCommunityBot) Implement an Asynchronous Task system using OOP principles (ported from PygameCommunityBot) Mar 31, 2022
@Mega-JC
Copy link
Member Author

Mega-JC commented Apr 10, 2022

This has now been successfully implemented.

@Mega-JC Mega-JC closed this as completed Apr 10, 2022
@Mega-JC Mega-JC added the Module.Jobs Something related to the `jobs/jobutils` modules label May 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Module.Jobs Something related to the `jobs/jobutils` modules
Projects
None yet
Development

No branches or pull requests

1 participant