Add task scheduling system to metagov-core

[gigxz]

[updated]

Requirements

Add a scheduling system in metagov-core. Enable Plugin authors to access the scheduler for these 2 purposes:

1. "Listener" plugins that send events to the Driver. When listening to a platform that doesn't have webhooks, plugin authors need to use polling to listen for changes. Metagov-core should expose some way for plugin authors to define a task that is invoked on a schedule.
2. Async Governance Processes that listen for state changes (a vote being cast, a vote being closed, etc).

Possible tools to use

Celery, django-apscheduler

Things to consider:

• Do what degree should the schedule intervals be configurable, and by whom?
  • If the Driver is evaluating the closing condition for a governance process, it needs to know the "latest" state of the process. The Driver should probably have the flexibility to set what the interval is, since it will affect the lag between the state change occurring and the Driver picking it up.
  • Consider that some gov processes last days or weeks. Could the polling interval be a function of the process closing_at date?
• How do we not overload our server? The number of active scheduled tasks might be, at most: N communities * M listener plugins * P governance processes

---

[old]

The current "listener" interface requires that the external platform already has support for web hooks, and that it emits web hooks for all the events that the governance author cares about.

Read about implementing a Listener in a Plugin here: https://docs.metagov.org/en/latest/plugin_tutorial.html#listener

Problem: not all platforms have support for webhooks!

One example is NEAR: as a plugin author, I want to support "listening" to transactions occurring on the community's smart contract, so that the governance Driver can react to events. There is a public NEAR indexer for explorer that exposes a public API for querying transactions (or DataHub API which is similarly a public indexer). There are also services like BlockNative that let you monitor blockchain transactions using webhooks, but none of these support NEAR.

Alternative to webhooks: the alternative to webhooks is some kind of "pull" API where we fetch recent events from an API at regular intervals. This is what PolicyKit does using Celery tasks (with RabbitMQ broker). How can we make it easy for plugin authors to implement something like that? What technology should we use?

[gigxz]

One possibility is to use a celery schedule in Metagov Core, and allow Metagov Plugins to register task functions. Then the Plugin author doesn't have to worry about how the polling works.

  @Registry.plugin
  class MyPlugin(Plugin):

    @Registry.periodic_task
    def poll_events(self):
        """
        Function that gets called every minute/hour/day (however schedule is configured).
        Useful when you can't use a webhook.
        """
        events = requests.get(...) # fetch events from platform API 

        self.handle_event(event) # process event and send to driver

We need to be able to dynamically add and and remove scheduled tasks without restarting Django or Celery, as well as maybe dynamically changing the schedules (hourly/daily/etc) based on the plugin config.

[shaunagm]

I think it would be fantastic for metagov to implement polling - it would make things much easier for plugin authors. I don't really know enough about this topic to make any recommendations, though.

[thelastjosh]

Just to clarify, this meaning of "poll" is "to check", not "to conduct a survey or questionnaire", right?

…

On Fri, Apr 9, 2021 at 9:35 AM Shauna ***@***.***> wrote: I think it would be fantastic for metagov to implement polling - it would make things much easier for plugin authors. I don't really know enough about this topic to make any recommendations, though. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACHA5PFQKANKU7FO5WOAGQLTH37DDANCNFSM42TMT3LA> .

-- http://www.joshuatan.com/research/

[shaunagm]

Yes, checking, not surveying.

[gigxz]

The instructions to Drivers for how to perform async governance processes was made simpler from this change: https://docs.metagov.org/en/latest/driver_tutorial.html#performing-governance-processes

The instructions for Metagov installation was made more complicated, because of the need to set up Celery: https://docs.metagov.org/en/latest/installation.html