pylightning: Add a plugin framework #2161

cdecker · 2018-12-10T18:59:16Z

This framework is very much inspired by flask, in that I wanted to make it as trivial as possible to get started. I had to jump through some hoops but I'm rather happy with the end result. A plugin with an option and a RPC passthrough is as simple as this:

from lightning import Plugin, monkey_patch
plugin = Plugin(autopatch=True)

@plugin.method("hello")
def hello(name, plugin):
    """Say hello"""
    greeting = plugin.get_option('greeting')
    plugin.log('{} {}'.format(greeting, name))
    return s

plugin.add_option('greeting', 'Hello', 'The greeting I should use.')
plugin.run()

It has support for monkey-patching stdout and stderr by default so that if run by lightningd it'll automatically wrap anything written to those streams into log notifications that get displayed nicely.

Depends-On: #2154

ZmnSCPxj · 2018-12-11T08:10:31Z

contrib/pylightning/lightning/plugin.py

+            try:
+                result = {
+                    "jsonrpc": "2.0",
+                    "result": self._dispatch(request),


It looks to me that this will result in "linear" dispatching, i.e. commands are handled one-at-a-time.

Would it be possible to have this handle dispatching in parallel, i.e. if a command stalls processing, another command can be processed?

Or is some Python limitation exists to prevent this, or it becomes complicated?

(maybe there can be a separate decoration for methods that might block)

Good point, hadn't thought about it yet. I wanted to keep away from multi-threading as much as possible, but I'll try to see what I can come up with.

I opened issue #2167 to track progress on this. Would love to hear feedback for the initial idea, which is future-inspired, but since the python futures are tightly bound to multi-threading I'm thinking of implementing my own AsyncResult instead.

This seems sensible.

rustyrussell · 2018-12-13T02:39:25Z

lightningd/plugin.c

@@ -895,6 +895,7 @@ void plugins_init(struct plugins *plugins, const char *dev_plugin_debug)
 	plugins->pending_manifests = 0;
 	uintmap_init(&plugins->pending_requests);

+	setenv("LIGHTNINGD_PLUGIN", "1", 1);


Convenient! We could have also added a commandline option, but this is easier for plugins to grab in most languages.

rustyrussell · 2018-12-13T02:40:26Z

contrib/pylightning/lightning/plugin.py

+    and stdin filedescriptor, so if we use them in some other way
+    (printing, logging, ...) we're breaking our communication
+    channel. This function
+    """


Did the rest of this comment get redirected to JSON? :)

It is a common problem that occurs when monkey patching system internals. The comment

rustyrussell · 2018-12-13T02:46:15Z

contrib/pylightning/lightning/plugin.py

@@ -17,6 +17,7 @@ class Plugin(object):
    def __init__(self, stdout=None, stdin=None, autopatch=True):
        self.methods = {}
        self.options = {}
+        self.option_values = {}


I don't understand the distinction between options and option_values?

options are the names of options that we registered with all their metadata, whereas option_values are the ones that were set by the init call from the plugin. We can merge them if desired of course.

cdecker · 2018-12-14T13:03:22Z

Merged options and option_values as suggested by @rustyrussell and fixed the comm

It might be useful to take special precautions inside a plugin when being run as a plugin (and not as a standalone executable). This env var is just set so plugins can differentiate correctly. I don't unset the variable since it shouldn't have any effect on `lightningd` itself. Signed-off-by: Christian Decker <decker.christian@gmail.com>

It's flask inspired with the Plugin instance and decorators to add methods to the plugin description. Signed-off-by: Christian Decker <decker.christian@gmail.com>

Signed-off-by: Christian Decker <decker.christian@gmail.com>

Should be a lot easier to see what happens :-) Signed-off-by: Christian Decker <decker.christian@gmail.com>

This is just cosmetic, and makes things like tracebacks much easier to read. Signed-off-by: Christian Decker <decker.christian@gmail.com>

@rustyrussell

Suggested-by: Rusty Russell <@rustyrussell> Signed-off-by: Christian Decker <decker.christian@gmail.com>

cdecker · 2018-12-15T12:21:30Z

Interpreting the accept-review as an ACK, rebased and squashed.

ACK 5dbc836

cdecker added the plugin label Dec 10, 2018

cdecker self-assigned this Dec 10, 2018

cdecker requested review from rustyrussell and renepickhardt December 10, 2018 18:59

cdecker force-pushed the plugin-6 branch from b694e5e to 792964c Compare December 10, 2018 19:21

ZmnSCPxj reviewed Dec 11, 2018

View reviewed changes

ZmnSCPxj mentioned this pull request Dec 11, 2018

Implement an asynchronous dispatch to the plugin framework in pylightning #2167

Closed

cdecker force-pushed the plugin-6 branch from 31eb298 to 17c69db Compare December 13, 2018 10:44

rustyrussell approved these changes Dec 14, 2018

View reviewed changes

cdecker added 6 commits December 15, 2018 13:19

pylightning: Added a tiny library for python plugins

1630528

It's flask inspired with the Plugin instance and decorators to add methods to the plugin description. Signed-off-by: Christian Decker <decker.christian@gmail.com>

pylightning: Add option handling to the plugin library

2b7def9

Signed-off-by: Christian Decker <decker.christian@gmail.com>

pylightning: Migrate the test plugin to use the new wrapper

34e8010

Should be a lot easier to see what happens :-) Signed-off-by: Christian Decker <decker.christian@gmail.com>

pylightning: Split log messages on newlines

69d3d60

This is just cosmetic, and makes things like tracebacks much easier to read. Signed-off-by: Christian Decker <decker.christian@gmail.com>

pylightning: Merge option_values into options

5dbc836

Suggested-by: Rusty Russell <@rustyrussell> Signed-off-by: Christian Decker <decker.christian@gmail.com>

cdecker force-pushed the plugin-6 branch from dcbb860 to 5dbc836 Compare December 15, 2018 12:20

cdecker merged commit 42d8e07 into master Dec 15, 2018

rustyrussell deleted the plugin-6 branch August 15, 2022 00:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

pylightning: Add a plugin framework #2161

pylightning: Add a plugin framework #2161

Uh oh!

cdecker commented Dec 10, 2018 •

edited

Loading

Uh oh!

ZmnSCPxj Dec 11, 2018

Uh oh!

cdecker Dec 11, 2018

Uh oh!

cdecker Dec 11, 2018

Uh oh!

ZmnSCPxj Dec 11, 2018

Uh oh!

rustyrussell Dec 13, 2018

Uh oh!

rustyrussell Dec 13, 2018

Uh oh!

ZmnSCPxj Dec 14, 2018

Uh oh!

cdecker Dec 14, 2018

Uh oh!

rustyrussell Dec 13, 2018

Uh oh!

cdecker Dec 14, 2018

Uh oh!

cdecker commented Dec 14, 2018

Uh oh!

cdecker commented Dec 15, 2018

Uh oh!

Uh oh!

pylightning: Add a plugin framework #2161

pylightning: Add a plugin framework #2161

Uh oh!

Conversation

cdecker commented Dec 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cdecker commented Dec 14, 2018

Uh oh!

cdecker commented Dec 15, 2018

Uh oh!

Uh oh!

cdecker commented Dec 10, 2018 •

edited

Loading