-
Notifications
You must be signed in to change notification settings - Fork 768
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Autobahn shared core API #472
Comments
Here is my current favorite sketch: from twisted.internet import reactor
from autobahn.twisted.wamp import Connection
connection = Connection('ws://localhost:8080/ws', u'realm1')
@inlineCallbacks
def on_open(session):
print('session connected: {}'.format(session))
connection.close()
connection.on('open', on_open)
connection.open()
reactor.run() This would establish a WAMP-over-WebSocket connection and attaching a session to a realm. When the session has joined, it prints and then closes the connection.
|
Here is a bigger example with multiple transports, automatic reconnection and authentication: from twisted.internet import reactor
from twisted.internet.defer import inlineCallbacks
from autobahn.wamp.auth import compute_wcs
from autobahn.twisted.wamp import Connection
# list of transports to try
transports = [
{
'type': 'rawsocket',
'endpoint': {
'type': 'unix',
'path': '/tmp/cb-raw',
}
},
{
'type': 'websocket',
'endpoint': {
'type': 'tcp',
'host': 'localhost',
'port': 8080
},
'url': 'ws://localhost:8080/ws'
}
]
# list of credentials to use for authentication
credentials = [{
'type': 'wampcra',
'id': 'user1',
'on_challenge': lambda challenge: return compute_wcs('password1', challenge)
}]
# options that control the connection's behavior
options = {
# http://autobahn.ws/js/reference.html#connection-options
'max_retries': 5
}
# create a connection object with all the bells we defined before
connection = Connection(transports, u'realm1', credentials=credentials,
extra=None, options=options, reactor=reactor)
# when the connection has established a transport, created an ISession object
# and the latter has joined a realm, this will fire
@inlineCallbacks
def on_open(session):
print('session connected: {}'.format(session))
# app code now just uses the session object for WAMP
try:
res = yield session.call("com.example.add2", 2, 3)
print('got result: {}'.format(result))
except Exception as e:
print('something bad: {}'.format(e))
connection.close()
# when the connection closes or goes through a reconnect cycle, this event fires
def on_close(reason):
print('session closed: {}'.format(reason))
# attach our event handlers to the connection object
connection.on('open', on_open)
connection.on('close', on_close)
# now actually open the connection
connection.open()
# enter event loop
reactor.run() |
Another sketch, with the listener-API on the ApplicationSession instances itself: from twisted.internet.task import react
from twisted.internet.defer import inlineCallbacks, Deferred
from twisted.internet.endpoints import clientFromString
def one_off_function(*args, **kw):
print("one_off_function", args, kw)
@inlineCallbacks
def register_api(session):
yield session.register(one_off_function, u"com.example.api.v0.one_off_function")
# ...
@inlineCallbacks
def register_management_api(session):
yield session.register(lambda: None, u"com.example.private.do_nothing")
# ...
def main(reactor):
session = ApplicationSession()
# or "session = MyComponent()" if you prefer subclassing to composition
connection = Connection(session, [
{
"type": "rawsocket",
"endpoint": clientFromString("tor:timaq4ygg2iegci7.onion:9443"),
}
])
session.on.open(register_api) # or session.on('open', register_api)
if True:
session.on.open(register_management_api)
session.on.ready(lambda session: session.publish(u"com.example.api.v0.ready"))
reactor.callLater(10, connection.close)
# .open will errback if connect (or re-connect) fails; otherwise callback w/ None on clean disconnect
return connection.open()
if __name__ == '__main__':
react(main) |
Regarding what events we want to expose (independent of where we expose those events) - I dug out our Slack conversation @meejah
|
Further to @oberstet's last comment above, "any async stuff" includes async-returns from previous handlers. For example, if there are 3 'join' listeners that return Deferreds, "ready" shouldn't fire until those 3 Deferreds are done. |
session = ApplicationSession()
# or "session = MyComponent()" if you prefer subclassing to composition Have been looking at these 2 lines for minutes .. and I know what slightly irritates me: The word "session" - for me - implies some ephemeral thing that is created for me, and which I then use. In above I create the session, and it's not ephemeral. It exists without any transport existing yet, and without having joined a realm. The funny thing is your commented line: Note the mix of "session" and "component": #383 In my perception, the word "component" is much more in line with the user creating the thing ("component like") rather than it being created for the user ("session like"). |
Another aspect. At the WAMP protocol level, there are indeed "session" and "transport". A session rides on top of a transport. And a transport can outlive a session: session 1 ends, and without closing the underlying transport, a new session 2 can start. The new session will get a new WAMP session ID though! However, when a transport goes away, the session is gone too. A WAMP session (today) cannot outlive a transport (but: crossbario/crossbar#300). If we had a user precreate session = ApplicationSession() the lifetime of that Python object hasn't a direct correspondence with "session" in the WAMP protocol sense. I think this discrepancy might be confusing. |
Component vs Session: yeah, I can see the confusion; was trying to copy the examples usage. Perhaps there's no value in even "letting" people choose between composition and inheritance? (As in: no inheritance allowed in new API?). I think there is value in leaving that ability in (even if the methods are re-named to PEP8 like Anyway, for the composition-only case, the session could just be created + owned by There's also an interesting edge-case to consider: what if an event has already happened, and a new listener is subscribed? Should it get immediately invoked (e.g. like when you .addCallbacks to an already-completed Deferred)? This is probably most-relevant for |
Lifecycle-wise: in some ways it does seem "more pure" to have the Also, if there is a All that said, the existing WAMP state-machine does have a "not [yet] connected" state and so from that perspective having a single WAMP "session object" that can join/leave and then join again isn't so outlandish... |
@meejah one thing we need to keep in mind: we need an approach which allows users to take unmodified code to run hosted under CB. That means, the user code needs to be wrapped inside a class (as today with |
Hmm. Yeah to host under CB right now with 'class' things you'd need an ApplicationSession subclass I guess. You could run the other ("listener"?)-style code as a guest worker right now. I suppose the equivalent we'd need in CB is a "function" (router + container) worker which imports a particular function that takes a session (and then adds the listeners it needs etc). So you'd have to write your code with that in mind I guess ... |
Whatever we come up with recommended 80% API in AB must work without changing any code when the component is taken to run under CB. If that can only be achieved via subclassing (the current ApplicationSession approach), then we'll stick to that. |
Yeah, that's a good point; have to think how best that fits in with a "composition"-style thing. Another (but maybe more complicated-sounding) option would be that CB provides configuration so you can specify the listeners directly, something like: {
"type": "function",
"on_join": ["package.register_api", "package.register_management_api"],
"on_leave": ["package.sub.on_leave"],
} |
That last idea points to bigger issues with a non-inheritance based approach: having functions like above in a module means they are forced to hold state in a module global var (or misuse the provided generic WAMP session/connection object to attach app state). Sidenote: me, coming from C++, this is a total no go. In Py, it seems an accepted approach (Django, Flask, ...). It's kind of replicating the mistakes of GIL and the failure to correctly wrap all interpreter state (like all sane, embeddable VMs do .. Lua, V8, ..) in a non-global at the app level. I don't want to encourage that. Now, the problem with having state in globals in above situation: what I run multiple instances of my component in a single worker (router/container)? Doesn't work. Then: having it at that level of detail in the CB config means: essentially mixing CB node config with app internals. A CB admin would not and should know about this in detail. |
Yeah, you would have to attach any state to the provided Putting any kind of state -- even "singleton"-pattern stuff -- in a module isn't very good (even if it is used occasionally). However, this isn't very different from the "override" case, where essentially you have to initialize all your state in So the recommendation I think doesn't change: you initialize any state you need into the session object, and you should do this in |
...but yes in a "non-dynamic" language (C++) it's not immediately obvious what a good solution is for things needing state (aka "instance variables") since they'd need to be declared (i.e. sub-classing). |
slightly related: CB deliberately does NOT provide the |
Maybe I'm using the wrong term, I didn't mean a "shared state" instance variable, I just meant that you can't magically create a thing in C++ via e.g. |
Perhaps a way to reconcile "state" between dynamic (JS, Python) and not (C, C++) without resorting to subclassing is to add a "state" attribute to the In C this can be a Encouraging this explicit encapsulation of session-state might help too with session freeze/thaw: we could say "if the 'state' object implements ISerializable [or whatever] then your session can be frozen/unfrozen". |
Some notes I got in mind: Let's be careful to keep congruent signaturesE.G, currently in Python you got:
But in JS you got:
An no error if you mix them up. I have several painful memories because of it. We should have sane defaultWhen you connect to redis on Python, you can do:
You don't need to do:
It relies on sane officially documented default. For autobahn and crossbar, it means it we should be able to do:
And choose some default, make it very clear in the doc and stick to it. Especially, we need to define an official default port (not 8080, which is often used by dev server) and an official default realm. "realm1" is not a really good name. "public" would be better, since all tutorials use it like some kind of chmod 777 anyway, and it's easier to remember. It will also make OS packager's life easier. Having a default port is good for them. Some stuff are better if not shared between languagesWe should definitely have a common API. But since some constructs are idiomatic in some languages, it is important to provide them as well. For Python, it's decorators:
Is a nice API. But since you are already using a decorator, this will be handy:
The good news is that we don't have to choose between them, we can do both:
For JS, it's promises:
This will make JS coders more "at home". You still should totally have the ordinary common syntax. But have these as well. Distinguish connection failing and closingCurrently in the JS API it's the same event, and you check a status in a variable to know what's really happening. A wrapper checking the status an trigerring explicitly named event would make things more obvious. Open should start the event loopIn JS the event loop is implicit. To give the same feeling of easyness, connection.open() should do reactor.fun() (or the asyncio equivalent), but have a "start_event_loop" parameter with a True default value, in case somebody wants more control. Indeed, we are aiming to facilitate the beginer use case, who will want to start as quickly and easily as possible. Advanced used are the one which should need additional work. I know that the strength of Twisted is that you can use many different modules and plug them together in the same reactor, but this will not be the objective of people trying WAMP. They will not think "let's write a custom protocol and plug in an IRC bridge with my websocket website" on the first day. Nor on the second. For them, the event loop we are using is an implementation detail. Exposing it should be explicitly requested with code, such as a "start_event_loop" parameter. Prevent importing stuff you need all the time and abstract some moreIf you use some things really often, they should be aliased in a namespace that is always imported and be abstracted away to hide differences between event loop. Stuff line @inlineCallbacks, @coroutine or returnValue should have wrappers, and the wrappers should be attached either to session (or an object containing session passed instead of session) or connection. Importing them all the time and having to know about the difference between the event loops is only useful for advanced usages. You can create decent websites and app without even needing to understand perfectly the event loop, just that you should not block and where to setup your "yield". The idea remains :
Check if name == "main" implicitlyLibs like begins started to do this with decorator like @begin.start. Instead of calling connection.open(), you have some method that trigger parameters parsing and command running only if the module is not imported.
But the more you do that, the more putting this code on a Connection object seems a bit weird. That's why last year I suggested the App centered Final APIIf you do all this, the hello world becomes:
That's easy. That's clean. That calls for a copy / paste to try it right away ! But I'd still advice to use something similar to the app API, plus this allow to override call and make it easy to be embeded in crossbar. |
Ok, here are a couple of ones where I do have a strong opinion (note: I will continue with all your @sametmax points tomorrow .. the other I need to think more, and now I need some sleep):
No. We can't do that, as e.g. in Crossbar containers, the event loop is started by the native worker. This is the reason why there is
This is correct for browsers and NodeJS runtimes .. which granted are the only relevant ones. And while Twisted does not allow to run multiple event loops (like the JS run-times mentioned before), asyncio does indeed. That's advanced, but consequent. The fact that Twisted requires explicit reactor start, but then does not allow to start multiple reactors is of course the worst of all I agree. Well. The likeliness that gets fixed is still higher than Python getting rid of the GIL;)
In general, no. IMO too much magic hurts. And auto-importing and aliasing in particular is a great source of hard to track down problems.
You simply can't write source code that runs unmodified across Py2/3 and tx/aio using co-routine style (the mechanisms under the hood for co-routine style are different). At least I don't know a way, and I haven't seen anything like it. You can write such code only without using co-routine style, using plain deferred/future, and without deferred chaining. This is what AB does internally, and it does that using txaio. Hence, there is no point in trying to alias away @inlineCallbacks and @coroutine.
No, argument parsing isn't something a WAMP component should be concerned with. It is the startup code that should do that. Plus: when the component is run under CB, there is no point in parsing any command line. If any, a component's custom config is provided to the component by CB. |
Another point regarding |
I think we should define the objectives of the API first, because all the proposal seems to tackle different objectives.
There are no wrong answers to these, only clear statements to make. |
Defining objectives first .. absolutely. Makes sense. Here is my (current) view:
|
I am posting example code resulting from a discussion with @meejah on Slack. This is how a complete working example would look like under that approach: from twisted.internet import reactor
from autobahn.twisted.wamp import Connection
def main(connection):
@inlineCallbacks
def on_join(session):
res = yield session.call(u'com.example.add2', 2, 3)
print(res)
connection.on('join', on_join)
if __name__ == '__main__':
connection = Connection(main)
connection.connect(reactor)
reactor.run() We've been talking about the design space:
What we have today is subclassing ( Above is doing composition (it just uses a Above assumes defaults like @sametmax suggested: WAMP-over-WebSocket connecting to Above user code (the stuff inside "components": [
{
"type": "function",
"entrypoint": "myapp.main",
"realm": "realm1",
"transports": [
{
"type": "websocket",
"endpoint": {
"type": "tcp",
"host": "127.0.0.1",
"port": 8080
},
"url": "ws://127.0.0.1:8080/ws"
}
]
}
] I am cautiously optimistic that above would satisfy most of the goals we have. But this needs to be investigated further. |
Here is a slight update of the API proposal: from autobahn.twisted.wamp import Connection
def on_join(session):
print('session connected: {}'.format(session))
session.leave()
def main(connection):
connection.on_join(on_join)
if __init__ == __main__:
connection = Connection()
connection.run(main) Slightly larger example which would run unmodified over Py2/3 and twisted/asyncio apart from changing the imports: from __future__ import print_function
from txaio.twisted import add_callbacks
from autobahn.twisted.wamp import Connection
def on_join(session):
result = session.call(u'com.example.myapp.add2', 2, 3)
add_callbacks(result, print, print)
def main(connection):
connection.on_join(on_join)
if __init__ == __main__:
connection = Connection()
finish = connection.run(main)
add_callbacks(finish, print, print) In above, I am gonna prototype that now and see how it works. |
Regarding prerequisite know-how .. there are these areas of knowledge a programmer can have:
E.g. every JS programmer knows about 2., and 3. is growing in the JS community. ES6 brings native promises to JS. Co-routines seem to be in the cooking for ES7 (see here). With Python, both Twisted and asyncio propagate not 3. and 4. It is true that a considerable fraction of Python programmers might only know about 1. - but there isn't anything we can do about in AutobahnPython about it. |
It's clean and easier than before, but we need to figure out how to allow a clean deorator syntax version, since right now you end up with:
Let's do this API, but add in the mix an additional object : a registry to collect all callbacks and which will automatically create something like "main()" for you:
Which will allow:
But even then, I'm a bit confused with "run()". We call it open() on JS, so maybe we should call it the same way in Python. And if not, it means they are different and we need to provide JS with run(). |
There isn't really the equivalent of Personally, I wouldn't put from autobahn.twisted import wamp
import example
api = wamp.Connection(example.api.setup) # on some "public" realm, for example
management = wamp.Connection(example.backend.setup) # on some private realm, for example
wamp.run([api, management], log_options=dict(level='debug')) ...and a module from twisted.internet.defer import inlineCallbacks
from autobahn.twisted import wamp
class _HelloWorld(object):
def __init__(self, db_connection):
self._db = db_connection
@wamp.register(u"com.exampe.api.v0.method_one")
def method_one(self, arg)
return "Hello, {}".format(arg)
@wamp.register(u"com.example.api.v0.another_one")
@inlineCallbacks
def another_one(self):
user = yield self._db.query(...)
other_info = yield self._db.query(...)
return u"You get the idea."
@inlineCallbacks
def _on_join(db, session):
api = _HelloWorld(db)
yield session.register(api, options=RegisterOptions(...))
yield session.publish(u"com.example.api.v0.ready")
@inlineCallbacks
def setup(connection):
db = yield connect_to_db(...)
connection.on('join', _on_join, db)
connection.on('leave', db.disconnect) Obviously, you could have several different |
from autobahn.twisted.wamp import Connection, CallbackList
connection, cb = Connection(), CallbackList()
@cb.register("foo.bar") # not defined in our previous example
def _():
return "foobar"
@cb.subscribe("doh") # not defined in our previous example
def _():
print('doh')
connection.run(cb.run_all) This code can't run unmodified when taken to CB. And it looks very weird to me, and it does too much magic. But even without taking decorators into consideration, we need to address below ..
Yep.
Good point. It's crap to mix these. Back to drawing table: from autobahn.twisted.wamp import Connection
def on_join(session):
print('session connected: {}'.format(session))
session.leave()
def main(connection):
connection.on_join(on_join)
if __init__ == __main__:
from twisted.internet import reactor
connection = Connection()
connection.open(main)
reactor.run()
Hiding @sametmax rgd |
Maybe we can support both composition and inheritance: class Connection(object):
def __init__(self, session_class=ApplicationSession, reactor=None):
pass
def open(self, main=None):
pass |
I like the concept of "run" as a simple method (no matter if it's actually called run) as to me it's a lot more intuitive that it just never returns. Maybe I just like it because it mimics |
@oberstet what about adding a "setup=" kwarg to Connection's init -- and it (if available) would be called with the connection object at some point after we've got a session instance and started the reactor, but before we've joined. I'm guessing that your class FunStuff(object):
def __init__(self, db_connection):
self._db = db_connection
@wamp.register(u"com.example.meaning")
def meaning(self):
return 42
@inlineCallbacks
def connection_setup(connection):
db = yield connect_to_db()
api_provider = FunStuff(db)
@inlineCallbacks
def joined(session):
yield session.register(api_provider, options=RegisterOptions())
connection.on('join', joined)
con0 = Connection(setup=connection_setup)
con1 = Connection(...)
run([con0, con1]) Now, if we can get away with having a "joined" and "left" Deferred/Future as attribute-accessible things on class FunStuff(object):
@wamp.register(u"com.example.meaning")
def meaning(self):
return 42
@inlineCallbacks
def main(session):
db = yield connect_to_db()
api_provider = FunStuff(db)
# ^ that's pre-session-join()ed setup stuff (no register, publish, etc)
yield session.joined
# now we have a connected session, so we can call register etc.
yield session.register(api_provider, options=RegisterOptions())
yield session.publish(u"com.example.ready")
# now cleanup handlers can run after we've left:
yield session.left
yield db.disconnect()
con0 = Connection(main=main)
con1 = Connection(...)
run([con0, con1]) That means: |
I have added a couple of examples of how user code could look like under an API as discussed above: |
Here is the one I am working on master: from twisted.internet.task import react
from twisted.internet.defer import inlineCallbacks as coroutine
from autobahn.twisted.connection import Connection
def main(reactor, connection):
@coroutine
def on_join(session, details):
print("on_join: {}".format(details))
def add2(a, b):
return a + b
yield session.register(add2, u'com.example.add2')
try:
res = yield session.call(u'com.example.add2', 2, 3)
print("result: {}".format(res))
except Exception as e:
print("error: {}".format(e))
finally:
session.leave()
connection.on('join', on_join)
if __name__ == '__main__':
connection = Connection()
react(connection.start, [main]) |
The fun thing is: you can also use inheritance: from twisted.internet.task import react
from twisted.internet.defer import inlineCallbacks as coroutine
from autobahn.twisted.wamp import Session
from autobahn.twisted.connection import Connection
class MySession(Session):
def on_join(self, details):
print("on_join: {}".format(details))
if __name__ == '__main__':
connection = Connection()
connection.session = MySession
react(connection.start) |
and here is yet another try ... how do you like that one? SUBSCRIBE:
transport = await client.connect(u'wss://example.com')
session = await transport.join(u'example1')
subscription = await session.subscribe(u'com.example.on_hello')
def on_hello(msg, seq, opt=None):
print('Received {} - {} - {}'.format(msg, seq, opt))
subscription.on(on_hello)
await sleep(60)
await subscription.unsubscribe()
await session.leave()
await transport.disconnect() |
PUBLISH:
transport = await client.connect(u'wss://example.com')
session = await transport.join(u'example1')
seq = 0
while seq < 60:
session.publish(u'com.example.on_hello', u'Hello, world!', seq, opt=[1, 2, 3])
await sleep(1)
await session.leave()
await transport.disconnect() |
I like the general flow -- and I think a "one-time connect" method is a good idea (vs. e.g. the more "run"/"start" style ones which do re-connection too). (That is, both should be possible) I'm not sure about the "sub = await subscribe(topic)" followed by the ".on()" thing -- what's the use-case? (i.e. when would you not do the .on() right away). What about allowing decorator-access? (ideally "in addition to" ...). Like so:
|
Further to the decorator idea, via the existing
(Similar for the other events you can subscribe to on |
I think the discussion is heading towards consensus (inheritance = bad, observers = good, ..). One thing though is to define/document the events that can be observed on a session. Eg., currently we have these https://github.com/crossbario/autobahn-python/blob/master/autobahn/wamp/component.py#L398 events:
@meejah what was start for again? also, we only have leave, not left (or similar) - does it make sense to distinguish between "right before WAMP closing handshake is started" and "ALL former "leave" user observers are done and the WAMP closing handshake is finished." |
So a transport is "connected" at the network level (TCP connection is established), and then "connected" to a session. "connected" is hence overloaded .. but we could also go with:
Any preferences? |
I think "start" is just fired when you "started running your Component" .. but yes is should probably have a matching "done" or "stopped" or similar if we want it? I think "join" / "leave" and "connect" / "disconnect" are the right pairs of words? (or it would want to be "joined" and "left"). IIRC we had semantics that the future returned from Looking in the code, "start" currently fires as soon as something calls Then the overall picture would be something like: start -> connected -> join -> joined/ready -> leave -> left -> disconnected -> end. The "connected -> ... -> disconnected" part would be a loop while doing retrys or re-connects and so "end" would only fire after we'd given up on all transports and retry/reconnect attempts (or were told stop). |
I think saying "a Session is attached to a transport" (or I guess "a transport is attached to a session") makes the most sense and is least ambiguous. |
subseded/fixed in #964 |
As we have been discussing various improvements to the AutobahnPython WAMP API like the new Connection class and just using a WAMP Session object instead of being forced to inherit from a Session base class, now is the chance to design a core API for WAMP that is highly similar between:
The text was updated successfully, but these errors were encountered: