Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deepstream.io client for Python #72

Closed
WolframHempel opened this issue Jan 13, 2016 · 36 comments

Comments

@WolframHempel
Copy link
Member

@WolframHempel WolframHempel commented Jan 13, 2016

Create a deepstream.io client for Python, that connects to deepstream via TCP and supports Authentication, Permissioning, Records, Events and RPCs.

Great resources to get started:
Writing a deepstream client
Deepstream Messaging structure
Detailed Cucumber Specs for Incoming and Outgoing messages (and associated behaviors)
Fully featured NodeJS Client as a reference

@jwanglof

This comment has been minimized.

Copy link
Contributor

@jwanglof jwanglof commented Jan 13, 2016

I would love to help out on this! Love Python (thought about creating an issue for this actually). Do you guys have any deadlines at all?

@WolframHempel

This comment has been minimized.

Copy link
Member Author

@WolframHempel WolframHempel commented Jan 13, 2016

That'd be amazing. No deadlines at all. There are only so many languages we know, so we're grateful for any help we can get.

@YavorPaunov

This comment has been minimized.

Copy link

@YavorPaunov YavorPaunov commented Apr 6, 2016

@jwanglof
Have you started working on this? If so, let me know if I can help out. Otherwise, I'd be happy to get it started.

@jwanglof

This comment has been minimized.

Copy link
Contributor

@jwanglof jwanglof commented Apr 6, 2016

Sadly no, real busy this spring. I would love to help if you get it started though =)

@AlexBHarley

This comment has been minimized.

Copy link
Contributor

@AlexBHarley AlexBHarley commented Apr 13, 2016

Hi guys, I have been working on this here alexbharley/deepstream.py. It's slow but steady going as I keep running into problems (my first time doing something like this)

@WolframHempel

This comment has been minimized.

Copy link
Member Author

@WolframHempel WolframHempel commented Apr 15, 2016

Looking really promising. Happy to tweet it to get some initial test users (although might be good to have a basic readme first)

@AlexBHarley

This comment has been minimized.

Copy link
Contributor

@AlexBHarley AlexBHarley commented Apr 16, 2016

@WolframHempel I'll get the events working to 100% and let you know on slack? At the moment the functionality is too limited

@yasserf yasserf removed the help wanted label Jul 13, 2016
@abagshaw

This comment has been minimized.

Copy link

@abagshaw abagshaw commented Aug 8, 2016

Is there any quick-start guide for any of the currently implemented features? Or is this not ready for testing yet? Would love to give it a shot!

@smorin

This comment has been minimized.

Copy link

@smorin smorin commented Sep 23, 2016

If you do this please implement it as a non-blocking python client with asyncio or twisted support

@SneakBug8

This comment has been minimized.

Copy link

@SneakBug8 SneakBug8 commented Oct 15, 2016

As i see, Python client by Alex is dead.

@srossross

This comment has been minimized.

Copy link

@srossross srossross commented Nov 28, 2016

Is this still valid in v2? There no longer seems to be a TCP port.

(and all of the links above are dead.)

@WolframHempel

This comment has been minimized.

Copy link
Member Author

@WolframHempel WolframHempel commented Nov 28, 2016

Afraid so, the current Python client is unofficial. We'll release a high performance version based on the C++ client in the near future

@t3db0t

This comment has been minimized.

Copy link

@t3db0t t3db0t commented Dec 2, 2016

@viridia

This comment has been minimized.

Copy link

@viridia viridia commented Dec 20, 2016

Any update on this? We're considering looking into deepstream but our backend is Python based.

Also, all of the links in the initial message are 404.

@YavorPaunov

This comment has been minimized.

Copy link

@YavorPaunov YavorPaunov commented Dec 21, 2016

@viridia I'm working on a pure Python based client at the moment on YavorPaunov/deepstreampy. Much of the functionality is there but it is not complete yet. Will probably have some time around the holidays when I can speed up development a bit.

@juicycleff

This comment has been minimized.

Copy link

@juicycleff juicycleff commented Jan 2, 2017

I will also love to contribute to the .NET cient. I am relatively new to deepstream to be exact 3 day old.

@yasserf

This comment has been minimized.

Copy link
Member

@yasserf yasserf commented Jan 2, 2017

@YavorPaunov I just looked through your python client, I don't have much experience in python tbh but the builds + steps + structure looks great!

If there is anything we can help with just let us know!

@rajanshah

This comment has been minimized.

Copy link

@rajanshah rajanshah commented Jan 3, 2017

@YavorPaunov I am interested in utilizing your deepstreampy client. I can install your package in my virtual environment. Is there a way, you can add quick-start guide on how to use it in standalone python program and publish some messages, etc...?

@albrnick

This comment has been minimized.

Copy link

@albrnick albrnick commented Mar 7, 2017

@YavorPaunov Your client looks very interesting! I was wondering if you the basic record getting/setting/subscripting, etc.. is working?

Thanks! :)

@YavorPaunov

This comment has been minimized.

Copy link

@YavorPaunov YavorPaunov commented Mar 15, 2017

@albrnick
Basically, it should work. Feel free to try it and please let me know if you find any issues.
I wouldn't say the client as a whole is in a production ready state though.

I am also planning to add documentation (complete README, examples and docstrings) in the near future. Life is pretty busy at the moment, unfortunately :(

@Serkan-devel

This comment has been minimized.

Copy link

@Serkan-devel Serkan-devel commented Apr 28, 2017

Does it support Python 3?

@charleswhchan

This comment has been minimized.

Copy link

@charleswhchan charleswhchan commented Jun 19, 2017

Just tested @YavorPaunov's deepstreampy (Python 2.7), and was able to emit() an event to DS. I created a PR as a really basic example to help others get started.

Look forward to an official Python client from DS team in future. 😃

@hipkiss91

This comment has been minimized.

Copy link

@hipkiss91 hipkiss91 commented Jul 25, 2017

What would be a rough date for official release?
Just started using python. I'm eager to get deepstream involved :)

@viridia

This comment has been minimized.

Copy link

@viridia viridia commented Jul 25, 2017

Turns out that there's a difficulty involved in writing a deepstream client for Python, which is that there are many different environments for handling asynchronous events - twisted, asyncio, etc - and the design of the deepstream client is probably going to be quite different depending on which environment it is built for. You won't be able to take a deepstream client written for asyncio and use it within a twisted environment, and vice versa.

In my own use case, for example, I need to be able to publish deepstream events from within a Django-REST server. Django-REST is based on WSGI, which is a simple run loop that processes incoming HTTP requests and invokes callbacks. Any asynchronous processing - such as handling incoming messages - has to be done in the context of servicing some request. Otherwise, there's no place for the incoming data to go. This doesn't fit with the deepstream model at all.

My solution in this case will be to implement just a tiny portion of the deepstream protocol - basically the parts that deal with authentication and sending events - and treat deepstream like a conventional RESTful service. My Django server won't be able to receive events or rpc calls, but that's OK, there are other microservices in the system (written in Node.js) that can perform that job.

Note that in languages like Node.js, this is not a problem since the async primitives (like Promises) are pretty much built into the language, so all of the various packages for dealing with async operations are somewhat interoperable.

@jlowin

This comment has been minimized.

Copy link

@jlowin jlowin commented Jul 26, 2017

@YavorPaunov has a nice Tornado implementation. My read of the latest blog post suggests that 1) the HTTP API may be open sourced in Deepstream 3.0 (which would solve @viridia's use case) and 2) that unfortunately there may be breaking changes to @YavorPaunov's version.

As for different Python async environments, while that's true, Python 3 has introduced a canonical framework (asycnio) as well as async/await primitives. Most async libraries have introduced compatibility with these native coroutines. The path of least resistance, absent some external factor, would be to build a library on asyncio that could be bridged to other compatible frameworks.

Unless of course we get a green light that @YavorPaunov's implementation is the right direction. It would be nice to hear from the devs about their plans here. @WolframHempel @yasserf Deepstream is such a neat technology but I've had to pass on it for a number of projects because there is no clarity on whether working to integrate it with Python is a waste of time. In particular, adding the current community-supported Tornado client and finding out that 3.0 obsoletes it or introduces a non-Tornado official client would be bad.

@AlexBHarley

This comment has been minimized.

Copy link
Contributor

@AlexBHarley AlexBHarley commented Jul 26, 2017

@viridia @jlowin the deepstream v3.0 release won't be breaking compatibility with any clients. The changes are mainly internal and we've marked this as a major release due to other reasons (stay tuned for the blog post soon).

@YavorPaunov's library looks great and is definitely what we'd like out of community SDKs. As with most open source projects I'm sure he would appreciate more contributors.

@jlowin

This comment has been minimized.

Copy link

@jlowin jlowin commented Jul 26, 2017

@AlexBHarley thanks!

@luis-kaufmann-silva

This comment has been minimized.

Copy link

@luis-kaufmann-silva luis-kaufmann-silva commented Aug 2, 2017

"Writing a deepstream client" page not found :(

@viridia

This comment has been minimized.

Copy link

@viridia viridia commented Sep 21, 2017

Right now the Tornado-based python client is not going to satisfy my use case.

What I am trying to do is replace WAMP/Crossbar in my environment with deepstream.io, since deepstream supports multiple server instances and Crossbar does not. We've already managed to do this for the main server and for the frontend, but there are a couple of other pieces which are stuck because of lack of driver support.

One of these pieces is a small daemon/app launcher that runs windows machines that our users connect to via VNC. One of the launcher's jobs is to mount various filesystems in response to various kinds of cloud events. It currently does this using the Autobahn/WAMP driver which is written on top of Twisted. What I want to do is replace the Autobahn/WAMP driver with an equivalent deepstream.io driver.

The daemon does not understand HTTP or any other kinds of network protocols other than RPCs over real-time websockets. Using deepstream would be ideal, since the daemon needs to communicate with the browser frontend, but the browser shouldn't need to know the exact IP address of the machine the daemon is running on. With something like deepstream the daemon can simply register rpc methods whose name includes the instance id.

The problem I have is lack of driver support for deepstream. Since the daemon doesn't talk HTTP, dropping in a webserver like Tornado makes no sense. (Also, Tornado is not recommended on Windows machines.) Right now a twisted-based driver would probably be the easiest, although if an asyncio driver showed up I would probably consider migrating to Python3. The other thing we've been considering is re-writing the daemon entirely in node.js, but that's a bigger job and would take more engineering time than we have to spare right now.

@yasserf

This comment has been minimized.

Copy link
Member

@yasserf yasserf commented Sep 22, 2017

I'm not very versed in python, but I'm guessing you can't easily our websocket libraries right? In java we allow websocket libraries to be injected if needed.

@alv000h

This comment has been minimized.

Copy link

@alv000h alv000h commented Nov 29, 2017

+1 python client w/ asyncio

@thakkardharmik

This comment has been minimized.

Copy link

@thakkardharmik thakkardharmik commented Dec 18, 2017

Is this still open?

@dpisarenko

This comment has been minimized.

Copy link

@dpisarenko dpisarenko commented Apr 17, 2018

@WolframHempel @AlexBHarley I'm trying to understand the scope of Python client project. I looked through the spec project and wrote out all features in this gist. Is it correct? If a Python client implementation has all the features listed there, is it ready then?

@WolframHempel

This comment has been minimized.

Copy link
Member Author

@WolframHempel WolframHempel commented Apr 17, 2018

@dpisarenko Your list looks very conclusive and is correctly based on the unterlying specs. From experience though I know that the devil is in the detail. While e.g. the protocol to record behaviour spec is fairly short, the actual client behaviour for merging paths, providing received notifications etc. can be quite challenging to get right. In general though I think the answer is "yes, a Python client implementing all features listed in this gist would be considered complete and would cover the full range of deepstream features"

@dpisarenko

This comment has been minimized.

Copy link

@dpisarenko dpisarenko commented Apr 17, 2018

@WolframHempel Thanks!

@yasserf

This comment has been minimized.

Copy link
Member

@yasserf yasserf commented Aug 5, 2019

I'm closing this issue since the messages within it are outdated to V3 and Text Protocol, and would probably just confuse anyone starting to look into it.

The goal is to write a client in C or C++ around the protobuf protocol and then get other clients to just call into that.

@yasserf yasserf closed this Aug 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.