Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deepstream.io client for Python #72

Closed
WolframHempel opened this issue Jan 13, 2016 · 38 comments
Closed

deepstream.io client for Python #72

WolframHempel opened this issue Jan 13, 2016 · 38 comments

Comments

@WolframHempel
Copy link
Member

@WolframHempel WolframHempel commented Jan 13, 2016

Create a deepstream.io client for Python, that connects to deepstream via TCP and supports Authentication, Permissioning, Records, Events and RPCs.

Great resources to get started:
Writing a deepstream client
Deepstream Messaging structure
Detailed Cucumber Specs for Incoming and Outgoing messages (and associated behaviors)
Fully featured NodeJS Client as a reference

@jwanglof
Copy link
Contributor

@jwanglof jwanglof commented Jan 13, 2016

I would love to help out on this! Love Python (thought about creating an issue for this actually). Do you guys have any deadlines at all?

@WolframHempel
Copy link
Member Author

@WolframHempel WolframHempel commented Jan 13, 2016

That'd be amazing. No deadlines at all. There are only so many languages we know, so we're grateful for any help we can get.

@YavorPaunov
Copy link

@YavorPaunov YavorPaunov commented Apr 6, 2016

@jwanglof
Have you started working on this? If so, let me know if I can help out. Otherwise, I'd be happy to get it started.

@jwanglof
Copy link
Contributor

@jwanglof jwanglof commented Apr 6, 2016

Sadly no, real busy this spring. I would love to help if you get it started though =)

@AlexBHarley
Copy link
Contributor

@AlexBHarley AlexBHarley commented Apr 13, 2016

Hi guys, I have been working on this here alexbharley/deepstream.py. It's slow but steady going as I keep running into problems (my first time doing something like this)

@WolframHempel
Copy link
Member Author

@WolframHempel WolframHempel commented Apr 15, 2016

Looking really promising. Happy to tweet it to get some initial test users (although might be good to have a basic readme first)

@AlexBHarley
Copy link
Contributor

@AlexBHarley AlexBHarley commented Apr 16, 2016

@WolframHempel I'll get the events working to 100% and let you know on slack? At the moment the functionality is too limited

@abagshaw
Copy link

@abagshaw abagshaw commented Aug 8, 2016

Is there any quick-start guide for any of the currently implemented features? Or is this not ready for testing yet? Would love to give it a shot!

@smorin
Copy link

@smorin smorin commented Sep 23, 2016

If you do this please implement it as a non-blocking python client with asyncio or twisted support

@SneakBug8
Copy link

@SneakBug8 SneakBug8 commented Oct 15, 2016

As i see, Python client by Alex is dead.

@srossross
Copy link

@srossross srossross commented Nov 28, 2016

Is this still valid in v2? There no longer seems to be a TCP port.

(and all of the links above are dead.)

@WolframHempel
Copy link
Member Author

@WolframHempel WolframHempel commented Nov 28, 2016

Afraid so, the current Python client is unofficial. We'll release a high performance version based on the C++ client in the near future

@t3db0t
Copy link

@t3db0t t3db0t commented Dec 2, 2016

@viridia
Copy link

@viridia viridia commented Dec 20, 2016

Any update on this? We're considering looking into deepstream but our backend is Python based.

Also, all of the links in the initial message are 404.

@YavorPaunov
Copy link

@YavorPaunov YavorPaunov commented Dec 21, 2016

@viridia I'm working on a pure Python based client at the moment on YavorPaunov/deepstreampy. Much of the functionality is there but it is not complete yet. Will probably have some time around the holidays when I can speed up development a bit.

@juicycleff
Copy link

@juicycleff juicycleff commented Jan 2, 2017

I will also love to contribute to the .NET cient. I am relatively new to deepstream to be exact 3 day old.

@yasserf
Copy link
Contributor

@yasserf yasserf commented Jan 2, 2017

@YavorPaunov I just looked through your python client, I don't have much experience in python tbh but the builds + steps + structure looks great!

If there is anything we can help with just let us know!

@rajanshah
Copy link

@rajanshah rajanshah commented Jan 3, 2017

@YavorPaunov I am interested in utilizing your deepstreampy client. I can install your package in my virtual environment. Is there a way, you can add quick-start guide on how to use it in standalone python program and publish some messages, etc...?

@albrnick
Copy link

@albrnick albrnick commented Mar 7, 2017

@YavorPaunov Your client looks very interesting! I was wondering if you the basic record getting/setting/subscripting, etc.. is working?

Thanks! :)

@YavorPaunov
Copy link

@YavorPaunov YavorPaunov commented Mar 15, 2017

@albrnick
Basically, it should work. Feel free to try it and please let me know if you find any issues.
I wouldn't say the client as a whole is in a production ready state though.

I am also planning to add documentation (complete README, examples and docstrings) in the near future. Life is pretty busy at the moment, unfortunately :(

@Serkan-devel
Copy link

@Serkan-devel Serkan-devel commented Apr 28, 2017

Does it support Python 3?

@charleswhchan
Copy link

@charleswhchan charleswhchan commented Jun 19, 2017

Just tested @YavorPaunov's deepstreampy (Python 2.7), and was able to emit() an event to DS. I created a PR as a really basic example to help others get started.

Look forward to an official Python client from DS team in future. 😃

@hipkiss91
Copy link

@hipkiss91 hipkiss91 commented Jul 25, 2017

What would be a rough date for official release?
Just started using python. I'm eager to get deepstream involved :)

@viridia
Copy link

@viridia viridia commented Jul 25, 2017

Turns out that there's a difficulty involved in writing a deepstream client for Python, which is that there are many different environments for handling asynchronous events - twisted, asyncio, etc - and the design of the deepstream client is probably going to be quite different depending on which environment it is built for. You won't be able to take a deepstream client written for asyncio and use it within a twisted environment, and vice versa.

In my own use case, for example, I need to be able to publish deepstream events from within a Django-REST server. Django-REST is based on WSGI, which is a simple run loop that processes incoming HTTP requests and invokes callbacks. Any asynchronous processing - such as handling incoming messages - has to be done in the context of servicing some request. Otherwise, there's no place for the incoming data to go. This doesn't fit with the deepstream model at all.

My solution in this case will be to implement just a tiny portion of the deepstream protocol - basically the parts that deal with authentication and sending events - and treat deepstream like a conventional RESTful service. My Django server won't be able to receive events or rpc calls, but that's OK, there are other microservices in the system (written in Node.js) that can perform that job.

Note that in languages like Node.js, this is not a problem since the async primitives (like Promises) are pretty much built into the language, so all of the various packages for dealing with async operations are somewhat interoperable.

@jlowin
Copy link

@jlowin jlowin commented Jul 26, 2017

@YavorPaunov has a nice Tornado implementation. My read of the latest blog post suggests that 1) the HTTP API may be open sourced in Deepstream 3.0 (which would solve @viridia's use case) and 2) that unfortunately there may be breaking changes to @YavorPaunov's version.

As for different Python async environments, while that's true, Python 3 has introduced a canonical framework (asycnio) as well as async/await primitives. Most async libraries have introduced compatibility with these native coroutines. The path of least resistance, absent some external factor, would be to build a library on asyncio that could be bridged to other compatible frameworks.

Unless of course we get a green light that @YavorPaunov's implementation is the right direction. It would be nice to hear from the devs about their plans here. @WolframHempel @yasserf Deepstream is such a neat technology but I've had to pass on it for a number of projects because there is no clarity on whether working to integrate it with Python is a waste of time. In particular, adding the current community-supported Tornado client and finding out that 3.0 obsoletes it or introduces a non-Tornado official client would be bad.

@AlexBHarley
Copy link
Contributor

@AlexBHarley AlexBHarley commented Jul 26, 2017

@viridia @jlowin the deepstream v3.0 release won't be breaking compatibility with any clients. The changes are mainly internal and we've marked this as a major release due to other reasons (stay tuned for the blog post soon).

@YavorPaunov's library looks great and is definitely what we'd like out of community SDKs. As with most open source projects I'm sure he would appreciate more contributors.

@jlowin
Copy link

@jlowin jlowin commented Jul 26, 2017

@AlexBHarley thanks!

@luis-kaufmann-silva
Copy link

@luis-kaufmann-silva luis-kaufmann-silva commented Aug 2, 2017

"Writing a deepstream client" page not found :(

@viridia
Copy link

@viridia viridia commented Sep 21, 2017

Right now the Tornado-based python client is not going to satisfy my use case.

What I am trying to do is replace WAMP/Crossbar in my environment with deepstream.io, since deepstream supports multiple server instances and Crossbar does not. We've already managed to do this for the main server and for the frontend, but there are a couple of other pieces which are stuck because of lack of driver support.

One of these pieces is a small daemon/app launcher that runs windows machines that our users connect to via VNC. One of the launcher's jobs is to mount various filesystems in response to various kinds of cloud events. It currently does this using the Autobahn/WAMP driver which is written on top of Twisted. What I want to do is replace the Autobahn/WAMP driver with an equivalent deepstream.io driver.

The daemon does not understand HTTP or any other kinds of network protocols other than RPCs over real-time websockets. Using deepstream would be ideal, since the daemon needs to communicate with the browser frontend, but the browser shouldn't need to know the exact IP address of the machine the daemon is running on. With something like deepstream the daemon can simply register rpc methods whose name includes the instance id.

The problem I have is lack of driver support for deepstream. Since the daemon doesn't talk HTTP, dropping in a webserver like Tornado makes no sense. (Also, Tornado is not recommended on Windows machines.) Right now a twisted-based driver would probably be the easiest, although if an asyncio driver showed up I would probably consider migrating to Python3. The other thing we've been considering is re-writing the daemon entirely in node.js, but that's a bigger job and would take more engineering time than we have to spare right now.

@yasserf
Copy link
Contributor

@yasserf yasserf commented Sep 22, 2017

I'm not very versed in python, but I'm guessing you can't easily our websocket libraries right? In java we allow websocket libraries to be injected if needed.

@alv000h
Copy link

@alv000h alv000h commented Nov 29, 2017

+1 python client w/ asyncio

@thakkardharmik
Copy link

@thakkardharmik thakkardharmik commented Dec 18, 2017

Is this still open?

@mentiflectax
Copy link

@mentiflectax mentiflectax commented Apr 17, 2018

@WolframHempel @AlexBHarley I'm trying to understand the scope of Python client project. I looked through the spec project and wrote out all features in this gist. Is it correct? If a Python client implementation has all the features listed there, is it ready then?

@WolframHempel
Copy link
Member Author

@WolframHempel WolframHempel commented Apr 17, 2018

@dpisarenko Your list looks very conclusive and is correctly based on the unterlying specs. From experience though I know that the devil is in the detail. While e.g. the protocol to record behaviour spec is fairly short, the actual client behaviour for merging paths, providing received notifications etc. can be quite challenging to get right. In general though I think the answer is "yes, a Python client implementing all features listed in this gist would be considered complete and would cover the full range of deepstream features"

@mentiflectax
Copy link

@mentiflectax mentiflectax commented Apr 17, 2018

@WolframHempel Thanks!

@yasserf
Copy link
Contributor

@yasserf yasserf commented Aug 5, 2019

I'm closing this issue since the messages within it are outdated to V3 and Text Protocol, and would probably just confuse anyone starting to look into it.

The goal is to write a client in C or C++ around the protobuf protocol and then get other clients to just call into that.

@yasserf yasserf closed this Aug 5, 2019
@Allan-Nava
Copy link

@Allan-Nava Allan-Nava commented Mar 17, 2020

Does it support Python 3?

Any news?

@yasserf
Copy link
Contributor

@yasserf yasserf commented Mar 24, 2020

Unfortunately not no, writing clients is very time consuming and it would be required to be a community effort

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet