-
Notifications
You must be signed in to change notification settings - Fork 194
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for asynchronous client #39
Conversation
@turu Thanks for this contrib. Do you mind filling out the CLA here: druid.io/community/cla.html |
@fjy I already did. Can you verify that you have my CLA registered? |
Note there's a commit from https://github.com/l-j in here |
@fjy @drcrallen Is there any update on the status of this pr? |
@turu apologies for delay, will ask people to finish reviewing this @drcrallen @xvrl please review |
@turu can we resolve the merge conflicts and I will merge this |
@@ -0,0 +1,157 @@ | |||
# |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please use the new license header
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed the year in license header, from 2013 to 2016. Is that ok? If there is a completely new header somewhere, which is supposed to be used, could you point me to it?
@fjy I resolved merge conflicts. To the best of my knowledge, everything should be ok. All tests are passing. |
@turu Thanks for the comprehensive description, a few questions:
|
Otherwise I'm 👍 on this one, assuming someone has tested that the existing client still works well in practice. |
* refactored and tested client abstraction * improved setup.py enabling pip extras * fixed python3 compatibility * fixed issues with test_export_tsv testcase * provided backward compatibility Fixes druid-io#29
@xvrl Regarding the points you mentioned:
|
Add support for asynchronous client
In a nutshell: This pull request adds, in a fully backward compatible way, a new optional, asynchronous client to the library.
Why this matters?
Druid can be very useful in many scenarios different than ad-hoc analysis of online data - f.e it can be successfully used to create flexible and interactive analytics dashboards or monitors. Applying asynchronous programming style is often a go-to architectural choice, when designing such systems - especially, when they are implemented in single threaded languages like python. The lack of asynchronous client can therefore be a serious problem. This pull request fixes it.
What has been done?
(accidentally ;)) Fixed issues:
PyDruid
not extendingobject
#29A note on implementation of AsyncPyDruid:
Asynchronous client uses elements of Tornado framework. This was a conscious choice based on the fact that Tornado is stable and mature, and integrates very well with python’s 3.5 native async support (async, await keywords) as well as other asynchronous frameworks. As a result one is not locked into using Tornado for the whole of his/her app.
See: http://www.tornadoweb.org/en/stable/guide/coroutines.html#python-3-5-async-and-await
A note on backward compatibility:
As a result of refactoring, clients are reusable. This is achieved through clients’ query methods returning Query objects filled with results, rather than result dictionary. However, this change is fully backward compatible - clients still expose the same (now deprecated) interface allowing one to access details of the most recently executed query. Query objects on the other hand, act as wrappers around result dictionary, giving users the same dictionary interface they had before, enriched to include methods for exporting results and accessing query details.
A note on python 2.x <-> 3.x compatibility:
The changes have been tested and should work reliably both in python 2.x and 3.x
Additional note on the authors:
Changes introduced by this pull request have been originally developed in https://github.com/turu/pydruid by me (github.com/turu), and github.com/l-j. Since then, they were forked to our organization (github.com/Dreamlab) and they are already used in production. We both signed the CLA.
Best regards,
Piotr Turek