Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python Integration RFC #2747

Merged
merged 4 commits into from Mar 11, 2019

Conversation

Projects
None yet
6 participants
@ibgreen
Copy link
Contributor

commented Mar 4, 2019

For #2746

Background

  • Propose a minimal official integration with Python/Jupyter Notebook

Change List

  • New RFC
Offering such an integration clearly makes sense, as long as we can do it with low effort and maintenance cost.


### Leveraging the deck.gl JSON API

This comment has been minimized.

Copy link
@gnavvy

gnavvy Mar 4, 2019

Member

@ibgreen could you link the JSON API doc/RFC here for easy reference?

My understanding is the deck.gl JSON interface is flexible enough that we don't need frequent updates in accordance with the deck.gl APIs. However, to ensure the generated JSON configs match with the latest deck.gl APIs, developers would still need to follow closely with the API changes and test end-to-end from Python to JSON to JS, which seems to beat the purpose of decoupling Python and JS?

This comment has been minimized.

Copy link
@ibgreen

ibgreen Mar 5, 2019

Author Contributor

could you link the JSON API doc/RFC here for easy reference?

Done

However, to ensure the generated JSON configs match with the latest deck.gl APIs, developers would still need to follow closely with the API changes and test end-to-end from Python to JSON to JS, which seems to beat the purpose of decoupling Python and JS?

Thanks for pointing this out. I added a bunch of text around completeness and stability of the JSON API.

For stability I feel it comes down to versioning. Our core libraries are carefully versioned. I believe that if we can apply the same versioning to the python module that we do to our JavaScript module (not sure if Python uses semver), it will actually be good that the Python API evolves with the deck.gl API so that the latest deck.gl features can always be accessed through an upgrade of the python module.

External Python libraries might want to define a thick abstraction layer over deck.gl to hide our API and keep things stable, but I personally don't think that should be our job.


## Technical Challenges

* Starting up a JavaScript iframe in Notebook

This comment has been minimized.

Copy link
@Pessimistress

Pessimistress Mar 4, 2019

Contributor

Worth mentioning that the solution will be loading pre-bundled deck.gl from CDN?

We do not publish the JSON module as part of the standalone bundle in 6.4. In 7.0 we plan to publish a master bundle (deckgl.min.js) as well as individual bundles for each submodule. We need to make the decision which bundles to include for the Python API.

This comment has been minimized.

Copy link
@ibgreen

ibgreen Mar 5, 2019

Author Contributor

Thanks. Added these comments into the text, together with a comment that we might be able to make the selection of layer modules bundles configurable by Python code.

* Maintain a network connection so that Python can push new payloads?

* Efficient data transfer
* Many Python JS integrations serialize data into the HTML/JS payload.

This comment has been minimized.

Copy link
@Pessimistress

Pessimistress Mar 4, 2019

Contributor

Is it possible for the Python end to start a server that hosts/updates a static data file?

This comment has been minimized.

Copy link
@gnavvy

gnavvy Mar 4, 2019

Member

Not within Jupyter notebook, I assume, unless we modify the jupyter notebook server. But users will have to use our version of jupyter notebook to leverage such capability.

  • Many Python JS integrations serialize data into the HTML/JS payload.

@ibgreen by serializing data into the payload you mean the data is embedded in the HTML file? We can pass in the data on-the-fly via a thin python wrapper like the way we pass props to React components. (e.g.)

This comment has been minimized.

Copy link
@ibgreen

ibgreen Mar 5, 2019

Author Contributor

Looked at the example you linked, following the render function call into the other file. I am not 100% sure but to me it looks like it just stringifies the JSON data into the generated HTML.

* Versioning schemes (do Python package managers use semver)?
* Where to put Python code? (in same github repo, `bindings/python`?)
* Maintenance: deck.gl programmers don't regularly use Python
* ...

This comment has been minimized.

Copy link
@ajduberstein

ajduberstein Mar 5, 2019

Testing/deployment, too.

@ibgreen ibgreen force-pushed the ib/python-rfc branch from df39826 to 6699325 Mar 11, 2019

@ibgreen ibgreen merged commit 649b544 into master Mar 11, 2019

3 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
license/cla Contributor License Agreement is signed.
Details
@coveralls

This comment has been minimized.

Copy link

commented Mar 11, 2019

Coverage Status

Coverage remained the same at 56.799% when pulling 6699325 on ib/python-rfc into 02d8995 on master.

@Pessimistress Pessimistress deleted the ib/python-rfc branch Mar 12, 2019

@heshan0131

This comment has been minimized.

Copy link
Member

commented Mar 22, 2019

Kepler.gl just created a jupyter notebook integration using jupyter's widget api. More details can be found here:

https://github.com/uber/kepler.gl-jupyter

After going back and forth around the 2 approch: a) iframe, b) jupyter widget. I am more inclined to use the widget approach.

Pros

  • bi-directional data binding. User can fully use the python <-> js interface for type checking and interpretation. You don't need to stringify json when you add data (which is the case for iframe). the widget interface will automatically do data transform between json <-> dict, boolean, int and float between js and py. User also has full accessing js app state via python. this is crucial when you want them to export data and config from js app.

  • Save widget state to kernel. One of the pain points of using iframe is that the js state is un-accessible from the python kernel, this makes it difficult to reload the map without rerun all the cell or export the notebook as html.

Cons

  • Have to write a slightly more complex app in both python and js, using the jupyter widget template, which is based on backbone.

You can take a look at the kepler.gl tabbleau widget here

@ibgreen

This comment has been minimized.

Copy link
Contributor Author

commented Mar 22, 2019

That looks really awesome

FWIW @ajduberstein's deck.gl integration #2750 is also going down the Widget path, we have repeated layer updates working without post message and are experimenting with interactivity (sending onClick and maybe onHover back to Python).

We are also thinking about developing some support for the iframe approach as a quick and easy mechanism for general web app integration (that may be outside the scope of deck.gl) and that will likely reuse iframe / post message opt-in ideas from kepler.

@heshan0131

This comment has been minimized.

Copy link
Member

commented Mar 22, 2019

What's the reason behind releasing it as a binding module inside the deck.gl repo, instead of starting a new repo? I can imagine having a monorepo approach would make it easier for keeping the dependencies in-sync, but it also adds to the size of the repo and combine issues that are un-related in the same place

@ibgreen

This comment has been minimized.

Copy link
Contributor Author

commented Mar 22, 2019

What's the reason behind releasing it as a binding module inside the deck.gl repo

Good question. I honestly don't think we have made any firm decision on that but I believe it is discussed in the RFC, at least in spirit: We just want to provide a very minimal basic reference binding with a minimal Python API tightly based on the deck.gl/json module and something small like that can live within the current repo.

Apart from the inconvenience of working with lots of little repos, the key challenge I see here is to keep this integration "alive" for the long term, since the current deck maintainers are mostly non-Python folks. Keeping it in the repo keeps it visible and relevant as we refactor and work on new core releases.

If this feature generates a big volume of issues that could certainly be a reason to split it out, or if doing so is helpful in other ways, but then we could end up with "out of sight, out of mind", and these binding could quickly get rusty....

And even if this feature grows big, it does not automatically mean it should be moved out. For a major example of multiple languages in a single repo, see apache arrow.

They have all their language bindings in a single repo and that enables leveraging e.g. github built-in per-language search filters etc. It works really well at least for my light usage: since I frequently reference e.g. the C++ code to understand things on the JS side better, I personally much prefer this setup to having to clone and sift through multiple repos.

@ajduberstein ajduberstein referenced this pull request Apr 9, 2019

Open

Add Python bindings and Jupyter notebook widget #2929

3 of 6 tasks complete

ajduberstein added a commit to ajduberstein/deck.gl that referenced this pull request Apr 19, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.