Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

node.js integration and static image generation #3047

Closed
wants to merge 31 commits into from
Closed
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
15401cc
Fix HTML5 canvas context's properties and methods only once
mattpap Oct 26, 2015
de7cc11
Resize HTML5 Canvas instance manually in node.js (jsdom)
mattpap Oct 28, 2015
a2c053f
Upgrade jsdom to version 7*
mattpap Oct 29, 2015
b67b80c
Add canvas, htmlparser2 and uuid to package.json
mattpap Oct 29, 2015
96fcc68
Allow to load models from a <script type="text/x-bokeh"> element
mattpap Oct 29, 2015
af8c682
Store serialized models in a <script type='text/x-bokeh'> element
mattpap Oct 29, 2015
c16d186
Make bokeh.js a node.js module, so we can use bokehRequire()
mattpap Oct 29, 2015
75657c1
Use window.Image for symmetry with window.Canvas
mattpap Oct 29, 2015
dc5aa82
Preliminary and very naive approach to render:done event
mattpap Oct 29, 2015
019fcd5
Use toLowerCase() together with element.nodeName
mattpap Oct 29, 2015
e756bcc
Preliminary offline rendering script for node.js
mattpap Oct 29, 2015
a839346
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Nov 2, 2015
eeb3871
Add scikit-learn to conda package under test/examples (#3041)
mattpap Nov 2, 2015
babd570
Add location and navigator packages to package.json
mattpap Nov 2, 2015
a6b60c0
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Nov 11, 2015
3ffc50a
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Nov 11, 2015
4a1af80
Update render.js to the new code base
mattpap Nov 15, 2015
5fad6dd
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Nov 15, 2015
244b979
Restructure render.js and allow stdin input
mattpap Nov 16, 2015
ae9cb9f
Make log level and JS/CSS paths configurable
mattpap Nov 16, 2015
851cc93
Make custom models be part of the JSON protocol
mattpap Nov 18, 2015
6e51490
Restore pretty (dev) output from serialize_json()
mattpap Nov 18, 2015
468c775
Preliminary plotting API for PNG output
mattpap Nov 18, 2015
7f69c3c
Add examples/plotting/png/bollinger.py
mattpap Nov 18, 2015
3eaf4a0
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Nov 22, 2015
1b9e5b5
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Nov 26, 2015
a60b37c
Separate node.js integration from bokehjs
mattpap Nov 26, 2015
df12d4e
Change _check_plot_objects -> _check_one_model after merging master
mattpap Nov 26, 2015
6e3dfc5
Ignore *.png files under examples/plotting/png
mattpap Nov 26, 2015
1f4e178
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Nov 27, 2015
cc6077b
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Dec 8, 2015
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 4 additions & 0 deletions bokeh/_json_encoder.py
Expand Up @@ -7,6 +7,8 @@
import decimal

from .util.serialization import transform_series, transform_array
from .settings import settings

import numpy as np

try:
Expand Down Expand Up @@ -79,4 +81,6 @@ def default(self, obj):
return self.transform_python_types(obj)

def serialize_json(obj, encoder=BokehJSONEncoder, **kwargs):
if settings.pretty(False):
kwargs["indent"] = 2
return json.dumps(obj, cls=encoder, allow_nan=False, **kwargs)
8 changes: 0 additions & 8 deletions bokeh/_templates/doc_js.js
@@ -1,11 +1,3 @@
{%- if custom_models -%}
Bokeh.Collections.register_models({
{% for name, impl in custom_models.items() -%}
"{{ name }}": {{ impl }},
{%- endfor %}
});
{% endif -%}

{% if websocket_url -%}
var websocket_url = "{{ websocket_url }}";
{%- else %}
Expand Down
5 changes: 5 additions & 0 deletions bokeh/_templates/script_tag.html
Expand Up @@ -5,6 +5,11 @@
:type js_code: str

#}
{%- if docs_json_id -%}
<script id="{{ docs_json_id }}" type="text/x-bokeh">
{{ docs_json|indent(4) }}
</script>
{% endif -%}
<script type="text/javascript">
{{ js_code }}
</script>
18 changes: 18 additions & 0 deletions bokeh/document.py
Expand Up @@ -319,6 +319,20 @@ def _initialize_references_json(cls, references_json, references):
del obj_attrs[key]
instance.update(**obj_attrs)

def _collect_custom_models(self):
custom_models = {}

for obj in self.roots:
for ref in obj.references():
impl = getattr(ref.__class__, "__implementation__", None)
if impl is not None:
name = ref.__class__.__name__

if name not in custom_models:
custom_models[name] = {'impl': impl, 'deps': {}}

return custom_models

def to_json_string(self):
''' Convert the document to a JSON string. '''

Expand All @@ -336,6 +350,10 @@ def to_json_string(self):
}
}

custom_models = self._collect_custom_models()
if custom_models:
json['models'] = custom_models

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will cause us problems; models code is not part of Document. Here we are tacking an extra payload on to Document that we'd like to go to JavaScript, because we happen to know we'll send the Document to JavaScript.
It's an unexpected side effect that Document.from_json on the coffee side loads custom models, it should only fill in Document, not have surprising side effects.

What I think would be cleaner: in _standalone_docs_json_and_render_items, redefine docs_json to include the models (maybe demote the current dict of docs, so it's { 'models' : { ... }, 'docs' : current_docs_json_here } and then in Bokeh.embed.embed_items of course the code has to be changed to match.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

another advantage of doing it in _standalone_docs_json_and_render_items is that if there's multiple docs we only send the models once

return serialize_json(json)

def to_json(self):
Expand Down
90 changes: 19 additions & 71 deletions bokeh/embed.py
Expand Up @@ -13,7 +13,6 @@

from __future__ import absolute_import

import re
import uuid
from warnings import warn

Expand Down Expand Up @@ -130,10 +129,8 @@ def components(plot_objects, resources=None, wrap_script=True, wrap_plot_info=Tr

with _ModelInDocument(plot_objects):
(docs_json, render_items) = _standalone_docs_json_and_render_items(plot_objects)
custom_models = _extract_custom_models(plot_objects)

script = _script_for_render_items(docs_json, render_items, custom_models=custom_models,
websocket_url=None, wrap_script=wrap_script)
script = _script_for_render_items(docs_json, render_items, websocket_url=None, wrap_script=wrap_script)
script = encode_utf8(script)

if wrap_plot_info:
Expand All @@ -153,47 +150,6 @@ def components(plot_objects, resources=None, wrap_script=True, wrap_plot_info=Tr
else:
return script, tuple(results)

def _escape_code(code):
""" Escape JS/CS source code, so that it can be embbeded in a JS string.

This is based on https://github.com/joliss/js-string-escape.
"""
def escape(match):
ch = match.group(0)

if ch == '"' or ch == "'" or ch == '\\':
return '\\' + ch
elif ch == '\n':
return '\\n'
elif ch == '\r':
return '\\r'
elif ch == '\u2028':
return '\\u2028'
elif ch == '\u2029':
return '\\u2029'

return re.sub(u"""['"\\\n\r\u2028\u2029]""", escape, code)

def _extract_custom_models(plot_objects):
custom_models = {}

def extract_from_model(model):
for r in model.references():
impl = getattr(r.__class__, "__implementation__", None)
if impl is not None:
name = r.__class__.__name__
impl = "['%s', {}]" % _escape_code(impl)
custom_models[name] = impl

for o in plot_objects:
if isinstance(o, Document):
for r in o.roots:
extract_from_model(r)
else:
extract_from_model(o)

return custom_models

def notebook_div(plot_object):
''' Return HTML for a div that will display a Bokeh plot in an
IPython Notebook
Expand All @@ -215,11 +171,8 @@ def notebook_div(plot_object):

with _ModelInDocument(plot_object):
(docs_json, render_items) = _standalone_docs_json_and_render_items([plot_object])
custom_models = _extract_custom_models([plot_object])

script = _script_for_render_items(docs_json, render_items,
custom_models=custom_models,
websocket_url=None)
script = _script_for_render_items(docs_json, render_items, websocket_url=None)

item = render_items[0]

Expand Down Expand Up @@ -285,9 +238,8 @@ def file_html(plot_objects,

(docs_json, render_items) = _standalone_docs_json_and_render_items(plot_objects)
title = _title_from_plot_objects(plot_objects, title)
custom_models = _extract_custom_models(plot_objects)
return _html_page_for_render_items(resources, docs_json, render_items, title=title,
custom_models=custom_models, websocket_url=None,
websocket_url=None,
js_resources=js_resources, css_resources=css_resources,
template=template, template_variables=template_variables,
use_widgets=_use_widgets(plot_objects))
Expand Down Expand Up @@ -400,28 +352,24 @@ def autoload_server(plot_object, app_path="/", session_id=DEFAULT_SESSION_ID, ur

return encode_utf8(tag)

def _script_for_render_items(docs_json, render_items, websocket_url,
custom_models, wrap_script=True):
# this avoids emitting the "register custom models" code at all
# just to register an empty set
if (custom_models is not None) and len(custom_models) == 0:
custom_models = None

plot_js = _wrap_in_function(
DOC_JS.render(
custom_models=custom_models,
websocket_url=websocket_url,
docs_json=serialize_json(docs_json),
render_items=serialize_json(render_items)
def _script_for_render_items(docs_json, render_items, websocket_url, wrap_script=True):
def plot_js(docs_json):
return _wrap_in_function(
DOC_JS.render(
websocket_url=websocket_url,
docs_json=serialize_json(docs_json),
render_items=serialize_json(render_items),
)
)
)

if wrap_script:
return SCRIPT_TAG.render(js_code=plot_js)
id = str(uuid.uuid4())
return SCRIPT_TAG.render(docs_json_id=id, docs_json=serialize_json(docs_json), js_code=plot_js(id))
else:
return plot_js
return plot_js(docs_json)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this refactor is incomplete and we end up with one function that really does two separate things. As it is, the function is named and expected elsewhere to return one hunk of script, which is optionally wrapped in one <script> tag. If we want to change it so that to embed render items, you need N script tags, we should rename the function _scripts_for_render_items, return a list, and have all callers handle the list result.

wrap_script should control whether each element is wrapped in a <script> tag, not how many elements there are. Though with docs_json in its own tag I think we have to always wrap_script so that parameter would maybe have to go.

The simplest thing for this PR is probably to drop the "externalize docs json into its own <script> tag" and do that as its own PR, but if you think externalizing the docs json is important to this PR, I think we should finish the refactor - do it always/consistently. There's no value I see to having two ways to do this: either externalize the docs_json into tags, or don't.

Going from one script to a list of scripts will break the components() API I think. If we don't want to break components() I'm not sure we should do this at all, is it really worth two codepaths?

If we do have two codepaths, I'd rather not overload wrap_script - it refers to whether to wrap the scripts in a script tag, which is separate from whether the docs_json is in its own script tag. Maybe a separate function _scripts_for_render_items_with_externalized_json ?


def _html_page_for_render_items(resources, docs_json, render_items, title, websocket_url,
custom_models, js_resources=None, css_resources=None,
js_resources=None, css_resources=None,
template=FILE, template_variables={}, use_widgets=True):
if title is None:
title = DEFAULT_TITLE
Expand Down Expand Up @@ -449,7 +397,7 @@ def _html_page_for_render_items(resources, docs_json, render_items, title, webso
css_resources = css_resources.use_widgets(use_widgets)
bokeh_css = css_resources.render_css()

script = _script_for_render_items(docs_json, render_items, websocket_url, custom_models)
script = _script_for_render_items(docs_json, render_items, websocket_url)

template_variables_full = template_variables.copy()

Expand Down Expand Up @@ -597,7 +545,7 @@ def server_html_page_for_models(session_id, model_ids, resources, title, websock
})

return _html_page_for_render_items(resources, {}, render_items, title,
websocket_url=websocket_url, custom_models=None)
websocket_url=websocket_url)

def server_html_page_for_session(session_id, resources, title, websocket_url):
elementid = str(uuid.uuid4())
Expand All @@ -609,4 +557,4 @@ def server_html_page_for_session(session_id, resources, title, websocket_url):
}]

return _html_page_for_render_items(resources, {}, render_items, title,
websocket_url=websocket_url, custom_models=None)
websocket_url=websocket_url)
57 changes: 54 additions & 3 deletions bokeh/io.py
Expand Up @@ -101,6 +101,12 @@ def output_file(filename, title="Bokeh Plot", autosave=False, mode="inline", roo
root_dir=root_dir
)

def output_png(filename, autosave=False):
_state.output_png(
filename,
autosave=autosave,
)

def output_notebook(resources=None, verbose=False, hide_banner=False):
''' Configure the default output state to generate output in
Jupyter/IPython notebook cells when :func:`show` is called.
Expand Down Expand Up @@ -242,17 +248,21 @@ def _show_with_state(obj, state, browser, new):

if state.notebook:
_show_notebook_with_state(obj, state)

elif state.session_id and state.server_url:
_show_server_with_state(obj, state, new, controller)

if state.file:
elif state.file:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the code was if rather than elif on purpose before, that is, output to both server and file at the same time is supposed to work. (I'm not advocating that, but it's my impression of the previous code's intent, and to unwind the intent there are more changes to be made I think - the docs are wrong for example.)

_show_file_with_state(obj, state, new, controller)
elif state.png:
_show_png_with_state(obj, state, new, controller)

def _show_file_with_state(obj, state, new, controller):
save(obj, state=state)
controller.open("file://" + os.path.abspath(state.file['filename']), new=_new_param[new])

def _show_png_with_state(obj, state, new, controller):
savepng(obj, state=state)
controller.open("file://" + os.path.abspath(state.png['filename']), new=_new_param[new])

def _show_notebook_with_state(obj, state):
if state.session_id:
push(state=state)
Expand Down Expand Up @@ -346,6 +356,47 @@ def _save_helper(obj, filename, resources, title, validate):
with io.open(filename, "w", encoding="utf-8") as f:
f.write(decode_utf8(html))

def savepng(obj, filename=None, state=None, validate=True):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should it be save_png with an underscore?

Why did you go with a separate savepng vs. making save() work on png output?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you go with a separate savepng vs. making save() work on png output?

Because save() takes extra arguments. This could be resolved, by inspection and raising a few exceptions. I actually intended to make at least save() know about savepng() (by inspecting file extension).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should it be save_png with an underscore?

Given that this is pylab-style API, I think not, but I'm fine with adding one.

if state is None:
state = _state

if filename is None and state.png:
filename = state.png['filename']

if filename is None:
raise RuntimeError("savepng() called but no filename was supplied and output_png(...) was never called, nothing saved")

with _ModelInDocument(obj):
if isinstance(obj, Component):
doc = obj.document
elif isinstance(obj, Document):
doc = obj
else:
raise RuntimeError("Unable to save object of type '%s'" % type(obj))

if validate:
doc.validate()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the code from with _ModelInDocument through validate() could probably be factored out since it's in _save_helper as well


from subprocess import Popen, PIPE
from os.path import join, dirname

from .resources import Resources

resources = Resources(mode="absolute-dev")

from .embed import _check_plot_objects, _standalone_docs_json_and_render_items
from ._json_encoder import serialize_json

plot_objects = [obj]
plot_objects = _check_plot_objects(plot_objects)
(docs_json, render_items) = _standalone_docs_json_and_render_items(plot_objects)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These underscore-prefixed functions aren't really meant to leave embed.py; it might be nice to add an embed.py function specifically for this, standalone_docs_json_for_png(plot_objects) which would essentially contain the above three lines.


script = join(dirname(dirname(__file__)), "bokehjs", "render.js")
cmd = "node %s --filename=%s --log-level=%s" % (script, filename, resources.log_level)

proc = Popen(cmd, stdin=PIPE, shell=True)
proc.communicate(input=serialize_json(docs_json))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you need our serialize_json or special _json_encoder here, because docs_json is plain JSON, not "weird JSON with PlotObject and numpy stuff in it". Simply json.dumps should work fine unless I'm missing something.


# this function exists mostly to be mocked in tests
def _push_to_server(websocket_url, document, session_id, io_loop):
session = push_session(document, session_id=session_id, url=websocket_url, io_loop=io_loop)
Expand Down
4 changes: 2 additions & 2 deletions bokeh/plotting.py
Expand Up @@ -14,8 +14,8 @@
from .document import Document
from .models import ColumnDataSource
from .io import (
curdoc, curstate, output_file, output_notebook, output_server, push,
reset_output, save, show, gridplot, hplot, vplot)
curdoc, curstate, output_file, output_png, output_notebook, output_server,
push, reset_output, save, show, gridplot, hplot, vplot)

# Names that we want in this namespace (fool pyflakes)
(GridPlot, Document, ColumnDataSource, gridplot,
Expand Down
14 changes: 14 additions & 0 deletions bokeh/state.py
Expand Up @@ -102,6 +102,10 @@ def document(self, doc):
def file(self):
return self._file

@property
def png(self):
return self._png

@property
def notebook(self):
return self._notebook
Expand All @@ -128,6 +132,7 @@ def autopush(self):

def _reset_keeping_doc(self):
self._file = None
self._png = None
self._notebook = False
self._session_id = None
self._server_url = None
Expand Down Expand Up @@ -193,6 +198,15 @@ def output_file(self, filename, title="Bokeh Plot", autosave=False, mode="inline
if os.path.isfile(filename):
logger.info("Session output file '%s' already exists, will be overwritten." % filename)

def output_png(self, filename, autosave=False):
self._png = {
'filename' : filename,
}
self._autosave = autosave

if os.path.isfile(filename):
logger.info("Session output file '%s' already exists, will be overwritten." % filename)

def output_notebook(self):
"""Generate output in Jupyter/IPython notebook cells.

Expand Down
7 changes: 6 additions & 1 deletion bokehjs/package.json
Expand Up @@ -56,7 +56,12 @@
"shasum": "^1.0.1",
"root-require": "^0.3.1",
"event-stream": "^3.3.1",
"jsdom": "^5.6.1",
"jsdom": "^7.0.2",
"canvas": "^1.3.0",
"location": "^0.0.1",
"navigator": "^1.0.1",
"htmlparser2": "^3.8.3",
"uuid": "^2.0.1",
"websocket":"^1.0.22"
},
"browserify": {
Expand Down