Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

node.js integration and static image generation #3047

Closed
wants to merge 31 commits into from
Closed
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
15401cc
Fix HTML5 canvas context's properties and methods only once
mattpap Oct 26, 2015
de7cc11
Resize HTML5 Canvas instance manually in node.js (jsdom)
mattpap Oct 28, 2015
a2c053f
Upgrade jsdom to version 7*
mattpap Oct 29, 2015
b67b80c
Add canvas, htmlparser2 and uuid to package.json
mattpap Oct 29, 2015
96fcc68
Allow to load models from a <script type="text/x-bokeh"> element
mattpap Oct 29, 2015
af8c682
Store serialized models in a <script type='text/x-bokeh'> element
mattpap Oct 29, 2015
c16d186
Make bokeh.js a node.js module, so we can use bokehRequire()
mattpap Oct 29, 2015
75657c1
Use window.Image for symmetry with window.Canvas
mattpap Oct 29, 2015
dc5aa82
Preliminary and very naive approach to render:done event
mattpap Oct 29, 2015
019fcd5
Use toLowerCase() together with element.nodeName
mattpap Oct 29, 2015
e756bcc
Preliminary offline rendering script for node.js
mattpap Oct 29, 2015
a839346
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Nov 2, 2015
eeb3871
Add scikit-learn to conda package under test/examples (#3041)
mattpap Nov 2, 2015
babd570
Add location and navigator packages to package.json
mattpap Nov 2, 2015
a6b60c0
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Nov 11, 2015
3ffc50a
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Nov 11, 2015
4a1af80
Update render.js to the new code base
mattpap Nov 15, 2015
5fad6dd
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Nov 15, 2015
244b979
Restructure render.js and allow stdin input
mattpap Nov 16, 2015
ae9cb9f
Make log level and JS/CSS paths configurable
mattpap Nov 16, 2015
851cc93
Make custom models be part of the JSON protocol
mattpap Nov 18, 2015
6e51490
Restore pretty (dev) output from serialize_json()
mattpap Nov 18, 2015
468c775
Preliminary plotting API for PNG output
mattpap Nov 18, 2015
7f69c3c
Add examples/plotting/png/bollinger.py
mattpap Nov 18, 2015
3eaf4a0
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Nov 22, 2015
1b9e5b5
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Nov 26, 2015
a60b37c
Separate node.js integration from bokehjs
mattpap Nov 26, 2015
df12d4e
Change _check_plot_objects -> _check_one_model after merging master
mattpap Nov 26, 2015
6e3dfc5
Ignore *.png files under examples/plotting/png
mattpap Nov 26, 2015
1f4e178
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Nov 27, 2015
cc6077b
Merge remote-tracking branch 'origin/master' into mattpap/1589_nodejs
mattpap Dec 8, 2015
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 5 additions & 0 deletions bokeh/_templates/script_tag.html
Expand Up @@ -5,6 +5,11 @@
:type js_code: str

#}
{%- if docs_json_id -%}
<script id="{{ docs_json_id }}" type="text/x-bokeh">
{{ docs_json|indent(4) }}
</script>
{% endif -%}
<script type="text/javascript">
{{ js_code }}
</script>
21 changes: 12 additions & 9 deletions bokeh/embed.py
Expand Up @@ -409,18 +409,21 @@ def _script_for_render_items(docs_json, render_items, websocket_url,
if (custom_models is not None) and len(custom_models) == 0:
custom_models = None

plot_js = _wrap_in_function(
DOC_JS.render(
custom_models=custom_models,
websocket_url=websocket_url,
docs_json=serialize_json(docs_json),
render_items=serialize_json(render_items)
def plot_js(docs_json):
return _wrap_in_function(
DOC_JS.render(
custom_models=custom_models,
websocket_url=websocket_url,
docs_json=serialize_json(docs_json),
render_items=serialize_json(render_items),
)
)
)

if wrap_script:
return SCRIPT_TAG.render(js_code=plot_js)
id = str(uuid.uuid4())
return SCRIPT_TAG.render(docs_json_id=id, docs_json=serialize_json(docs_json), js_code=plot_js(id))
else:
return plot_js
return plot_js(docs_json)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this refactor is incomplete and we end up with one function that really does two separate things. As it is, the function is named and expected elsewhere to return one hunk of script, which is optionally wrapped in one <script> tag. If we want to change it so that to embed render items, you need N script tags, we should rename the function _scripts_for_render_items, return a list, and have all callers handle the list result.

wrap_script should control whether each element is wrapped in a <script> tag, not how many elements there are. Though with docs_json in its own tag I think we have to always wrap_script so that parameter would maybe have to go.

The simplest thing for this PR is probably to drop the "externalize docs json into its own <script> tag" and do that as its own PR, but if you think externalizing the docs json is important to this PR, I think we should finish the refactor - do it always/consistently. There's no value I see to having two ways to do this: either externalize the docs_json into tags, or don't.

Going from one script to a list of scripts will break the components() API I think. If we don't want to break components() I'm not sure we should do this at all, is it really worth two codepaths?

If we do have two codepaths, I'd rather not overload wrap_script - it refers to whether to wrap the scripts in a script tag, which is separate from whether the docs_json is in its own script tag. Maybe a separate function _scripts_for_render_items_with_externalized_json ?


def _html_page_for_render_items(resources, docs_json, render_items, title, websocket_url,
custom_models, js_resources=None, css_resources=None,
Expand Down
7 changes: 6 additions & 1 deletion bokehjs/package.json
Expand Up @@ -56,7 +56,12 @@
"shasum": "^1.0.1",
"root-require": "^0.3.1",
"event-stream": "^3.3.1",
"jsdom": "^5.6.1",
"jsdom": "^7.0.2",
"canvas": "^1.3.0",
"location": "^0.0.1",
"navigator": "^1.0.1",
"htmlparser2": "^3.8.3",
"uuid": "^2.0.1",
"websocket":"^1.0.22"
},
"browserify": {
Expand Down
93 changes: 93 additions & 0 deletions bokehjs/render.js
@@ -0,0 +1,93 @@
var fs = require("fs");
var path = require("path");
var uuid = require("uuid");
var argv = require("yargs").argv;
var jsdom = require("jsdom");
var htmlparser2 = require("htmlparser2");

var all_models;
var file = argv._[0];
var ext = path.extname(file);
var basename = path.basename(file, ext);
var dirname = path.dirname(file);

switch (ext) {
case ".html":
var all_texts = [];
var all_text = null;
var parser = new htmlparser2.Parser({
onopentag: function(name, attrs) {
if (name == "script" && attrs.type == "text/x-bokeh") {
all_text = "";
}
},
ontext: function(text) {
if (all_text !== null) {
all_text += text;
}
},
onclosetag: function(name) {
if (name == "script" && all_text !== null) {
all_texts.push(all_text);
all_text = null;
}
}
});
parser.write(fs.readFileSync(file));
parser.end();
switch (all_texts.length) {
case 0:
throw new Error("no 'text/x-bokeh' sections found");
break;
case 1:
all_models = JSON.parse(all_texts[0]);
break;
default:
throw new Error("too many 'text/x-bokeh' sections");
}
break;
case ".json":
all_models = require(file);
break;
default:
throw new Error("expected an HTML or JSON file");
}

global.document = jsdom.jsdom();
global.window = document.defaultView;
global.location = require("location");
global.navigator = require("navigator");
global.window.Canvas = require("canvas");
global.window.Image = global.window.Canvas.Image;

global.bokehRequire = require("./build/js/bokeh.js").bokehRequire;
require("./build/js/bokeh-widgets.js");

var Bokeh = global.window.Bokeh;

var head = document.getElementsByTagName('head')[0];
var link = document.createElement('link');
link.rel = 'stylesheet';
link.href = './build/css/bokeh.css';
head.appendChild(link);

Bokeh.set_log_level("debug");
Bokeh.load_models(all_models);

Bokeh.Collections("PlotContext").each(function(model) {
var el = document.createElement("div");
el.setAttribute("class", "plotdiv");
document.body.appendChild(el);

Bokeh.Events.on("render:done", function(plot_view) {
var nodeCanvas = plot_view.canvas_view.canvas[0]._nodeCanvas;
var name = basename + "-" + plot_view.model.id + ".png";
var outfile = path.join(dirname, name)
Bokeh.logger.info("writing " + outfile);
var out = fs.createWriteStream(outfile);
nodeCanvas.pngStream().on('data', function(chunk) { out.write(chunk); });
});

var view = new model.default_view({model: model, el: el});
Bokeh.index[model.id] = view;
});
33 changes: 20 additions & 13 deletions bokehjs/src/coffee/common/canvas.coffee
Expand Up @@ -31,9 +31,7 @@ class CanvasView extends ContinuumView
@canvas_overlay = @$('div.bk-canvas-overlays')
@map_div = @$('div.bk-canvas-map') ? null

# Create context. This is the object that gets passed arount while drawing
@ctx = @canvas[0].getContext('2d')
@ctx.glcanvas = null # init without webgl support (can be overriden in plot.coffee)
@ctx = @_get_context(@canvas[0])

logger.debug("CanvasView initialized")

Expand Down Expand Up @@ -65,17 +63,26 @@ class CanvasView extends ContinuumView
@canvas_events.attr('style', "z-index:100; position:absolute; top:0; left:0; width:#{width}px; height:#{height}px;")
@canvas_overlay.attr('style', "z-index:75; position:absolute; top:0; left:0; width:#{width}px; height:#{height}px;")

if @canvas[0]._nodeCanvas?
@canvas[0]._nodeCanvas = new window.Canvas(width*ratio, height*ratio)
@ctx = @_get_context(@canvas[0])

@ctx.scale(ratio, ratio)
@ctx.translate(0.5, 0.5)

@model.new_bounds = false

_get_context: (canvas) ->
ctx = canvas.getContext('2d')
ctx.glcanvas = null # init without webgl support (can be overriden in plot.coffee)

# work around canvas incompatibilities
# todo: this is done ON EACH DRAW, is that intended?
@_fixup_line_dash(@ctx)
@_fixup_line_dash_offset(@ctx)
@_fixup_image_smoothing(@ctx)
@_fixup_measure_text(@ctx)
@_fixup_line_dash(ctx)
@_fixup_line_dash_offset(ctx)
@_fixup_image_smoothing(ctx)
@_fixup_measure_text(ctx)

@model.new_bounds = false
ctx

_fixup_line_dash: (ctx) ->
if (!ctx.setLineDash)
Expand All @@ -96,10 +103,10 @@ class CanvasView extends ContinuumView

_fixup_image_smoothing: (ctx) ->
ctx.setImageSmoothingEnabled = (value) ->
ctx.imageSmoothingEnabled = value;
ctx.mozImageSmoothingEnabled = value;
ctx.oImageSmoothingEnabled = value;
ctx.webkitImageSmoothingEnabled = value;
ctx.imageSmoothingEnabled = value
ctx.mozImageSmoothingEnabled = value
ctx.oImageSmoothingEnabled = value
ctx.webkitImageSmoothingEnabled = value
ctx.getImageSmoothingEnabled = () ->
return ctx.imageSmoothingEnabled ? true

Expand Down
14 changes: 13 additions & 1 deletion bokehjs/src/coffee/common/embed.coffee
Expand Up @@ -117,7 +117,19 @@ fill_render_item_from_script_tag = (script, item) ->

logger.info("Will inject Bokeh script tag with params #{JSON.stringify(item)}")

embed_items = (docs_json, render_items, websocket_url) ->
embed_items = (docs_json_or_id, render_items, websocket_url) ->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nicer perhaps if we either always do it the "embed the doc as a script tag" way or the "embed the doc as JSON" way, rather than sometimes doing one and sometimes doing the other.

It looks like you added the "embed doc as a script" way in order to scrape a document out of an HTML file?

If so I would say we should always go from Application to PNG/SVG, rather than scrape-json-from-html-file to PNG/SVG. The Application could be spelled as a JSON file, which gives you the JSON case (add application/spellings/json.py which does Document.from_json_string and you have that).

Command line implications:

  • bokeh json myapp.py - dumps myapp.py as JSON (creates app document and saves document.to_json_string())
  • bokeh html myapp.json - sticks the JSON in an HTML file (needs a json spelling handler)
  • bokeh png myapp.py, bokeh png myapp.json - output the app from whatever format to PNG

I know you aren't done with the PR so maybe you've already gone in another direction.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like you added the "embed doc as a script" way in order to scrape a document out of an HTML file?

This is a secondary goal (thought useful for testing), but really I wanted to externalize the JSON for quite some time already. This is not the final form, because of render_items and custom models.

(...) rather than scrape-json-from-html-file (...)

Obviously, but as I said, the current approach is good for testing. I'm currently working on the proper API/CLI.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we end up with both externalized docs_json and in-script docs_json, let's make two separate functions here in Bokeh.embed rather than type-introspecting the parameter...

if _.isString(docs_json_or_id)
element = document.getElementById(docs_json_or_id)
if not element
throw new Error("element ##{docs_json_or_id} not found")
if element.nodeName.toLowerCase() != "script" or element.getAttribute("type") != "text/x-bokeh"
throw new Error("element ##{docs_json_or_id} must be a script with type='text/x-bokeh'")
docs_json = JSON.parse(element.innerHTML)
else if _.isObject(docs_json_or_id)
docs_json = docs_json_or_id
else
throw new Error("expected a string identifier or an object")

if websocket_url?
set_websocket_url(websocket_url)

Expand Down
7 changes: 7 additions & 0 deletions bokehjs/src/coffee/common/events.coffee
@@ -0,0 +1,7 @@
Backbone = require "backbone"

class Events extends Backbone.Events

module.exports = {
Events: Events
}
3 changes: 3 additions & 0 deletions bokehjs/src/coffee/common/plot.coffee
Expand Up @@ -17,6 +17,7 @@ Solver = require "./solver"
ToolManager = require "./tool_manager"
plot_template = require "./plot_template"
properties = require "./properties"
{Events} = require "./events"


# Notes on WebGL support:
Expand Down Expand Up @@ -359,6 +360,8 @@ class PlotView extends ContinuumView
# TODO - This should only be on in testing
# @$el.find('canvas').attr('data-hash', ctx.hash());

Events.trigger("render:done", this)

resize: () =>
@resize_width_height(true, false)

Expand Down
1 change: 1 addition & 0 deletions bokehjs/src/coffee/main.coffee
Expand Up @@ -24,6 +24,7 @@ Bokeh.index = require("./common/base").index
# common
Bokeh.Collections = require("./common/base").Collections
Bokeh.Config = require("./common/base").Config
Bokeh.Events = require("./common/events").Events
Bokeh.Document = require("./common/document").Document
Bokeh.CartesianFrame = require("./common/cartesian_frame")
Bokeh.Canvas = require("./common/canvas")
Expand Down
2 changes: 1 addition & 1 deletion bokehjs/src/coffee/renderer/glyph/image_url.coffee
Expand Up @@ -10,7 +10,7 @@ class ImageURLView extends Glyph.View
@image = (null for img in @url)

for i in [0...@url.length]
img = new Image()
img = new window.Image()
img.onload = do (img, i) =>
return () =>
@image[i] = img
Expand Down
4 changes: 4 additions & 0 deletions bokehjs/src/js/prelude.js
Expand Up @@ -21,5 +21,9 @@
newRequire(entry[i]);
}

if (typeof exports !== "undefined") {
exports.bokehRequire = newRequire;
}

return newRequire;
})
1 change: 1 addition & 0 deletions conda.recipe/meta.yaml
Expand Up @@ -65,6 +65,7 @@ test:
- ipython-notebook
- matplotlib
- sympy
- scikit-learn
- ggplot
- seaborn
- icalendar
Expand Down