-
Notifications
You must be signed in to change notification settings - Fork 9
DevelopingDiamondWebApps
With the JSON Blaster introduced in OpenDiamond 8, it is straightforward to integrate support for Diamond searches into a new or existing web application. The existing Diamond infrastructure remains largely unchanged; there must still be a scope server, one or more dataretrievers, and one or more diamondds. (Of course, the scope server and/or dataretriever components may be integrated into your web application.) To support your web applications, you will also need to configure one JSON Blaster alongside your Diamond servers; its job is to present your cluster of Diamond servers as a web service. Your web application will be responsible for the user interface allowing users to construct searches and view search results.
An example web application is available in the webapp directory of the diamond-example Git repository.
- When ready to start a search, your application submits a
SearchConfigrequest to the JSON Blaster. This request specifies the scope cookies, filters, and filter parameters to be used. The Blaster responds with aSearchConfigResultgiving the search URL and search key.
Filter code and blob arguments cannot be embedded in the SearchConfig JSON object. If possible, these objects should be made available at resolvable URLs specified in the SearchConfig object. The Blaster will fetch the blobs from these URLs when configuring the search. If this is not possible, your application can POST blob data to the JSON Blaster's /blob endpoint (one blob per POST). The Blaster's response will include a Location header containing the URI to be included in the SearchConfig object.
-
The web application sends the search URL and key to the client browser.
-
The browser connects directly to the search URL using SockJS and passes in the search key. Diamond provides a JavaScript library that simplifies this process. The browser will receive a JavaScript callback when a new search result becomes available, when an error occurs, and with periodic search statistics.
Result objects contain Diamond attribute data for a particular result. When an attribute can be encoded in JSON, it is included directly in the result object. For other attributes, such as images, two URLs are included instead: one for the raw (unmodified) attribute data, and one for the attribute data in an image format likely to be displayable in a browser. (The difference arises when an attribute contains e.g. a PPM image.)
Result objects also contain a URL, stored under the _ResultURL attribute, which can be used to re-fetch the result object at a later time. This is useful, for example, if your web application needs to provide a separate page containing a "detail view" of a particular search result.
The scope cookie specification has been extended to include a Blaster header that specifies the root URL of the JSON Blaster to be used. It is strongly recommended that you use the JSON Blaster specified in the scope cookie rather than hardcoding a URL into your web application. Existing scope servers should be modified to include a Blaster URL in their scope cookies.
Each scope cookie within a megacookie can specify a different Blaster URL. In this case, the web application should start multiple searches, one through each Blaster. Each Blaster should receive only the subset of scope cookies that name it. The browser should accordingly create multiple search connections, one for each Blaster.
Web applications written in Python can use the OpenDiamond library to determine which cookies within a megacookie should be sent to each Blaster:
from opendiamond.scope import ScopeCookie, get_blaster_map
parsed_cookies = [ScopeCookie.parse(c)
for c in ScopeCookie.split(megacookie)]
blaster_to_cookies = get_blaster_map(parsed_cookies)
blaster_to_encoded_megacookie = dict(
(b, ''.join(c.encode() for c in clist))
for b, clist in blaster_to_cookies.iteritems()
)
- /
- The root URL of the JSON Blaster. Search requests should be POSTed here.
- /blob
- If the web application is unable to provide its filters and blob arguments via resolvable URLs, they should be POSTed here.
All other endpoints should be obtained from JSON Blaster responses.
The message schemas are defined by the opendiamond.blaster.json Python module. To obtain a list of available schemas, execute:
python -m opendiamond.blaster.json
Then, to obtain a textual representation of a particular schema in JSON Schema draft 3 format, use a command like the following:
python -m opendiamond.blaster.json SchemaClassName strict
For messages that will be sent by your application, the last argument should be strict. For messages received by your application, the last argument should be permissive.
OpenDiamond ships with two helper libraries: json-blaster.js and json-blaster-autopause.js.
json-blaster.js requires SockJS and provides a thin, Diamond-specific wrapper around it. The browser can start a configured search with:
var blaster_socket = new JSONBlasterSocket(socket_url, search_key);
Methods on blaster_socket allow you to pause, resume, and stop the search and to configure event callbacks. At a minimum, you will want to configure a result callback, but there are others that may be useful:
blaster_socket.onopen(function() {
// Runs when the connection is established
});
blaster_socket.onclose(function(data) {
// Runs when the connection is terminated
});
blaster_socket.on('search_started', function(data) {
// Runs when the search has started. Primarily useful for obtaining
// the search ID for logging.
});
blaster_socket.on('result', function(data) {
// Runs when a search result has been produced.
});
blaster_socket.on('statistics', function(data) {
// Runs periodically (about once a second) with newly-updated search
// statistics.
});
blaster_socket.on('search_complete', function(data) {
// Runs when the search has finished processing the objects in scope.
// After firing this event, the Blaster will close the connection.
});
blaster_socket.on('error', function(data) {
// Run when the Blaster wants to report an error.
});
See the ServerToClientEvent JSON schema for the attributes of the data argument for each event callback.
json-blaster-autopause.js implements automatic pausing of the search result stream, and requires jQuery. Without some sort of pause mechanism, a long-running search will stream results into the browser window indefinitely, eventually running the browser out of memory. Autopause works by tracking which result objects have been visible to the user, and pausing the search when a certain number of undisplayed objects (by default, 50) have accumulated. When the number of undisplayed objects again drops below that threshold, the search is automatically resumed.
Note that the pause mechanism is not precise: once the search has been paused, the JSON Blaster may continue to send objects for an indefinite period of time. (Each diamondd has the opportunity to send one additional object after the pause request has been processed.)
To initialize:
var autopause = new AutoPause(blaster_socket_or_list_of_sockets);
Then, in your result handler, after you have created an HTML element representing the result object:
autopause.element_added(element);
That's it! Autopause will take care of the rest.
The JSON Blaster also supports executing a search against an object provided by the web application. (This can be used to determine what the filters would do with an example object provided by the user.) To do this, the application should POST an EvaluateRequest object to the evaluate_url returned in the SearchConfigResponse. The object data is specified by the same mechanisms as for filters and blob arguments in a SearchConfig request. The Blaster will respond with a ResultObject.
The JSON Blaster has special support for rendering heat maps. The image_url attribute endpoint takes an optional query parameter, tint, specifying a HTML hex color with which to tint the image. 100% black pixels in the image will be converted to fully transparent; 100% white pixels will be converted to the specified color, and pixels in between will have partial transparency. The resulting image can be overlaid on top of the original image within the browser. The CSS opacity property can be used to further increase the transparency of the overlay.
SockJS handles cross-origin issues automatically, so the result stream does not require any special handling. However, support for scripted cross-origin requests to the _ResultURL and attribute URL endpoints varies by browser.
Attribute URLs only have cross-origin concerns when the application attempts to read a binary attribute with XMLHttpRequest or import an image attribute into an HTML canvas element. The attribute URL endpoints set the Access-Control-Allow-Origin: * header, so sufficiently-recent browsers should allow these actions (including pixel-level access to a canvas after a result image is added).
The _ResultURL endpoint also sets Access-Control-Allow-Origin. For older browsers that don't support this header, the endpoint also supports JSONP via the jsonp query parameter. Note that Internet Explorer 8 and 9 do not support Access-Control-Allow-Origin via XMLHttpRequest; you must use a separate XDomainRequest object. If this is unacceptable (e.g., jQuery doesn't support it), you can fall back to JSONP.