Writing a Plugin

This guide is for somebody who:

1. Has already developed a standalone program;
2. Wants to display something within Overview proper; and
3. Knows how to set up a web server (in any language).

A change of perspective

A Plugin is deceptively simple. All you need to do is think about it.

Think about the sample code generator. You've already seen it. It's a Plugin.

Look at its source code.

Here come some mind benders. Absorb them:

• A Plugin is a web server.
• The person who calls the web server is the user. (It's an iframe in the Overview web page.)
• The user requests /metadata with no arguments to verify that a Plugin URL is valid.
• When the user creates a View that refers to your Plugin, he or she will request /show with origin, server, documentSetId and apiToken in the query string. The response will go into the iframe.
• That is all.

A Plugin needn't use the Overview API. It needn't use cookies. It needn't worry about authentication. Heck, you can upload a couple of flat files to S3 and call that a Plugin.

In sum: a Plugin is an website that is invoked in an iframe.

Data and Messages

Overview's web page includes your plugin in an iframe. Great. But what does your plugin do?

Most plugins provide a means of searching. They'll want to index documents and to help Overview search them. They may store plugin-specific data within their own databases (using apiToken as a unique key), or they may read and/or write Overview's metadata, tag or store structures.

The plugin will need to communicate with its own server and with Overview. That means passing messages along one of several channels:

• The plugin iframe can send and receive messages from its parent frame (Overview's main page) using window.postMessage(). For instance, the plugin can tell Overview's main page to change its search parameters. These calls are fire-and-forget: they don't have return values and the plugin can't detect errors in Overview's main page. Messages go both ways: Overview's main page will notify the plugin iframe when the user changes documents or search parameters. This is the best way to communicate with Overview (because the main page will show the user feedback if you, say, edit the current document's metadata) -- but it has the fewest methods (for instance, it can't stream documents) and the most restrictions (for instance, you can't detect errors with it). See Accessing the main page from a Plugin for instructions.
• The plugin iframe can make HTTPS requests to Overview's API. For instance, it can write document metadata or stream lists of documents. In the plugin's JavaScript, (new URL(document.location)).searchParams.get('origin') is the URL of Overview's API server, and you'll need to set an Authorization header of 'Basic ' + window.btoa((new URL(document.location)).searchParams.get('apiToken') + ':x-auth-token') with every request. This is particularly useful when your HTTP plugin is a static website.
• The plugin iframe can make HTTPS requests to the plugin server. This is the most flexible sort of request: the server can format data the way the plugin JavaScript wants it. In the plugin's JavaScript, window.origin will be the URL of the plugin server. Your request should probably include apiToken, server and documentSetId (which are in (new URL(document.location).searchParams) so the server can communicate with Overview's API.
• The plugin server can make HTTPS requests to Overview's API. From the plugin server, request.params.server is the URL of Overview's API server, and request.params.documentSetId and request.params.apiToken will identify the document set.

Avoid these misconceptions:

• origin and server aren't interchangeable. origin is the window.location.origin of the plugin parent's iframe, which is the URL of the Overview server from the perspective of the user. server is the URL of the Overview server from the perspective of the plugin server. In development mode, the user will view Overview at http://localhost:9000 (that's the origin) and the plugin will view Overview at http://overview-web within a Docker network (that's the server).
• documentSetId does not uniquely describe a document set. Only (server, documentSetId) uniquely describe a document set. Your plugin server may be used by different instances of Overview.

Using a web framework

Let's pretend Django is your favorite web framework. (I, the author, chose Django because I have never used it and I want to prove you can use any framework.)

First, we start a web server according to the Django tutorial. We don't need a database, which makes things easier:

django-admin.py startproject my_plugin
cd my_plugin/my_plugin
edit settings.py
  # (optional) Delete all INSTALLED_APPS except for django.contrib.staticfiles and my_plugin
  # (optional) Delete all MIDDLEWARE_CLASSES
  # (optional) Delete all DATABASES
edit urls.py
  # Add these patterns:
  # url(r'^metadata$', 'my_plugin.views.metadata', name='metadata'),
  # url(r'^show$', 'my_plugin.views.show', name='show'),
cd ..
python3 manage.py runserver

Now write my_plugin/my_plugin/views.py:

from django.http import HttpResponse
from django.shortcuts import render

def metadata(request):
    response = HttpResponse('ok')
    response['Access-Control-Allow-Origin'] = '*'
    return response

def show(request):
    return render(request, 'show.html', { 'request': request })

Then create my_plugin/my_plugin/templates and write my_plugin/my_plugin/templates/show.html:

<!doctype html>
<html>
  <head>
    <title>My Plugin</title>
  </head>
  <body>
    <p>This is my view app.</p>
    <p>My parent is <strong>{{ request.GET.server }}</strong>.</p>
  </body>
</html>

You're done! http://localhost:8000 is a Plugin, and it's up and running.

Trying it out

1. Browse to http://localhost:9000.
2. Open a document set.
3. Click "New View / Custom..."; enter a title and http://localhost:8000 as the URL; click through the SSL warning; click "Create View".
4. Look in the frame.

Here's what happens:

• When you enter the URL and click through the SSL warning, your browser verifies that it can access http://localhost:8000. (This is why you need to add the Access-Control-Allow-Origin header.)
• When you submit the form, Overview verifies that it can access http://localhost:8000. (Advanced features may require this.)
• Overview creates a new API token with access to the document set.
• Your browser opens an iframe to http://localhost:8000/show?server=http://localhost:9000&documentSetId=ID&apiToken=TOKEN
• Your Plugin serves the show template.

Developing

Easier refresh

You may prefer to open the iframe in its own tab for some stages of development. That way, Refresh will be faster.

API calls

Let's say you're a Django fiend and you want to use it to access the Overview API. No problem!

First, pip3 install requests to get a decent request library. Then change your show method in views.py:

import requests
...

def show(request):
    # URL to list documents
    url = '%s/api/v1/document-sets/%s/documents' % (
        request.GET['server'],
        request.GET['documentSetId'],
    )
    r = requests.get(url, auth=(request.GET['apiToken'], 'x-auth-token'))
    documents = r.json()
    return render(request, 'show.html', { 'documents': documents['items'] })

And change the body of your templates/show.html:

<p>This is my plugin. It can show my documents:</p>
<ul>
  {% for document in documents %}
    <li>{{ document.title }}</li>
  {% endfor %}
</ul>

Refresh the page, and you'll see your documents in the app.

Deploying

Deploy your Plugin as you would deploy any website. Be sure it's accessible both to external users and to the Overview web server.

Alternate approach: flat files

If you use a web framework, you need to deploy it. That means it's your server using bandwidth, processing power and memory. If you get a spike in users, it's your problem.

The easiest web server to set up is one without server-side logic. Just host flat files!

You need two files:

• metadata, which may be empty. Be sure your web server sends the Access-Control-Allow-Origin: * header.
• show, which uses JavaScript to parse the query string, call the Overview API, and so forth.

An example with Gulp, Jade, Less, CoffeeScript and Browserify

It can be tedious to write HTML, CSS and JavaScript by hand. Many developers prefer to use tools to make it easier. Look at an example app to see how it's done.

Deploying

Flat-file Apps can scale to everybody on the Internet. Upload all your files to whatever file server you're comfortable with.

Be sure your file server adds the header, Access-Control-Allow-Origin: * when responding to .../metadata.

Advantages

Flat-file Plugins are superior to entire web servers because:

• They cut out a source of errors. No server means no server failures.
• They scale better. You can jump from one user to a million users without any slowdown.
• They're easier to deploy.

Limitations

Flat files can't do everything. In particular, they have trouble with:

• Background tasks. You can run long-running tasks in the browser, but they stop as soon as the user leaves the page.
• Programming languages. If you can't abide JavaScript, you may not enjoy using flat files.
• Queries from the API. For some advanced features (e.g., custom selections), the API server needs to query your Plugin. You can't use flat files for that.
• Cross-domain requests. Web browsers won't let you do requests to web services that don't allow them. (This probably won't affect you, and if it does you can always build a proxy server in the future to work around the problem.)
• Bandwidth. If you use flat files, the client must talk directly with the API; that means the client must pay for the bandwidth. If your App needs to process the entire text of a 1GB document set, that'll take over 13 minutes on a 10mbps connection; compare that to 8 seconds for a server-side app hosted on Amazon EC2 in the us-east-1 region. Aside from time, big uploads and downloads will cost users money, slow down their other programs, and fail frequently.
• Optimization. Text processing can take lots of CPU and memory; if you need to optimize your code, you'll find a web browser more limiting than other programming environments.
• Compatibility. Any code you put in flat files must work on IE10, Firefox, Chrome and Safari.

Despite these limitations, we recommend that you use flat files whenever possible. If you run into one of the above-mentioned problems, you can create a server at that point and use it to complement the client.

Why? Well, it's a bit self-serving, but ... we'd love to host your Plugins for you, so all our users can benefit from them. With flat files, hosting is a no-brainer.