-
Notifications
You must be signed in to change notification settings - Fork 0
Home
Ziziphus is an application for editing documents in VRA Core 4 XML. It was developed in a co-operation between eXist Solutions and the Heidelberg Research Architecture at the Heidelberg Centre for Transcultural Studies (HCTS) the University of Heidelberg.
After several years of development and usage in practice an architectural review was conducted to verify the approach.
As an outcome of this review a refresh of some of the base technologies was decided. The results are described in this document.
To point out the differences, the old system will be introduced on a high-level. Further, the motivation for the changes will be presented.
N.b.: For the remainder of this document the terms 'Ziziphus' or '1.0' refers to the 'old' system while 'prototype' or 'Ziziphus 2' will denote the new application.
The Ziziphus project was started at Heidelberg to allow editing of VRA Core 4 XML records.
The VRA Core 4 XML Schema is hosted by the Library of Congress and maintained by the Visual Resources Association. The XML Schema defines a set of metadata for "the description of images and works of art and culture". One or more VRA records can then be used to describe a single visual resource, e.g. multiple depictions in VRA image records are related to one object in a VRA work record. In practice these data come encoded in XML.
The first implementation of Ziziphus uses a browser-based UI to talk to eXist-db as the backend. eXist-db is a native XML database and stores, indexes and queries the VRA records for display in the browser.
For the UI part W3C XForms was used to provide a form-based editing of the records. The choice was not accidental - one of goals of the project was to use as many open standards as possible. With XML, XQuery, HTML and XForms the whole project relies on W3C standards.
The key feature of Ziziphus is its generative approach to the UI. All front-end views are generated from the VRA XML Schema with the help of a set of XSLT transformations. As XForms are XML documents themselves they can easily be created the same way.
This worked fine and allowed even to use slightly different XML Schemas that added some extensions developed at the University. The whole UI can be re-created with a single command to switch back and forth between different Schema variants.
However XML Schema was not designed to describe user interfaces and an additional mechanism was needed to sort the raw data and to refine the display of certain data items.
An example: a element in the data should be represented as a textarea in the UI but there's no information in the Schema that tells us that this may be a longer text.
Therefore, we created a UI Definition to solve this issue. An initial UI Definition file is created from the VRA Schema with each run of the UI generator. This is a boiled-down version of the XML Schema that just keeps element names, cardinalities and parent/child relationships.
A snippet from the UI Definition:
<set name="vra:descriptionSet" idref="b-d2e466">
<group name="vra:description" repeated="true" idref="b-d2e511">
<node name="@type" detail="false" idref="b-d2e613"/>
<node name="vra:text" detail="false" control="textarea" idref="b-d2e517"/>
<group name="vra:author" repeated="true" idref="b-d2e538">
<group name="vra:name" idref="b-d2e541">
<textNode detail="false" control="autocomplete" queryType="names" idref="b-d2e11"/>
<node name="@type" detail="false" idref="b-d2e548"/>
</group>
<group name="vra:role" idref="b-d2e563">
<node name="@type" detail="false" hidden="true" idref="b-d2e570"/>
<textNode detail="false" control="select1" code-table="role" idref="b-d2e11"/>
</group>
</group>
This file is auto-generated but allows manual editing. E.g. by adding control="textarea" it tells the generator to output a textarea instead of the default input control.
Further, the group and node elements can be reordered for improved usability. Nodes can even be completely hidden from the view.
Due to the simple structure change management for the definition file is manageable: the auto-generated file and the edited version are kept in different places and can easily be merged should a Schema change take place.
In sum, the generative approach allowed to build the application in much less time than having to hand-code the forms.
Ziziphus (old and new) are implemented as eXist-db applications. That means, the whole source-tree is packaged in a single '.xar' file, which is simply a zip-file with a custom file extension.
A typical xar application will contain XQuery scripts, HTML pages, resources like CSS, JavaScript and images, and XML data. Sometimes, the data are split into their own package to keep them independent of the the application. This was done in Ziziphus.
Application packages are then uploaded to the database for installation and will be available right away from the browser.
As mentioned before, Ziziphus makes heavy use of open standards. One reason is sustainability and usually broad availability of tools and knowledge.
Unfortunately, using standards does not automatically mean that nothing changes. In case of Ziziphus, the decision for using XForms was made some years ago, when the XForms standard was still under active maintenance. However, as time went by, the situation changed. It became increasingly clear that XForms was on the decline for several reasons:
First of all, it wasn't picked up by the browser vendors and therefore it didn't get much attention in the web development world.
Though being extremely powerful, XForms is also not easy to master for a beginner which makes it much harder for a software professional looking for a forms solution. As a result, XForms never found broad acceptance though it is used very successfully in certain domains.
Eventually, during the development of XForms 2.0, the W3C Working Group was discontinued.
In lack of a valid business model the XForms implementation 'betterFORM' that is used by Ziziphus stopped its active development.
All these factors led the project to reconsider Ziziphus' reliance on XForms and to look for an alternative.
In order to escape the XForms trap a new technology for editing XML in the browser was needed. This already factors out the many JavaScript frameworks that are around today as their lifetime is often just a few years before another framework takes over the marketplace.
Thus, just an open standard with broad acceptance could avoid that situation. HTML5 certainly can be considered such a standard as it is the very foundation of the internet today.
A new and emerging standard that is part of HTML5 is 'Web Components'. Though not yet finalized, the fact that this specification is part of HTML5 itself makes us optimistic that an unwanted lock-in situation can be avoided this time and that this technology will be part of the common web platform for a long time in the future.
Prominent players like Google massively invest into Web Components and even have implemented their flagship 'youtube' with Web Components. The browser support is already quite broad.
It would exceed the limits of this article to give a full introduction to Web Components but the basics have to be explained here for a better understanding of the new architecture.
Web Components relies on four technologies:
- "Custom Elements" allow to invent your own custom HTML Elements e.g.
my-element. These are treated as every other built-in element likediv. Actually a custom element will behave like adivby default. That means that you can attach event-listeners to these elements, style them which CSS or otherwise manipulate them with JavaScript. The only requirement for a Custom Element is the fact that it needs a '-' character in its name. - "HTML template": the
<template>element of HTML5 defines an inert piece of markup that is meant to be stamped into DOM. 'inert' means that the contents of the template is not parsed, URLs will not be resolved and Script will not be run until the template is instanciated and added to the documents' DOM somewhere. - "Shadow DOM" is probably the biggest and most important part. It allows to attach document fragments to certain Custom Elements that render as if they were part of the main document. This technology allows to encapsulate Components - CSS from the main document will not influence the Shadow DOM, events will not pass the boundaries of a component. With Shadow DOM true components can be implemented that co-exist with arbitrary other code on a page without influencing it.
- "HTML imports" allow to load a component. They work just like the usual CSS imports e.g.
<link rel="import" href="my-component.html">. It should however be noted that HTML imports won't survive in the longer run. They couldn't find consensus under the participants of the working group and will be replaced by ES6 module loaders. As the code change is an automated step this can savely be ignored.
Not all of these parts need to present to make a Web Component. A different mechanism may be used to load the components and not every component needs a Shadow DOM. However, each Web Component is a Custom Element. If that Custom Element has its own logic we talk of a Web Component.
The approach taken with Ziziphus 2 builds on a simple assumption: Web Components should be easily mappable to XML.
Consider the following XML element:
<vra:work id="abc"></vra:work>
It can be mapped to:
<vra-work id="abc"></vra-work>
This way, whole XML documents can be translated into valid HTML5 documents that mirror the original XML by using Custom Elements.
But as always, the devil is in the details: XML and Web Components have slightly different naming rules:
- as Custom Elements are HTML they use lowercase letters only for tag-names. XML can have mixed or uppercase names.
- a Web Component must have a dash character ('-') in its name to become a valid Custom Element
The latter can easily be solved by using the prefix used in the XML, append a dash and use it as the prefix for the component name (e.g. 'vra-' + 'work'). If there's no prefix in the XML we can invent one.
The first is a bit more complicated. For example, there are element names like <agentSet>. The uppercase 'S' is forbidden in Web Components. In this case, we might have invented clever pattern-matching to camelcase names and the like, but we finally decided for an explicit name mapping. To setup this mapping we need a description file that knows about these mappings.
Here, the generator comes into play again. We already had a working generation pipeline with XSLT that outputs the UI Definition mentioned above. This file describes the element structure, the attributes and text nodes and is a perfect location to attach our mapping information.
This specification has been reviewed and refactored for enhanced clarity and to also cover the name mapping requirements.
Here's an example of the new mapping file in Ziziphus 2:
<element name="vra" prefix="vra" ns="http://..." skin="ziziphus-styles" map-to="vra">
<element name="work" add-mode="menu" map-to="work">
<element name="agent-set" label="Agents" map-to="agentSet">
<elementset name="agent" map-to="agent">
<element name="agent-name" map-to="name" label="Name">
<attribute name="type" type="agentNameTypeType">
<option>personal</option>
<option>corporate</option>
<option>family</option>
<option>other</option>
</attribute>
<textNode control="autocomplete" queryType="names" label="Agent Name"/>
</element>
Please note the following changes:
-
<set>has become<element> -
<group/@repeated="true">has become<elementset> -
<node>has become<attribute>
The @name attribute now contains the name of the Web Component without the prefix. The @map-to attribute holds the name of the respective XML element. As an element definition might occur several times in the same tree there's also a @skip attribute which will tell the code generator to ignore the entry.
With these modifications the original UI Definition file which was used for the generation of XForms is now capable of outputting Web Components.
It has been one important cornerstone of the project to keep former efforts where possible and sensible. A large part of the Ziziphus 1.0 generator could be kept and re-used to create the initial mapping.
For Ziziphus 2.0 a new generator has been built that works on top of the older one and covers the code generation for Web Components.
The following markup shows a complete generated Web Component destilled from the mapping file:
<dom-module id="vra-source">
<template>
<style include="ziziphus-styles">
</style>
<div class="wrapper">
<paper-icon-button id="star" icon="star-border" mini on-click="_setPreferred"></paper-icon-button>
<paper-menu-button horizontal-align="right">
<paper-icon-button icon="more-vert" mini slot="dropdown-trigger"></paper-icon-button>
<paper-listbox slot="dropdown-content">
<!--<paper-item>attributes...</paper-item>-->
<paper-item on-click="_showEmptySlots">show all</paper-item>
<paper-item on-click="_removeEntry">delete</paper-item>
</paper-listbox>
</paper-menu-button>
<slot name="refid"></slot>
<slot name="source-name"></slot>
</div>
</template>
<script>
class VraSource extends Polymer.Element {
static get is() {
return 'vra-source';
}
static get properties() {
return {}
}
constructor() {
super();
console.log("VraSource constructor...")
}
connectedCallback() {
super.connectedCallback();
console.log("VraSource connected");
this.setAttribute("slot", "source");
}
createSlotElements() {
var slots = this.shadowRoot.querySelectorAll("slot")
console.log("slots ", slots);
for (var i = 0, len = slots.length; i < len; i++) {
console.log("slot: ", slots[i].assignedNodes());
if (slots[i].assignedNodes().length == 0) {
var slot = slots[i].name
console.log("slot has no content - creating ", slot);
var e = document.createElement('vra-' + slot)
e.setAttribute("slot", slot)
this.appendChild(e);
}
}
}
_showEmptySlots(e) {
this.createSlotElements();
}
_removeEntry(e) {
console.log("vra-source delete");
if (confirm("really delete?")) {
this.parentNode.removeChild(this);
}
}
_setPreferred(e) {
this.$.star.icon = "star";
}
_valueChanged(val) {
console.log("newVal: ", val);
this.value = val;
this.innerHTML = val;
}
}
customElements.define(VraSource.is, VraSource);
</script>
</dom-module>
All repeating logic is auto-generated and the components are ready to display and edit their respective counterparts in XML.
In addition to code generation, the enhanced mapping file is now used at runtime to convert XML to Web Components and back.
The control flow is as follows:
- A user opens the application and selects a record for editing.
- This sends a request to the server that calls a generic XQuery module called 'webcomponents.xqm'. This in turn will locate the mapping file and output the Web Components representing the XML to be edited.
- The user will get a UI with controls to change the data values of the markup right in their browser.
- Once the user is done with editing the DOM is serialized and sent back to 'webcomponents.xqm' for mapping back to XML, which will then be stored. As the controls are living in the Shadow DOM of the components they don't pollute the original DOM and won't be part of the serialized browser document.
Ziziphus 2 in its current state is a prototype. But it is also a successful proof-of-concept.
The main accomplishments of the project are:
- A new Web Component Generator has been built that re-uses the former generator pipeline.
- The original UI Definition has evolved into a mapping file that both handles code generation as well as runtime mapping of elements.
- A user is able to browse a list of images, open a record, edit it and save the results as XML on the server.
- The UI allows repeating elements with auto-generated logic for adding and removing entries.
- The UI supports different controls like dropdown, input, textarea and the like.
- Elements that do not yet exist in the XML can be created dynamically in the UI while maintaining the desired order between siblings.
- In 1.0 the forms needed to be split into related chunks (one form for e.g. 'agentSet') to avoid memory overload and slow loading times. In 2.0 it is possible to load a complete document at once or use whatever granularity is appropriate.
- The application builds upon future-proof technolgies that are (or soon become) a native part of the browser platform.
The application is not yet ready for end-users. While the editing part works stable, other parts like layout, styling and overall usability were not put into focus for this prototype.
Although not all functionalities of Ziziphus 1.0 could be reproduced, it has been shown that it is do-able.
The mapping was developed with the assumption that the whole XML document uses a single namespace. Multiple namespaces have not been explored but it should be possible to implement this.
In summary, Ziziphus 2.0 has shown that a mapping from XML to Web Components and back is possible.
By evolving the former UI Definition a new Web Component Generator was built that outputs ready-to-use components that support CRUDS operations on the data.
Though the application was built with VRA Core 4 XML in mind it quickly showed to be a generic approach that will work for other XML languages in the same way. The Web Component Generator and the 'webcomponents.xqm' XQuery module are completely generic and work with any XML.
-
todo: link to the github project once it's there