The web integration feature is intended to simplify usage of the Pipeline API in websites/applications that are built using a particular web framework.
The exact details will vary significantly based on the web framework that is being used. The capabilities and common usage of the target web framework will need to be fully understood so that the integration is in keeping with that ecosystem, rather than simply repeating what worked for some other language/framework.
Some common tasks that the web integration performs are:
- Creation of Pipeline on start-up.
- Manage Flow Data lifecycle
- Create Flow Data when an HTTP request comes in
- Populate Evidence values and call 'Process'
- Make the processed Flow Data easily accessible to other parts of the application so the results can be used.
- Handle resource cleanup once request handling is completed.
- Handling requests for client-side functionality.
- Set HTTP response headers to request additional information if needed.
The web integration makes use of several Flow Elements in order to support client-side functionality.
Where the web integration is responsible for creation of the Pipeline, it will also need to ensure that these additional Flow Elements are present.
- The Sequence Element MUST be present. If not, it MUST be added as the first element in the Pipeline.
- The JSON Builder Element MUST be present. If not, it MUST be added as the penultimate element. (Immediately before the JavaScript Builder Element).
- The JavaScript Builder Element MUST be present. If not, it MUST be added after all other elements.
Another Flow Element is needed to allow the web integration to automatically set HTTP response headers.
- The Set Headers Element MUST be present. If not, it MUST be added after all other elements.
Note that Set Headers Element does not interact with JavaScript Builder Element, etc. Consequently, it doesn't matter which one is actually the last element, they just need to be added after any non-core Elements.
For all of these elements, there MUST be configuration options to allow users to prevent them being added (or disable them) if they wish to do so.
The web integration SHOULD automatically populate Evidence from the web request.
It MAY use the Pipeline's accepted Evidence feature to determine if individual values need to be added to Evidence or not.
An alternative approach would be to wrap the web request structure itself as Evidence, and hence defer processing until the Evidence is actually used.
In any case, the list below illustrates the type of data that can be used and the Evidence names they would be associated with.
- All HTTP headers. (Except cookies header) Key is
header.[header name] - All cookies. Key is
cookie.[cookie name] - All query string parameters. Key is
query.[parameter name] - All form parameters from POST requests. Key is
query.[parameter name] - Where there is an HTTP session object, data stored in the session. Key is
session.[value name] - Public client IP. Key is
server.client-ip - Request protocol. Can come from the request itself or headers such as
X-Origin-ProtoorX-Forwarded-Proto. Key isheader.protocol
The Set Headers Element will produce an output that describes which response headers need to be set to which values.
However, it has no access to the web request itself. Consequently, the web integration logic will need to take this output and actually set the necessary response headers.
Where the relevant headers are already set to some value, append the new value, rather than replacing the existing value.
There are two major client-side features that the web integration provides:
- Enable the execution of JavaScript snippets which can be used to gather additional Evidence. Handle passing this data back to the Pipeline for processing.
- Allow the user to access results of Pipeline processing in client side code.
Generally, this is achieved by adding a specific JavaScript include to the HTML page. The web integration intercepts the request to this URL and serves a JavaScript object that can be used to access the results from processing.
Elements can return Properties with a type of Javascript. For these Properties, the Property value contains JavaScript code that is intended to be executed on the client device.
As these JavaScript Property values are part of the result set, they will already be accessible in client-side code through the JavaScript include that is discussed in the section above.
Therefore, in order to meet this requirement, the JavaScript include will need to:
- Identify these JavaScript Property values
- Execute the snippets
- Call back to the server, including the results from the execution of the snippets
- Use the result from the callback to update the result data set
- Repeat if there are any JavaScript Property values in the new result set. (only repeat up to a configurable limit in order to avoid infinite loops)
These features are implemented using the web integration logic itself, along with several Flow Elements:
- Sequence Element - Used to prevent the infinite loops described in the section above.
- JSON Builder Element - Converts the Property values in the current Flow Data into a JSON data object.
- JavaScript Builder Element - Takes the output from JSON Builder Element and packages it within a JavaScript template.
In order for this to work, the web integration needs to intercept requests to two urls. These SHOULD be configurable, but the default values are:
/51Degrees.core.js- Serves the JavaScript produced by JavaScript Builder Element./51dpipeline/json- Serves the JSON produced by JSON Builder Element.
Where possible, we want to prevent the client from evaluating JavaScript or making requests unnecessarily. To achieve this, the web integration will need to include logic to set HTTP cache control headers when responding to requests to these intercepted endpoints.
Requests to the JSON endpoint will be POST requests, as opposed to the GET requests made to the JavaScript endpoint.
This is a problem because most browsers do not consider POST requests to be cache-able and will not respect cache headers.
The reference implementations work around this by using the session storage API. See the JavaScript Builder Element for more details.
The HTTP response headers that MUST be set for the JSON and JavaScript endpoints are:
| Header | Values | Purpose |
|---|---|---|
| Cache-Control | max-age=1800 private | Cache item lifetime is 30 minutes. Only local client caches can cache this content. |
| Vary | all headers in Pipeline Evidence key filter (e.g. User-Agent) | Let the cache know that if one of these headers changes, the cached content will need to be re-fetched. |
| ETag | Calculated hash of ALL the Evidence values in Evidence filter key for this Pipeline | Assists caches in re-validating expired content |
| Content-Type | application/x-javascript or application/JSON | Indicate the type of content being returned |
| Content-Length | Content length in bytes | Indicate the expected length of the content |
In order to validate the ETag, the JSON and JavaScript endpoints will need to check for an 'If-None-Match' header in the request. This will be sent when a cached item’s lifetime has expired and the cache needs to check if what it holds is still valid.
If the value in the 'If-None-Match' header matches the calculated ETag value for the current request, then the content in the cache is still valid and the endpoint can just return a 304 status code. (Not modified)
This process could lead to responses that are not strictly correct if the JavaScript snippet is getting data values that might change over time. For example, if we're using JavaScript to get location and the user has moved then session storage will still contain the previous location lookup response. In practice this seems unlikely to cause any real issues, as long as the cache max-age is not too long. Max-age SHOULD be configurable to allow this to be addressed if it is found to be an issue for a particular client.