#6: Serving docs with gateway identity filled-in

doc/README.md

@@ -1,5 +1,9 @@
 # m-ld Gateway Documentation
 
-This `doc` folder is intended to form the basis of a documentation portal for the **m-ld** Gateway, published on the web via a CDN.
+This `doc` folder forms the basis of the **m-ld** Gateway public documentation.
 
-The entry point is [Getting Started](getting-started).
+The folder is built using Eleventy to the `_site` folder, and then served by the running Gateway in the class `GatewayWebsite`.
+
+cURLs in markdown files are generated from the corresponding .http files using `http-client.env.json`, please keep them in sync.
+
+All files are treated as Liquid templates. They are processed once by Eleventy, and then _again_ by the gateway – hence the occasional use of double-encoded template tags such as `{{ '{{ origin }}' }}`.

doc/_includes/index-example.js

@@ -6,9 +6,9 @@ const domainId = uuid();
 
 const meld = await clone(new MemoryLevel(), IoRemotes, {
   '@id': uuid(),
-  '@domain': `${domainId}.public.gw.m-ld.org`,
+  '@domain': `${domainId}.public.{{ '{{ domain }}' }}`,
   genesis: true, // Other clones will have `false`
-  io: { uri: 'https://gw.m-ld.org' }
+  io: { uri: "{{ '{{ origin }}' }}" }
 });
 
 // Tell other clones the domain ID so they can join!

doc/accounts.md

@@ -2,8 +2,6 @@
 layout: doc.liquid
 title: accounts
 ---
-[//]: # (cURLs in this file are generated from the .http file using http-client.env.json)
-
 # Accounts
 
 Accounts have two purposes in the **m-ld** Gateway.
@@ -19,7 +17,7 @@ Account names (`≪account≫` in the below) must be composed only of **lowercas
 First, request an activation code with an email address.
 
 ```bash
-curl -X POST --location "https://≪gateway≫/api/v1/user/≪account≫/activation" \
+curl -X POST --location "{{ '{{ origin }}' }}/api/v1/user/≪account≫/activation" \
     -H "Accept: application/json" \
     -H "Content-Type: application/json" \
     -d "{ \"email\": \"≪email≫\" }"
@@ -32,7 +30,7 @@ An email will be sent to the given address, containing a six-digit activation co
 The account can then be created with another HTTP request:
 
 ```bash
-curl -X POST --location "https://≪gateway≫/api/v1/user/≪account≫/key" \
+curl -X POST --location "{{ '{{ origin }}' }}/api/v1/user/≪account≫/key" \
     -H "Authorization: Bearer ≪jwe≫" \
     -H "X-Activation-Code: ≪emailed activation code≫" \
     -H "Accept: application/json"
@@ -45,7 +43,7 @@ The body of the response will be of the form `{ "auth": { "key": "≪my-key≫"
 The Gateway root account can be used to create any user account directly.
 
 ```bash
-curl -X POST --location "https://≪gateway≫/api/v1/user/≪account name≫/key" \
+curl -X POST --location "{{ '{{ origin }}' }}/api/v1/user/≪account name≫/key" \
     -H "Accept: application/json" \
     --basic --user ≪root≫:≪root key≫
 ```
@@ -62,7 +60,7 @@ When [connecting to subdomains](clone-subdomain), clients may need to provide au
 The required option can be set as follows.
 
 ```bash
-curl -X PATCH --location "https://≪gateway≫/api/v1/user/≪account name≫" \
+curl -X PATCH --location "{{ '{{ origin }}' }}/api/v1/user/≪account name≫" \
     -H "Accept: application/json" \
     -H "Content-Type: application/json" \
     -d "{ \"@insert\": { \"remotesAuth\": \"≪remotes auth option≫\" } }" \

doc/clone-api.md

@@ -2,8 +2,6 @@
 layout: doc.liquid
 title: clone API
 ---
-[//]: # (cURLs in this file are generated from the .http file using http-client.env.json)
-
 # Clone API
 
 Let's say we have an account and a named subdomain. Because we're using [named subdomains](named-subdomains), the Gateway has a clone of the information in the (sub)domain.
@@ -18,7 +16,7 @@ The 'clone' (noun, not verb) API is provided for such a client to access the dat
 Some data can be added to the domain with:
 
 ```bash
-curl -X POST --location "https://≪gateway≫/api/v1/domain/≪account name≫/≪subdomain≫/state" \
+curl -X POST --location "{{ '{{ origin }}' }}/api/v1/domain/≪account name≫/≪subdomain≫/state" \
     -H "Content-Type: application/json" \
     -d "{
           \"@id\": \"Client-0005\",
@@ -30,7 +28,7 @@ curl -X POST --location "https://≪gateway≫/api/v1/domain/≪account name≫/
 Data in the domain can be queried with:
 
 ```bash
-curl -X GET --location "https://≪gateway≫/api/v1/domain/≪account name≫/≪subdomain≫/state?q=%7B%22%40describe%22%3A%22%3Fid%22%2C%22%40where%22%3A%7B%22%40id%22%3A%22%3Fid%22%7D%7D" \
+curl -X GET --location "{{ '{{ origin }}' }}/api/v1/domain/≪account name≫/≪subdomain≫/state?q=%7B%22%40describe%22%3A%22%3Fid%22%2C%22%40where%22%3A%7B%22%40id%22%3A%22%3Fid%22%7D%7D" \
     -H "Accept: application/json" \
     --basic --user ≪account name≫:≪account key≫
 ```

doc/getting-started.md

@@ -6,26 +6,26 @@ title: getting started
 
 > I'm just getting started with **m-ld**. I want to try building an app, without having to deploy a messaging service.
 
-We run a free public Socket.io messaging service for **m-ld**, on our cloud Gateway! Your domains will be named like `≪uuid≫.public.gw.m-ld.org`, and you'll need an internet connection to create new domains and keep clones synchronised (although individual clones can work offline, as usual). For usage instructions see [UUID subdomains](uuid-subdomains).
+The Gateway serves free public messaging for **m-ld**! Your domains will be named like `≪uuid≫.public.{{ '{{ domain }}' }}`, and you'll need a network connection to create new domains and keep clones synchronised (although individual clones can work offline, as usual). For usage instructions see [UUID subdomains](uuid-subdomains).
 
 ---
 
 > I'm building a client-only browser or desktop app with **m-ld**. I want to offer secure backup of my users' data.
 
-On our cloud Gateway you can sign up for your own free account, and create domains which are backed up there. Your domains will be named like `≪name≫.≪account≫.gw.m-ld.org`, and you'll need an internet connection to create new domains and keep clones synchronised (although individual clones can work offline, as usual). For usage instructions see [named subdomains](named-subdomains).
+You can sign up for your own free Gateway account, and create domains which are backed up here. Your domains will be named like `≪name≫.≪account≫.{{ '{{ domain }}' }}`, and you'll need an internet connection to create new domains and keep clones synchronised (although individual clones can work offline, as usual). For usage instructions see [named subdomains](named-subdomains).
 
 ---
 
 > I'm building an app with a service tier, using **m-ld** for data distribution.
 
-You can use our cloud Gateway to provide messaging and secure backup of your domains. You service tier can just talk to the cloud Gateway as if it were a client. To set your own service levels, or to work with a restricted network, you might choose to [self-host your own Gateway](self-host).
+You can use the Gateway to provide messaging and secure backup of your domains. You service tier can just talk to the Gateway as if it were a client. To set your own service levels, or to work with a restricted network, you might choose to [self-host your own Gateway](self-host).
 
 ---
 
 > I'm upgrading a legacy app, with a service tier and database, to offer live document sharing using **m-ld**.
 
 Keeping a database in sync with information in **m-ld** requires a dedicated clone, local to the database, which can offer the kind of serialised updates that conventional databases like.
 
-The best deployment approach will be to embed this local clone in your service tier, where it can be animated directly from the application logic. If an engine doesn't exist for your server platform though, you can [deploy a **m-ld** Gateway](self-host) in a "sidecar" arrangement with your services. The Gateway [clone API](clone-api) can be used to provide serialised state to your app, for synchronisation with the database.
+The best deployment approach will be to embed this local clone in your service tier, where it can be animated directly from the application logic. If an engine doesn't exist for your server platform though, you can [deploy a Gateway](self-host) in a "sidecar" arrangement with your services. The Gateway [clone API](clone-api) can be used to provide serialised state to your app, for synchronisation with the database.
 
 > 🚧 More detail will be available here soon. In the meantime, please do [get in touch](http://m-ld.org/hello/) to discuss your use-case!

doc/http-client.env.json

@@ -6,7 +6,7 @@
     "subdomain": "my-subdomain"
   },
   "doc": {
-    "gateway": "https://≪gateway≫",
+    "gateway": "{{ '{{ origin }}' }}",
     "rootAccount": "≪root≫",
     "rootKey": "≪root key≫",
     "account": "≪account name≫",

doc/named-subdomains.md

@@ -2,8 +2,6 @@
 layout: doc.liquid
 title: named subdomains
 ---
-[//]: # (cURLs in this file are generated from the .http file using http-client.env.json)
-
 # Using Named Subdomains
 
 Named subdomains are _cloned_ in the Gateway, and are thereby backed up.
@@ -15,7 +13,7 @@ To use named subdomains, you first need [an account](accounts). (If your Gateway
 A new domain can be created with:
 
 ```bash
-curl -X PUT --location "https://≪gateway≫/api/v1/domain/≪account name≫/≪subdomain≫" \
+curl -X PUT --location "{{ '{{ origin }}' }}/api/v1/domain/≪account name≫/≪subdomain≫" \
     -H "Accept: application/json" \
     --basic --user ≪account name≫:≪account key≫
 ```

doc/uuid-subdomains.md

@@ -2,9 +2,6 @@
 layout: doc.liquid
 title: uuid subdomains
 ---
-
-[//]: # (cURLs in this file are generated from the .http file using http-client.env.json)
-
 # Using UUID Subdomains
 
 UUID subdomains are _unmanaged_ by the Gateway, and are not backed up. The Gateway provides a message relay to allow clones on the subdomain to communicate.
@@ -19,7 +16,7 @@ To use UUID subdomains, you first need [an account](accounts).
 By default, a Gateway account only allows [named subdomains](named-subdomains). To enable UUID subdomains for an account:
 
 ```bash
-curl -X PATCH --location "https://≪gateway≫/api/v1/user/≪account name≫" \
+curl -X PATCH --location "{{ '{{ origin }}' }}/api/v1/user/≪account name≫" \
     -H "Content-Type: application/json" \
     -d "{ \"@insert\": { \"naming\": \"uuid\" } }" \
     --basic --user ≪account name≫:≪account key≫
@@ -34,7 +31,7 @@ The domain name must take the form `≪uuid≫.my-account.my-gateway`, where `
 For convenience, you can request suitable configuration for a new UUID subdomain from the Gateway, as follows. Note that this does not create anything new on the Gateway, but it will generate a compliant UUID for the domain name.
 
 ```bash
-curl -X POST --location "https://≪gateway≫/api/v1/domain/≪account name≫" \
+curl -X POST --location "{{ '{{ origin }}' }}/api/v1/domain/≪account name≫" \
     -H "Accept: application/json"
 ```
 

src/http/EndPoint.ts

@@ -44,7 +44,7 @@ function nextifyHandler(handler: NextFreeHandler): RequestHandler {
   };
 }
 
-type Verb = 'del' | 'get' | 'put' | 'post' | 'patch';
+type Verb = 'del' | 'get' | 'head' | 'put' | 'post' | 'patch';
 type RouteDef = string | RegExp | RouteOptions;
 type Routable = Pick<RestServer, Verb>;
 
@@ -62,6 +62,7 @@ interface EndPointSetup {
 export class EndPoint<Outer extends Routable> implements Routable {
   del: RestServer['del'];
   get: RestServer['get'];
+  head: RestServer['head'];
   put: RestServer['put'];
   post: RestServer['post'];
   patch: RestServer['patch'];
@@ -79,7 +80,7 @@ export class EndPoint<Outer extends Routable> implements Routable {
       EndPoint.checkRoute(route);
       return `${stem}${route}`;
     };
-    for (let verb of ['del', 'get', 'put', 'post', 'patch'] as Verb[]) {
+    for (let verb of ['del', 'get', 'head', 'put', 'post', 'patch'] as Verb[]) {
       this[verb] = (route: string, ...handlers) => outer[verb](
         api(route),
         ...this.useHandlers,
@@ -149,6 +150,7 @@ function handlerDecorator(verb: Verb | 'use', opts: RouteDef = '') {
 export const use = handlerDecorator('use');
 export const del = handlerDecorator('del');
 export const get = handlerDecorator('get');
+export const head = handlerDecorator('head');
 export const put = handlerDecorator('put');
 export const post = handlerDecorator('post');
 export const patch = handlerDecorator('patch');

src/http/GatewayWebsite.ts

@@ -1,31 +1,45 @@
-import { EndPoint, post } from './EndPoint.js';
+import { EndPoint, get, head, post } from './EndPoint.js';
 import { plugins, Request, Response, Server } from 'restify';
-import { Gateway, Notifier } from '../server/index.js';
-import { fileURLToPath } from 'url';
-import { AccountOwnedId, validate } from '../lib/index.js';
+import { AccountOwnedId, resolveGateway, validate } from '../lib/index.js';
 import { as } from '../lib/validate.js';
-import { Liquid } from 'liquidjs';
 import { pipeline } from 'stream/promises';
-import { toHttpError } from './errors.js';
-
-/** Directory of the m-ld-gateway package */
-const siteDir = fileURLToPath(new URL('../../_site/', import.meta.url));
+import { MethodNotAllowedError, toHttpError } from './errors.js';
+import type { Gateway, Notifier } from '../server/index.js';
+import type { Liquid } from 'liquidjs';
 
 export class GatewayWebsite extends EndPoint<Server> {
-  private liquid = new Liquid({ root: siteDir });
+  private readonly pageVars: Promise<{ origin: string, domain: string }>;
+  private readonly startTime = new Date();
 
   constructor(
     readonly gateway: Gateway,
     server: Server,
-    private notifier: Notifier
+    private readonly notifier: Notifier,
+    private readonly liquid: Liquid
   ) {
     super(server, '', ({ useFor }) =>
       useFor('post', plugins.bodyParser()));
-    server.get('/*', plugins.serveStatic({
-      directory: siteDir, default: 'index'
+    this.pageVars = Promise.resolve(resolveGateway(gateway.config.gateway)).then(url => ({
+      origin: url.origin, domain: gateway.domainName
     }));
   }
 
+  @get('/*')
+  async getPage(req: Request, res: Response) {
+    if (req.path() === '/activate')
+      throw new MethodNotAllowedError;
+    await this.renderHtml(res, req.path().slice(1) || 'index', await this.pageVars);
+  }
+
+  @head('/*')
+  async headPage(req: Request, res: Response) {
+    if (req.path() === '/activate')
+      throw new MethodNotAllowedError;
+    // This will throw ENOENT if not found
+    await this.liquid.parseFile(req.path().slice(1) || 'index');
+    this.setHtmlHeaders(res).send();
+  }
+
   @post('/activate')
   async activation(req: Request, res: Response) {
     const { account, email, code, jwe } = req.body;
@@ -47,8 +61,19 @@ export class GatewayWebsite extends EndPoint<Server> {
     } catch (e) {
       pageVars = { account, email, error: toHttpError(e).toJSON() };
     }
-    const html = await this.liquid.renderFileToNodeStream('activate', pageVars);
+    await this.renderHtml(res, 'activate', pageVars);
+  }
+
+  private async renderHtml(res: Response, file: string, pageVars: {}) {
+    // This will throw ENOENT if not found
+    const html = await this.liquid.renderFileToNodeStream(file, pageVars);
+    await pipeline(html, this.setHtmlHeaders(res));
+  }
+
+  private setHtmlHeaders(res: Response) {
+    res.header('content-type', 'text/html');
     res.header('transfer-encoding', 'chunked');
-    await pipeline(html, res);
+    res.header('last-modified', this.startTime);
+    return res;
   }
 }

src/http/errors.ts

@@ -1,11 +1,11 @@
 import errors from 'restify-errors';
 import { as } from '../lib/validate.js';
 
-export function toHttpError(e: any) {
-  return e instanceof errors.HttpError ? e :
-    as.isError(e) ? new BadRequestError(e) :
-      new InternalServerError(e);
-}
+export const toHttpError = (e: any) =>
+  e instanceof errors.HttpError ? e :
+    e?.code === 'ENOENT' ? new NotFoundError(e) :
+      as.isError(e) ? new BadRequestError(e) :
+        new InternalServerError(e);
 
 export const UnauthorizedError = errors.UnauthorizedError;
 export const MethodNotAllowedError = errors.MethodNotAllowedError;

src/http/index.ts

@@ -1,15 +1,20 @@
 import { createServer, plugins, pre, Server as RestServer } from 'restify';
 import LOG from 'loglevel';
-import { Gateway, Notifier } from '../server/index.js';
 import { formatter, HTML_FORMAT, JSON_LD_FORMAT } from './EndPoint.js';
 import { ApiEndPoint } from './ApiEndPoint.js';
 import { SubdomainEndPoint } from './SubdomainEndPoint.js';
 import { SubdomainStateEndPoint } from './SubdomainStateEndPoint.js';
 import { UserEndPoint } from './UserEndPoint.js';
 import { DomainEndPoint } from './DomainEndPoint.js';
 import { GatewayWebsite } from './GatewayWebsite.js';
+import type { Gateway, Notifier } from '../server/index.js';
+import type { Liquid } from 'liquidjs';
 
-export function setupGatewayHttp(gateway: Gateway, notifier: Notifier): RestServer {
+export function setupGatewayHttp({ gateway, notifier, liquid }: {
+  gateway: Gateway,
+  notifier: Notifier,
+  liquid: Liquid
+}): RestServer {
   const server = createServer({
     formatters: {
       'application/ld+json': formatter(JSON_LD_FORMAT),
@@ -24,14 +29,17 @@ export function setupGatewayHttp(gateway: Gateway, notifier: Notifier): RestServ
     });
   if (LOG.getLevel() <= LOG.levels.DEBUG) {
     server.pre(function (req, _res, next) {
-      LOG.debug(`${req.method} ${req.url} ${JSON.stringify({
-        ...req.headers, authorization: undefined
-      })}`);
+      if (LOG.getLevel() <= LOG.levels.TRACE)
+        LOG.trace(req.method, req.url, {
+          ...req.headers, authorization: undefined
+        });
+      else
+        LOG.debug(req.method, req.url);
       return next();
     });
   }
   // Set up endpoints
-  new GatewayWebsite(gateway, server, notifier);
+  new GatewayWebsite(gateway, server, notifier, liquid);
   const apiEndPoint = new ApiEndPoint(gateway, server);
   new UserEndPoint(apiEndPoint, notifier);
   new DomainEndPoint(apiEndPoint);

src/lib/BaseGateway.ts

@@ -48,23 +48,22 @@ export class BaseGateway {
  *
  */
 export function resolveGateway(
-  address: string
-): { root: URL | Promise<URL>, domainName: string } {
-  if (isFQDN(address)) {
-    return { root: new URL(`https://${address}/`), domainName: address };
+  address: string | URL
+): URL | Promise<URL> {
+  if (address instanceof URL) {
+    return address;
+  } else if (isFQDN(address)) {
+    return new URL(`https://${address}/`);
   } else {
     const url = new URL('/', address);
     const domainName = url.hostname;
     if (domainName.endsWith('.local')) {
-      return {
-        root: dns.lookup(domainName).then(a => {
-          url.hostname = a.address;
-          return url;
-        }),
-        domainName
-      };
+      return dns.lookup(domainName).then(a => {
+        url.hostname = a.address;
+        return url;
+      });
     } else {
-      return { root: url, domainName };
+      return url;
     }
   }
 }