Orderby support in entity odata endpoint

docs/api.yaml

@@ -36,6 +36,14 @@ info:
 
     Here major and breaking changes to the API are listed by version.
 
+    ## ODK Central v2024.1
+
+    **Added**:
+
+    - OData Data Document for requests of Submissions and Entities now allow use of `$orderby`.
+    - ETag headers on all Blobs.
+
+
     ## ODK Central v2023.5
 
     **Added**:
@@ -556,7 +564,6 @@ tags:
 
     * The actual data documents, linked from the Service Document, are a simple JSON representation of the submission data or entity, conforming to the schema we describe in our Metadata Document.
 
-    As our focus is on the bulk-export of data from ODK Central so that more advanced analysis tools can handle the data themselves, we do not support most of the features at the Intermediate and above conformance levels, like `$sort` or `$filter`.
 - name: System Endpoints
   description: There are some resources available for getting or setting system information
     and configuration. You can set the [Usage Reporting configuration](/central-api-system-endpoints/#usage-reporting-configuration)
@@ -5417,6 +5424,7 @@ paths:
       responses:
         200:
           description: OK
+          headers:
             ETag:
               schema:
                 type: string
@@ -11572,7 +11580,7 @@ paths:
 
         The `$top` and `$skip` querystring parameters, specified by OData, apply `limit` and `offset` operations to the data, respectively. The `$count` parameter, also an OData standard, will annotate the response data with the total row count, regardless of the scoping requested by `$top` and `$skip`. If `$top` parameter is provided in the request then the response will include `@odata.nextLink` that you can use as is to fetch the next set of data. While paging is possible through these parameters, it will not greatly improve the performance of exporting data. ODK Central prefers to bulk-export all of its data at once if possible.
 
-        As of ODK Central v1.1, the [`$filter` querystring parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358948) is partially supported. In OData, you can use `$filter` to filter by any data field in the schema. The operators `lt`, `le`, `eq`, `ne`, `ge`, `gt`, `not`, `and`, and `or` are supported. The built-in functions `now`, `year`, `month`, `day`, `hour`, `minute`, `second` are supported. These supported elements may be combined in any way, but all other `$filter` features will cause an error.
+        As of ODK Central v1.1, the [`$filter` querystring parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358948) is partially supported. In OData, you can use `$filter` to filter by certain data fields in the schema. The operators `lt`, `le`, `eq`, `ne`, `ge`, `gt`, `not`, `and`, and `or` are supported. The built-in functions `now`, `year`, `month`, `day`, `hour`, `minute`, `second` are supported. These supported elements may be combined in any way, but all other `$filter` features will cause an error.
 
         The fields you can query against are as follows:
 
@@ -11599,6 +11607,8 @@ paths:
 
         + Child properties of repeats can't be requested using `$select`
 
+        As of ODK Central v2024.1, the [`$orderby` query parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358952) is now supported, and can sort on the same fields as `$filter`, noted above. The order can be specified as `ASC` (ascending) or `DESC` (descending), which are case-insensitive. Multiple sort expressions can be used together, separated by commas, e.g. `$orderby=__system/submitterId ASC, __system/reviewState DESC`.
+
         As the vast majority of clients only support the JSON OData format, that is the only format ODK Central offers.
       operationId: Data Document
       parameters:
@@ -11663,6 +11673,12 @@ paths:
         schema:
           type: string
         example: year(__system/submissionDate) lt year(now())
+      - name: '%24orderby'
+        in: query
+        description: If provided, will sort responses according to specified order expression. Only the same fields as `$filter` above can be used to sort. Multiple expressions can be used together.
+        schema:
+          type: string
+        example: __system/submitterId asc, __system/updatedAt desc
       - name: '%24expand'
         in: query
         description: Repetitions, which should get expanded. Currently, only `*` is
@@ -12067,15 +12083,12 @@ paths:
 
         The `$top` and `$skip` querystring parameters, specified by OData, apply `limit` and `offset` operations to the data, respectively. The `$count` parameter, also an OData standard, will annotate the response data with the total row count, regardless of the scoping requested by `$top` and `$skip`. If `$top` parameter is provided in the request then the response will include `@odata.nextLink` that you can use as is to fetch the next set of data.
 
-        The [`$filter` querystring parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358948) can be used to filter by any data field in the system-level schema, but not the Dataset properties. The operators `lt`, `le`, `eq`, `ne`, `ge`, `gt`, `not`, `and`, and `or` are supported. The built-in functions `now`, `year`, `month`, `day`, `hour`, `minute`, `second` are supported.
+        The [`$filter` querystring parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358948) can be used to filter certain data fields in the system-level schema, but not the Dataset properties. The operators `lt`, `le`, `eq`, `ne`, `ge`, `gt`, `not`, `and`, and `or` are supported. The built-in functions `now`, `year`, `month`, `day`, `hour`, `minute`, `second` are supported.
 
         The fields you can query against are as follows:
 
         | Entity Metadata         | OData Field Name     |
         | ------------------------| -------------------- |
-        | Entity UUID             | `__id`               |
-        | Entity Name (same as UUID) | `name`            |
-        | Entity Label            | `label`              |
         | Entity Creator Actor ID | `__system/creatorId` |
         | Entity Timestamp        | `__system/createdAt` |
         | Entity Update Timestamp | `__system/updatedAt` |
@@ -12087,6 +12100,8 @@ paths:
 
         The [`$select` query parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358942) will return just the fields you specify and is supported on `__id`, `__system`, `__system/creatorId`, `__system/createdAt` and `__system/updatedAt`, as well as on user defined properties.
 
+        The [`$orderby` query parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358952) will return Entities sorted by different fields, which come from the same list used by `$filter`, as noted above. The order can be specified as `ASC` (ascending) or `DESC` (descending), which are case-insensitive. Multiple sort expressions can be used together, separated by commas, e.g. `$orderby=__system/creatorId ASC, __system/conflict DESC`.
+
         As the vast majority of clients only support the JSON OData format, that is the only format ODK Central offers.
       operationId: Data Document for Dataset
       parameters:
@@ -12135,6 +12150,12 @@ paths:
         schema:
           type: string
         example: year(__system/createdAt) lt year(now())
+      - name: '%24orderby'
+        in: query
+        description: If provided, will sort responses according to specified order expression. Only the same fields as `$filter` above can be used to sort. Multiple expressions can be used together.
+        schema:
+          type: string
+        example: __system/creatorId asc, __system/updatedAt desc
       - name: '%24select'
         in: query
         description: If provided, will return only the selected fields.

lib/data/odata-filter.js

@@ -87,5 +87,33 @@ const odataFilter = (expr, odataToColumnMap) => {
   return op(ast);
 };
 
-module.exports = { odataFilter };
+const odataOrderBy = (expr, odataToColumnMap, stableOrderColumn = null) => {
+  let initialOrder = null;
+  const clauses = expr.split(',').map((exp) => {
+    const [col, order] = exp.trim().split(/\s+/);
+
+    // validate field
+    if (!odataToColumnMap.has(col))
+      throw Problem.internal.unsupportedODataField({ text: col });
+
+    // validate order (asc or desc)
+    if (order && !order?.toLowerCase().match(/^(asc|desc)$/))
+      throw Problem.internal.unsupportedODataField({ text: order });
+
+    const sqlOrder = (order?.toLowerCase() === 'desc') ? sql`DESC NULLS LAST` : sql`ASC NULLS FIRST`;
+
+    // Save the order of the initial property to use for the stable sort column order
+    if (initialOrder == null)
+      initialOrder = sqlOrder;
+
+    return sql`${sql.identifier(odataToColumnMap.get(col).split('.'))} ${sqlOrder}`;
+  });
+
+  if (stableOrderColumn != null)
+    clauses.push(sql`${sql.identifier(stableOrderColumn.split('.'))} ${initialOrder}`);
+
+  return sql`ORDER BY ${sql.join(clauses, sql`,`)}`;
+};
+
+module.exports = { odataFilter, odataOrderBy };
 

lib/http/endpoint.js

@@ -313,7 +313,7 @@ const isJsonType = (x) => /(^|,)(application\/json|json)($|;|,)/i.test(x);
 const isXmlType = (x) => /(^|,)(application\/(atom(svc)?\+)?xml|atom|xml)($|;|,)/i.test(x);
 
 // various supported odata constants:
-const supportedParams = [ '$format', '$count', '$skip', '$top', '$filter', '$wkt', '$expand', '$select', '$skiptoken' ];
+const supportedParams = [ '$format', '$count', '$skip', '$top', '$filter', '$wkt', '$expand', '$select', '$skiptoken', '$orderby' ];
 const supportedFormats = {
   json: [ 'application/json', 'json' ],
   xml: [ 'application/xml', 'atom' ]

lib/model/query/entities.js

@@ -13,7 +13,7 @@ const { equals, extender, unjoiner, page, markDeleted } = require('../../util/db
 const { map, mergeRight, pickAll } = require('ramda');
 const { blankStringToNull, construct } = require('../../util/util');
 const { QueryOptions } = require('../../util/db');
-const { odataFilter } = require('../../data/odata-filter');
+const { odataFilter, odataOrderBy } = require('../../data/odata-filter');
 const { odataToColumnMap, parseSubmissionXml, getDiffProp, ConflictType } = require('../../data/entity');
 const { isTrue } = require('../../util/http');
 const Problem = require('../../util/problem');
@@ -449,7 +449,9 @@ WHERE
   AND entities."deletedAt" IS NULL
   AND entity_defs.current=true
   AND  ${odataFilter(options.filter, odataToColumnMap)}
-ORDER BY entities."createdAt" DESC, entities.id DESC
+${options.orderby ? sql`
+  ${odataOrderBy(options.orderby, odataToColumnMap, 'entities.id')}
+  `: sql`ORDER BY entities."createdAt" DESC, entities.id DESC`}
 ${page(options)}`)
     .then(stream.map(_exportUnjoiner));
 

lib/model/query/submissions.js

@@ -11,7 +11,7 @@ const { map } = require('ramda');
 const { sql } = require('slonik');
 const { Frame, table } = require('../frame');
 const { Actor, Form, Submission } = require('../frames');
-const { odataFilter } = require('../../data/odata-filter');
+const { odataFilter, odataOrderBy } = require('../../data/odata-filter');
 const { odataToColumnMap, odataSubTableToColumnMap } = require('../../data/submission');
 const { unjoiner, extender, equals, page, updater, QueryOptions, insertMany } = require('../../util/db');
 const { blankStringToNull, construct } = require('../../util/util');
@@ -359,7 +359,9 @@ where
   ${odataFilter(options.filter, options.isSubmissionsTable ? odataToColumnMap : odataSubTableToColumnMap)} and
   ${equals(options.condition)}
   and submission_defs.current=true and submissions."formId"=${formId} and submissions."deletedAt" is null
-order by submissions."createdAt" desc, submissions.id desc
+${options.orderby ? sql`
+  ${odataOrderBy(options.orderby, odataToColumnMap, 'submissions.id')}`
+    : sql`order by submissions."createdAt" desc, submissions.id desc`}
 ${page(options)}`;
 };
 

lib/util/db.js

@@ -383,6 +383,11 @@ class QueryOptions {
       result.filter = query.$filter;
     if ((params.table === 'Submissions') && (query.$skiptoken != null))
       result.skiptoken = QueryOptions.parseSkiptoken(query.$skiptoken);
+    if (query.$orderby != null)
+      result.orderby = query.$orderby;
+
+    if (result.orderby && result.skiptoken)
+      throw Problem.internal.notImplemented({ feature: 'using $orderby and $skiptoken together' });
 
     return new QueryOptions(result);
   }
@@ -397,6 +402,11 @@ class QueryOptions {
       result.filter = query.$filter;
     if (query.$skiptoken != null)
       result.skiptoken = QueryOptions.parseSkiptoken(query.$skiptoken);
+    if (query.$orderby != null)
+      result.orderby = query.$orderby;
+
+    if (result.orderby && result.skiptoken)
+      throw Problem.internal.notImplemented({ feature: 'using $orderby and $skiptoken together' });
 
     return new QueryOptions(result);
   }

lib/util/problem.js

@@ -229,7 +229,7 @@ const problems = {
 
     // returned when we don't support certain kinds of odata filter expressions.
     unsupportedODataExpression: problem(501.4, (({ at, type, text }) => `The given OData filter expression uses features not supported by this server: ${type} at ${at} ("${text}")`)),
-    unsupportedODataField: problem(501.5, (({ at, text }) => `The given OData filter expression references fields not supported by this server: ${text} at ${at}`)),
+    unsupportedODataField: problem(501.5, (({ at, text }) => `The given OData filter expression references fields not supported by this server: ${text}${(at != null) ? ` at ${at}` : ''}`)),
     unsupportedODataExpandExpression: problem(501.6, (({ text }) => `The given OData expand expression is not supported by this server: "${text}". Currently, only "$expand=*" is supported.`)),
 
     invalidDatabaseConfig: problem(501.7, ({ reason }) => `The server's database configuration is invalid. ${reason}`),