Native List Reads

Native list reads

The native list endpoints /documents, /projects, /updates and /notifications read from Cosmos or the index. The mirror lists and the mirror datasets of GET /search read from Cosmos only. This page owns their store choice, paging, sort keys and filters.

GET /search routing and query building: Search-Query-Construction. Index fields: Search-Index-Reference. Keyword search inside documents: Deep-Search-Queries.

Native document lists choose a store

GET /documents reads one project's plain list, and an ids= read, from Cosmos. A plain list is a project or ids parameter with no q, sort, displayNameContains, isFeatured, legislation or date filter. Every other list pages in the index, cross-project lists without q included.

The Cosmos list reads by overfetch-and-slice against a repository that clamps maxItemCount to 1000. The container indexes neither datePosted nor displayName, so it cannot page or order a large project.

Index-path list metadata is stale by up to the indexer's PT5M pass. Visibility is not, because setDocumentPublished and the project cascade write ACLs straight into the index. Update-form images are left out on both paths.

Native GET /documents

Paging uses page (from 1), pageSize and count=true. The index returns the page and @odata.count in one request, and x-total-count is that count.

On the index path these are a 400:

• a pageSize above 500
• a filter or contains filter with no index column, because the Cosmos re-read does not apply CONTAINS and the filter would vanish
• any flag parameter (see "Native list filters, shared by every kind")

The index returns only ids. The rows are re-read from Cosmos by id, which applies each record's own field dials, and put back in index order. A stale index makes a page short; it does not leak.

Order follows orderby, always ending in id so pages do not overlap:

Request	Order
sort named	that key's index column (below), then id
q, no sort	relevance (search.score() desc), then id
no q, no sort	datePosted desc, then id

Sort keys take a - prefix for descending: datePosted, displayName (column displayNameSort), sortOrder, type, milestone, projectPhase, documentAuthorType, isPublished, legislation, isFeatured, documentAuthor (column documentAuthorSort). Most rows hold sortOrder 0, so a sortOrder sort breaks ties by datePosted desc, then displayNameSort asc, then id. Any other key is a 400.

Filter	Takes
project	alias of projectId; several values
typeId, milestoneId, projectPhaseId, documentAuthorTypeId	several values each
legislation	several values, each matching ^\d{1,4}$
isFeatured	one value, true or false
datePostedStart, datePostedEnd	one YYYY-MM-DD day each; the end day is included
displayNameContains	a name match, see "Contains filters"

These values reach the index as unquoted literals. Each is checked before any OData is built, and a bad value is a 400, because legislation=1) or (true would otherwise close the ACL clause and expose @odata.count.

The searchVis clause is sent only when q is present. Its definition: Search-Index-Reference "searchVis on documents".

Native list filters, shared by every kind

Every native list parses its query with parseNativeQuery (src/helpers/native-query.js). These are a 400:

• a parameter the kind does not declare
• a control parameter (page, pageSize, count, sort, q, ids) or single-value filter given twice
• a sort key outside the kind's list, cannot sort on: <key>

A sort or filter on a field the caller cannot see is a 400 before any read. A caller below staff who filters on a staff-only field gets [<entity>] cannot filter on a field this caller cannot see: <field>. Every sort key has an included path in the container's index policy (azure/modules/cosmos-nosql.bicep), because Cosmos cannot serve an ORDER BY on an unindexed path.

Shape	Wire	Rule
exact	field=value	A repeatable filter takes several values, at most 100, matched as in. Others take one.
range	fieldStart, fieldEnd	One YYYY-MM-DD calendar day each, UTC. Start is from 00:00:00.000, end to 23:59:59.999, so the end day is included. A bad day is a 400.
contains	fieldContains	Case-insensitive substring. Trimmed; an empty value is ignored; over 200 characters is a 400.
presence	hasField	true or false only, else 400. true means the field holds a value other than null or ''; false is the exact complement.
flag	field	true or false only, else 400 <field> takes only true or false. false also matches a row where the field is missing.

Every clause honours the record's own dial on the field. A record that hides a field below the caller's level never matches a filter on it, so count=true cannot reveal that value.

Contains filters

On a Cosmos read the filter is CONTAINS(c.<field>, text, true), a substring match. On an index path it is the word match in Search-Query-Construction "and[nameContains] matches a fragment of the name".

Two filters use the index match. Documents displayNameContains matches displayName, documentFileName and fileNameTokens; projects nameContains with q matches name and nameTokens.

A filter the documents index has no column for is a 400, cannot match text in <field> here.

Lists with q=

On /projects, /updates and /notifications, a list with q reads at most 250 ranked ids from the index and pages them in memory, so its total is capped at 250. Filters with an index column also narrow the index request. Every filter, range, contains and presence clause is then applied again in the Cosmos re-read of those ids, with the dials.

A list without q is read from Cosmos alone. /documents is described above.

Native GET /projects

Default order name asc. Sort keys: name, dateUpdated, proponentName, projectType, sector, region, currentPhaseName.name, eacDecisionSort. Phase sorts on the phase's name, not its id.

eacDecisionSort holds eacDecision.name. It is a separate field so that a project with no decision does not gain an eacDecision object that readers show as a decision. src/scripts/backfill-project-sort-fields.js fills it on rows written earlier.

Filter	Takes
projectType, region, projectState, proponentName, sector, eacDecision, currentPhaseName, CEAAInvolvement	repeatable
regionalDistrict, municipality, electoralDistrict	one value
decisionDateStart, decisionDateEnd, dateUpdatedStart, dateUpdatedEnd	ranges
nameContains	contains, on name

The last three repeatable filters name the List ref and match its _id. sector is an exact match. projectType matches both stored spellings (see Search-Index-Reference on PROJECT_TYPE_SPELLINGS). With q, nameContains uses the index word match and the Cosmos re-read of the ranked ids still applies the substring check. Without q it is a substring match on Cosmos.

pcp is repeatable and filters on the project's comment periods: open, upcoming, closed or pending. pending is a period that starts within the next 7 days. The result holds projects with a period in any named status that the caller may read. Another value is a 400, pcp takes open, upcoming, closed, pending. GET /search/counts does not accept pcp.

Cosmos drops a row that lacks its ORDER BY field, so upsert writes every sort field it can fill (sortFieldFills, src/repositories/projects.js):

• proponentName, projectType, sector and region as null when absent
• dateUpdated from dateAdded as an ISO time, or null when dateAdded is absent or does not parse
• currentPhaseName as {_id: null, name: null}

A field the row already holds, null included, is left alone.

Native GET /updates

Default order dateAdded desc. Sort keys: dateAdded, dateUpdated, publishDate, headline, type, pinned, status. The mirror stores pinned and complianceAndEnforcement as booleans (false when Eagle sets none) and status as null when there is none. src/scripts/backfill-update-sort-fields.js fills rows written earlier.

Filter	Takes
project	alias of projectId, the Eagle project id as stored; repeatable, at most 100 ids
type	repeatable
dateAddedStart, dateAddedEnd	one YYYY-MM-DD day each; the end day is included
complianceAndEnforcement, hasDocumentUrl	true or false

A null documentUrl and an empty string both count as no document. Callers below staff see published rows only.

Native GET /notifications

Default order notificationReceivedDate desc. Sort keys: notificationReceivedDate, name, type, region, decision, proponent, subType.

type, region and decision are repeatable. pcp takes one value, and none is a real value. A notification has no project axis, so role ACL alone decides which rows a caller sees.

Eagle mirror datasets, _schemaName and x-total-count

GET /search accepts dataset values User, Group, Inspection, InspectionElement and InspectionItem. These are Cosmos list reads of the Eagle mirrors under the caller's access, not index queries. They take no keywords: q answers 400.

• User has no project axis, so a project filter answers no rows.
• and[inspection] filters InspectionElement and InspectionItem by the parent inspection's Eagle id.
• and[element] filters InspectionItem by its element.

dataset=Item fetches one record by Eagle id. It requires _id and _schemaName and takes nothing else. _schemaName is one of the model names above, or Comment, CommentPeriod, ProjectNotification or RecentActivity. It answers a bare array of zero or one record, not the search envelope. _schemaName on any other dataset answers 400.

and[eaoStatus] applies to dataset=Comment and takes Pending, Published, Rejected or Deferred, comma-separated for several. Any other value answers 400.

Every search response carries an x-total-count header with the measured total, the same number as meta[0].searchResultsTotal. On dataset=Item it is the length of the array. The header is exposed to CORS, and HEAD /search returns the headers with no body.

Reads of the user, group and inspection mirrors

These routes take passiveAuthMiddleware.strict and credentialsMiddleware, the same as GET /documents. strict answers a rejected Bearer token with its 401 instead of serving the anonymous view. GET /search takes the lenient passiveAuthMiddleware. Each list route has a by-id route at /:id.

Route	Filter
GET /users	none
GET /groups	project
GET /inspections	project
GET /inspection-elements	inspection
GET /inspection-items	inspection, element

Each sort key is in the container's index policy.

Route	Sort keys
GET /comment-periods	dateStarted, dateCompleted, isPublished
GET /organizations	name, companyType, companyLegal
GET /users	displayName, orgName, email, phoneNumber
GET /groups	name

companyLegal on organizations and email and phoneNumber on users are staff-only. A caller below staff who sorts on one gets a 400, for example [users] cannot filter on a field this caller cannot see: email.

By-id reads of groups take ?project=. The other by-id reads of inspection kinds take ?inspection=, the partition key.