#4941 - Approved exception does not repeat - New JSON Lib #5010

andrewsignori-aot · 2025-07-31T20:25:47Z

Replaced the JSONPath with the jsonata lib to allow the filter and transformations that will be required once the new exceptions structure is in place.
Adapt the load-consolidate-data BPMN, and a few workflows that were using the JSONPath expression.

Important note about the expressions

Assuming the below JSON (array with a single item).

{ 
  "courseDetails": [
      {
          "courseCode": "code",
          "courseName": "name"
      }
  ]
}

Using the expression below

courseDetails.{
  "courseCode": courseCode
}

will produce the result as an object, not an array.

{
  "courseCode": "code"
}

To produce an array even when there is only one item, the expression should be as below.

courseDetails[].{
  "courseCode": courseCode
}

to produce

[
  {
    "courseCode": "code"
  }
]

Manual tests

Created a new 2025-26 full-time application including two parents (one self-declare and the other doesn't) and also an exception.
Created a change request to remove one of the parents.
Create a full-time application including dependents.
Created a 2025-26 part-time application.
Created a 2024-25 application and an appeal to add dependents.

Copilot

Pull Request Overview

This PR replaces the JSONPath library with the JSONata library to enable more advanced filtering and transformations needed for the new exceptions structure. This change allows for more powerful data manipulation capabilities in workflow expressions.

Key changes:

Replace JSONPath with JSONata library in package dependencies
Update all workflow BPMN files to use JSONata syntax instead of JSONPath expressions
Refactor utility functions to use JSONata's asynchronous evaluation API

Reviewed Changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
sources/packages/backend/package.json	Updates dependency from jsonpath to jsonata library
sources/packages/backend/apps/workers/src/utilities/jsonpath/jsonpath-utils.ts	Removes old JSONPath utility functions
sources/packages/backend/apps/workers/src/utilities/json/json-utils.ts	Adds new JSONata utility functions with async support
sources/packages/backend/apps/workers/src/utilities/index.ts	Updates export to use new json-utils instead of jsonpath-utils
sources/packages/backend/apps/workers/src/controllers/supporting-user/supporting-user.controller.ts	Updates to use JSONata syntax and async evaluation
sources/packages/backend/apps/workers/src/controllers/assessment/assessment.controller.ts	Updates to use async filterObjectProperties
sources/packages/backend/workflow/src/workflow-definitions/supporting-user-information-request.bpmn	Updates JSONPath expressions to JSONata syntax
sources/packages/backend/workflow/src/workflow-definitions/load-assessment-consolidated-data.bpmn	Updates all JSONPath expressions to JSONata syntax
sources/packages/backend/workflow/src/workflow-definitions/assessment-gateway-v2.bpmn	Updates JSONPath expression to JSONata syntax
sims.code-workspace	Adds jsonata to spell check dictionary and reorganizes entries

Files not reviewed (1)

sources/packages/backend/package-lock.json: Language not supported

Copilot · 2025-07-31T20:55:39Z

sources/packages/backend/apps/workers/src/utilities/json/json-utils.ts

+ * where the values for each property are the values
+ * resulting from the jsonata expressions execution.
+ */
+export async function filterObjectProperties(


The function processes filters sequentially in a for-of loop. Consider using Promise.all() to evaluate all JSONata expressions in parallel for better performance when dealing with multiple filters.

The expressions are expected to be evaluated synchronously.

andrewsignori-aot · 2025-07-31T21:34:36Z

.../route-controllers/student/_tests_/e2e/student.external.controller.searchStudent.e2e-spec.ts

    currentProgramYear = await db.programYear
      .createQueryBuilder("py")
-      .where("NOW() BETWEEN py.startDate AND py.endDate")
+      .where("CURRENT_DATE BETWEEN py.startDate AND py.endDate")


This test fails on the last day of a program year because it uses a date+time when executing the comparison between two date-only columns.
It would pass tomorrow 😄

github-actions · 2025-07-31T21:34:52Z

Backend Unit Tests Coverage Report

Totals
Statements:	21.65% ( 4093 / 18903 )
Methods:	9.67% ( 234 / 2419 )
Lines:	25.01% ( 3540 / 14155 )
Branches:	13.7% ( 319 / 2329 )

sonarqubecloud · 2025-07-31T21:35:12Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2025-07-31T21:35:15Z

E2E Workflow Workers Coverage Report

Totals
Statements:	73.53% ( 714 / 971 )
Methods:	72.57% ( 82 / 113 )
Lines:	75.81% ( 564 / 744 )
Branches:	59.65% ( 68 / 114 )

github-actions · 2025-07-31T21:37:42Z

E2E Queue Consumers Coverage Report

Totals
Statements:	86.43% ( 1548 / 1791 )
Methods:	84.62% ( 176 / 208 )
Lines:	88.75% ( 1278 / 1440 )
Branches:	65.73% ( 94 / 143 )

.../packages/backend/apps/workers/src/controllers/supporting-user/supporting-user.controller.ts

github-actions · 2025-07-31T21:38:33Z

E2E SIMS API Coverage Report

Totals
Statements:	73.79% ( 7269 / 9851 )
Methods:	71.51% ( 896 / 1253 )
Lines:	77.22% ( 5623 / 7282 )
Branches:	56.99% ( 750 / 1316 )

sh16011993

Great work @andrewsignori-aot 👍 Thanks for clarifying my questions.

dheepak-aot · 2025-07-31T23:50:01Z

sources/packages/backend/workflow/src/workflow-definitions/assessment-gateway-v2.bpmn

@@ -1,5 +1,5 @@
 <?xml version="1.0" encoding="UTF-8"?>
-<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:dc="http://www.omg.org/spec/DD/20100524/DC" xmlns:zeebe="http://camunda.org/schema/zeebe/1.0" xmlns:di="http://www.omg.org/spec/DD/20100524/DI" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:bioc="http://bpmn.io/schema/bpmn/biocolor/1.0" xmlns:color="http://www.omg.org/spec/BPMN/non-normative/color/1.0" xmlns:modeler="http://camunda.org/schema/modeler/1.0" xmlns:camunda="http://camunda.org/schema/1.0/bpmn" id="Definitions_1" targetNamespace="http://bpmn.io/schema/bpmn" exporter="Camunda Modeler" exporterVersion="5.35.0" modeler:executionPlatform="Camunda Cloud" modeler:executionPlatformVersion="8.6.0" camunda:diagramRelationId="f874f098-96b6-491d-81ec-ecf7a0d8c67a">
+<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:dc="http://www.omg.org/spec/DD/20100524/DC" xmlns:zeebe="http://camunda.org/schema/zeebe/1.0" xmlns:di="http://www.omg.org/spec/DD/20100524/DI" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:bioc="http://bpmn.io/schema/bpmn/biocolor/1.0" xmlns:color="http://www.omg.org/spec/BPMN/non-normative/color/1.0" xmlns:modeler="http://camunda.org/schema/modeler/1.0" xmlns:camunda="http://camunda.org/schema/1.0/bpmn" id="Definitions_1" targetNamespace="http://bpmn.io/schema/bpmn" exporter="Camunda Modeler" exporterVersion="5.37.0" modeler:executionPlatform="Camunda Cloud" modeler:executionPlatformVersion="8.6.0" camunda:diagramRelationId="f874f098-96b6-491d-81ec-ecf7a0d8c67a">


dheepak-aot · 2025-08-01T00:07:40Z

sources/packages/backend/apps/workers/src/controllers/assessment/assessment.controller.ts

        hasNOAApproval,
      );
-      const outputVariables = filterObjectProperties(
+      const outputVariables = await filterObjectProperties(


There is an e2e test for assessment consolidated data based on the JSON path expression. Not sure how it is passing.

The "$" at the beginning is compatible with the new framework. I can remove from this to avoid confusion in the future.

dheepak-aot

Great work finding the better way to parse json data and thanks for the sync up. One minor observation on E2E tests.

New lib updates

fe62e97

andrewsignori-aot requested a review from Copilot July 31, 2025 20:25

This comment was marked as outdated.

Sign in to view

andrewsignori-aot added 2 commits July 31, 2025 13:32

Pre review fixes

dc74d4d

Fixed array expressions

20a4420

andrewsignori-aot requested a review from Copilot July 31, 2025 20:55

Copilot AI reviewed Jul 31, 2025

View reviewed changes

andrewsignori-aot marked this pull request as ready for review July 31, 2025 20:58

andrewsignori-aot self-assigned this Jul 31, 2025

andrewsignori-aot added Camunda Workers Camunda Worflow Involves camunda workflow changes labels Jul 31, 2025

andrewsignori-aot added 2 commits July 31, 2025 14:27

Fix E2E test due to end of program year

81fead0

Fix E2E test

85c539b

andrewsignori-aot commented Jul 31, 2025

View reviewed changes

sh16011993 reviewed Jul 31, 2025

View reviewed changes

.../packages/backend/apps/workers/src/controllers/supporting-user/supporting-user.controller.ts Show resolved Hide resolved

dheepak-aot self-requested a review July 31, 2025 23:19

sh16011993 approved these changes Jul 31, 2025

View reviewed changes

dheepak-aot reviewed Jul 31, 2025

View reviewed changes

andrewsignori-aot enabled auto-merge August 1, 2025 00:01

dheepak-aot reviewed Aug 1, 2025

View reviewed changes

dheepak-aot approved these changes Aug 1, 2025

View reviewed changes

andrewsignori-aot added this pull request to the merge queue Aug 1, 2025

Merged via the queue into main with commit 135e634 Aug 1, 2025
22 checks passed

andrewsignori-aot deleted the feature/#4941-approve-exceptions-once-json-lib-replacement branch August 1, 2025 00:11

andrewsignori-aot temporarily deployed to DEV August 1, 2025 00:16 — with GitHub Actions Inactive

andrewsignori-aot temporarily deployed to DEV August 1, 2025 00:17 — with GitHub Actions Inactive

andrewsignori-aot had a problem deploying to DEV August 1, 2025 00:17 — with GitHub Actions Failure

andrewsignori-aot temporarily deployed to DEV August 1, 2025 00:17 — with GitHub Actions Inactive

andrewsignori-aot temporarily deployed to DEV August 1, 2025 00:19 — with GitHub Actions Inactive

#4941 - Approved exception does not repeat - New JSON Lib #5010

#4941 - Approved exception does not repeat - New JSON Lib #5010

Uh oh!

Conversation

andrewsignori-aot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Important note about the expressions

Manual tests

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

andrewsignori-aot Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

andrewsignori-aot Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 31, 2025

Backend Unit Tests Coverage Report

Uh oh!

sonarqubecloud bot commented Jul 31, 2025

Quality Gate passed

Uh oh!

github-actions bot commented Jul 31, 2025

E2E Workflow Workers Coverage Report

Uh oh!

github-actions bot commented Jul 31, 2025

E2E Queue Consumers Coverage Report

Uh oh!

Uh oh!

github-actions bot commented Jul 31, 2025

E2E SIMS API Coverage Report

Uh oh!

sh16011993 left a comment

Choose a reason for hiding this comment

Uh oh!

dheepak-aot Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

dheepak-aot Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewsignori-aot Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

dheepak-aot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

andrewsignori-aot commented Jul 31, 2025 •

edited

Loading

dheepak-aot Aug 1, 2025 •

edited

Loading