TDI-17/TDI-34 File parse by chelsea-EYDS · Pull Request #12 · bcgov/PaymentCommonComponent

chelsea-EYDS · 2022-11-19T00:12:13Z

What has changed:

Added class definitions for TDI17/TDI34 files.
Added sample files to a root directory (could not access sharepoint files from the terminal)
Added Makefile commands to parse local. Parsing requires a file path and file type, either passed in the event, or, set as env vars. Env vars are set for local dev in makefile cmd.

apps/backend/src/tdi-parser/tdi-parser.service.ts

apps/backend/package.json

apps/backend/src/app.controller.ts

apps/backend/src/dto/sample.TXT

vyasworks · 2022-11-22T23:26:20Z

apps/backend/src/lambdas/parseTDI.ts

+import { TDI17Record } from '../tdi-parser/tdi17/TDI17Record';
+import { TDI17Header } from '../tdi-parser/tdi17/TDI17Header';
+import { TDI17Details } from '../tdi-parser/tdi17/TDI17Details';
+import { TDI17Trailer } from '../tdi-parser/tdi17/TDI17Trailer';
+import { TDI34Record } from '../tdi-parser/tdi34/TDI34Record';
+import { TDI34Header } from '../tdi-parser/tdi34/TDI34Header';
+import { TDI34Details } from '../tdi-parser/tdi34/TDI34Details';
+import { TDI34Trailer } from '../tdi-parser/tdi34/TDI34Trailer';
+import { TDI34Enum } from './const/tdi34.const';
+import { TDI17Enum } from './const/tdi17.const';


could be imported as a single module for 17 and 34 each.

vyasworks · 2022-11-22T23:30:22Z

apps/backend/src/lambdas/parseTDI.ts

+  appLogger.log({ event });
+  appLogger.log({ context });
+
+  const filetype = event?.type ?? process.env.TDI_TYPE;


Suggestion - This line should throw an error if the event type does not exist. Defaulting should not be an option

vyasworks · 2022-11-22T23:36:20Z

apps/backend/src/lambdas/parseTDI.ts

+    const outputFile = Buffer.from(JSON.stringify(output));
+
+    appLogger.log(outputFile);
+    await s3manager.putObject(


I would prefer this to have it's own try catch.

vyasworks · 2022-11-22T23:42:50Z

apps/backend/src/lambdas/parseTDI.ts

+  const header = TDI_17 ? new TDI17Header({}) : new TDI34Header({});
+  const trailer = TDI_17 ? new TDI17Trailer({}) : new TDI34Trailer({});


I don't think we want to store the header and trailer for TDI files.

The transactions line items is all we want.

vyasworks · 2022-11-22T23:45:38Z

apps/backend/src/lambdas/parseTDI.ts

+  const record = TDI_17 ? new TDI17Record({}) : new TDI34Record({});
+
+  record.details = TDI_17 ? tdi17RecordDetails : tdi34RecordDetails;
+  record.trailer = trailer;
+  record.header = header;
+
+  return {
+    header: record.header.resource,
+    details: record.details.map((itm: { resource: unknown }) => itm.resource),
+    trailer: record.trailer.resource,
+  };


we need only the array of details.

vyasworks · 2022-11-22T23:46:56Z

apps/backend/src/tdi-parser/tdi17/TDI17Details.ts

@@ -0,0 +1,200 @@
+import {


create folder level index files and export from them.

vyasworks · 2022-11-22T23:48:04Z

apps/backend/src/tdi-parser/tdi17/TDI17Header.ts

@@ -0,0 +1,121 @@
+import {


We could skip the header and trailer as we don't have any use for it YET. So we can leave it here and use if we need it later.

Okay sounds good. I thought we might want the file header as there is the file creation date. So for now I will add an index, and export only the details for 17/34?

Yes.

The date seems to also be in each of the transaction items, so it might just be redundant.
Is the header date different or providing us any additional context? 🤔

vyasworks · 2022-11-23T16:26:01Z

Makefile

+# Add Local Tdi Files
+# ===================================
+
+put-local-tdi17:


the make command can also accept paths

https://stackoverflow.com/a/2826069

example: https://github.com/bcgov/hcap/blob/master/Makefile#L56

Usage: https://github.com/bcgov/hcap#to-seed-the-database-with-data-in-an-excel-sheet

vyasworks · 2022-11-23T16:29:14Z

apps/backend/src/lambdas/parseTDI.ts

+
+    await uploadParsedTDI(TDI_FILE, s3manager, output, appLogger);
+  } catch (e) {
+    appLogger.error(e);


we should exit the handler here gracefully.

or pass the error back to the root fn and exit there

It is also okay to not handle it because we will see the error in the console, but more logs and structure will help debugging in the future

vyasworks · 2022-11-23T16:32:34Z

apps/backend/src/tdi-files/const.ts

+  return {
+   TYPE: fileType,
+   BUCKET:  'bc-pcc-data-files-local',
+   LOCALSTACK_PATH:  `${fileType.toLowerCase()}/${fileType.toUpperCase()}.TXT`,


to be removed all together for providing flexible testing

vyasworks · 2022-11-23T16:33:14Z

apps/backend/src/tdi-files/const.ts

+export const generateEnum =(fileType: string)=> {
+  return {
+   TYPE: fileType,
+   BUCKET:  'bc-pcc-data-files-local',


Bucket name needs to change by ENV

apps/backend/src/lambdas/parseTDI.ts

vyasworks · 2022-11-24T19:18:23Z

apps/backend/package.json

+    "parseTDI17": "NODE_ENV=local RUNTIME_ENV=local LAMBDA_ARG=TDI17 ts-node ./scripts/handler.ts",
+    "parseTDI34": "NODE_ENV=local RUNTIME_ENV=local LAMBDA_ARG=TDI34 ts-node ./scripts/handler.ts"


ts-node ./scripts/handler.ts console.log(handler.({type: "TDI34", path: "pcc-bc-xxxx-local"}))

something like this

vyasworks · 2022-11-24T19:18:54Z

apps/backend/scripts/handler.ts

@@ -0,0 +1,7 @@
+import {handler} from '../src/lambdas/parseTDI'


Proabbaly dont need this file

vyasworks · 2022-11-24T19:21:09Z

Makefile

+# ===================================
+
+parse-local-tdi17:
+	cd './apps/backend' && yarn run parseTDI17


node -e 'require("./lambdas/parseTDI").handler({type: "TDI17", path: "/xasxasx/xasxas/xasxas"})'"

chelsea-EYDS requested a review from vyasworks November 19, 2022 00:12

chelsea-EYDS force-pushed the tdiparse branch from 96bc0e0 to eee8b50 Compare November 19, 2022 00:14

vyasworks reviewed Nov 21, 2022

View reviewed changes

apps/backend/src/tdi-parser/tdi-parser.service.ts Outdated Show resolved Hide resolved

chelsea-EYDS changed the title ~~TDI-17 File parse~~ TDI-17/TDI-34 File parse Nov 22, 2022

chelsea-EYDS marked this pull request as draft November 22, 2022 20:13

chelsea-EYDS force-pushed the tdiparse branch 6 times, most recently from ac3d1dc to f8e7b01 Compare November 22, 2022 20:45

chelsea-EYDS marked this pull request as ready for review November 22, 2022 20:45

chelsea-EYDS force-pushed the tdiparse branch 2 times, most recently from 93384f1 to e778ec8 Compare November 22, 2022 21:22

vyasworks suggested changes Nov 22, 2022

View reviewed changes

chelsea-EYDS force-pushed the tdiparse branch from 4b06182 to 1d9b778 Compare November 23, 2022 02:13

vyasworks reviewed Nov 23, 2022

View reviewed changes

chelsea-EYDS requested a review from vyasworks November 23, 2022 21:31

chelsea-EYDS commented Nov 24, 2022

View reviewed changes

apps/backend/src/lambdas/parseTDI.ts Show resolved Hide resolved

vyasworks reviewed Nov 24, 2022

View reviewed changes

chelsea-EYDS added 2 commits November 24, 2022 17:43

TDI parsing

7e9dbc4

removed handler and scripts dir

975c833

chelsea-EYDS force-pushed the tdiparse branch from ea473ce to 975c833 Compare November 25, 2022 01:44

chelsea-EYDS requested a review from vyasworks November 25, 2022 01:44

vyasworks approved these changes Nov 25, 2022

View reviewed changes

chelsea-EYDS merged commit b14f944 into main Nov 25, 2022

chelsea-EYDS deleted the tdiparse branch January 10, 2023 23:53

		const header = TDI_17 ? new TDI17Header({}) : new TDI34Header({});
		const trailer = TDI_17 ? new TDI17Trailer({}) : new TDI34Trailer({});

		"parseTDI17": "NODE_ENV=local RUNTIME_ENV=local LAMBDA_ARG=TDI17 ts-node ./scripts/handler.ts",
		"parseTDI34": "NODE_ENV=local RUNTIME_ENV=local LAMBDA_ARG=TDI34 ts-node ./scripts/handler.ts"

		@@ -0,0 +1,7 @@
		import {handler} from '../src/lambdas/parseTDI'

Conversation

chelsea-EYDS commented Nov 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chelsea-EYDS commented Nov 19, 2022 •

edited

Loading