Feature/spline 954 acid tx by wajda · Pull Request #1107 · AbsaOSS/spline

wajda · 2022-09-28T17:11:30Z

Fixes #954

Removed arangodb-AQL-functions module and moved the code to the arangodb-foxx-services
Moved custom AQL functions lifecycle management responsibility from the Admin CLI to the Foxx setup/teardown scripts.
Moved lineage ingestion functionality from the Producer service layer to the Foxx layer
Changed the lineage persistence logic according to the proposed application level ACID Tx protocol.

cerveada · 2022-10-04T08:38:25Z



-export function getExecutionEventFromEventKey(eventKey: DocumentKey): ExecutionEvent {
+export function getExecutionEventFromEventKey(eventKey: DocumentKey): Progress {


What is the reason behind this change ExecutionEvent -> Progress ?

Just eliminating duplicated type definition. More changes are yet to be made to the model as different parts of it have being moved from other modules and consolidated on the Foxx layer.

cerveada · 2022-10-04T08:56:57Z

@@ -0,0 +1,56 @@
+/*


What is the meaning of the d int the name?

I was struggling to find an understandable name for the directory and decided to just name it after the type of the files it contains. It's temporary here, I'm going to remove it as soon as it's promoted to the DefinitelyTyped project.

.d.ts - is a standard TypeScript type declaration file extension. In this case it's extra definition of the Foxx API that are missing form the latest version of the @types/arangodb package

cerveada · 2022-10-04T09:06:19Z

      .generateJs()

-    generatedJS should be {
+    generatedJS shouldEqual {


What is the reason for this change?

consistency. The other spec statements in the file use shouldEqual. Moreover to me it seems more appropriate and for the string comparison assertion, while be implies a predicate following it.

There seems to be a functional difference between equal and be.
AS I understand it be always works the same as == in Scala, whereas equal will use a user-defined implicit Equality if available in scope.

Maybe it would be better to use be as a default and equal only when we need the additional functionality?

https://www.scalatest.org/user_guide/using_matchers#checkingEqualityWithMatchers

We could. I don't see too much practical difference to be honest, but I'm fine with either. I more care about consistency though, so that the same logic of choice is applied throughout the whole code base. We can discuss it on CQC.

…o the 'tx.execute' method

…ery.LastResultPlaceholder"

… from arangodb-foxx-api module

… Remove AppTxBuilder.

…S based modules.

…eation, replace UUIDv4 TxId with ArangoDB autogen key.

…nge AQL func JS file name pattern to '*.func.js'

…in AQL queries, with the help of AQLGenHelper

…-services'

sonarqubecloud · 2022-10-10T15:04:56Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.2% Duplication

korel-san · 2022-10-11T09:30:04Z

+                        RETURN DISTINCT readEvent
+    `
+
+    Logger.debug(query)


@wajda debug

korel-san · 2022-10-11T09:30:50Z

+            FOR p IN allObservedEvents RETURN p
+    `
+
+    Logger.debug(query)


@wajda do you really want to leave it in the codebase?

korel-san · 2022-10-11T09:34:07Z

@@ -0,0 +1,337 @@
+/*


@wajda is it typings for the fs module?

if yes, I would recommend leaving @types/node in package.json and in the tsconfig.json
https://stackoverflow.com/questions/43048113/use-fs-in-typescript

I tried that but unfortunately it didn't work. ArangoDB for some reason uses a slightly different File system API.

The implementation tries to follow the CommonJS Filesystem/A/0 specification where possible.

Not sure how many specs for the FS module exist in the JavaScript world, but the one @types/node provides doesn't fit what ArangoDB is using. My intention for this type-def file was to replace it with a library one as soon as possible (either I find something existing, or make a PR to the @types/arangodb)

korel-san · 2022-10-11T09:46:23Z

@@ -0,0 +1,49 @@
+/*


@wajda I would recommend reorganizing the number of subfolders, due to the complexity and avoiding the Doorway Effect

Sure! Would you suggest anything specific? :)

dk1844

Read the code, have not found anything suspicious. But I must admit that I do not understand every bit of the code.

wajda force-pushed the feature/spline-954-acid-tx branch from bb321fe to 5039bcb Compare September 28, 2022 17:19

cerveada reviewed Oct 4, 2022

View reviewed changes

wajda added 28 commits October 4, 2022 20:07

spline #954 Add "counter" collection

5310b6b

spline #954 TxBuilder refactoring

1ab28bd

spline #954 refactoring: simplify tx queries

1520b9d

spline #954 add a custom _key to the 'progressOf' edge

f5dae9d

spline #954 minor: add parentheses

32c89ed

spline #954 refactoring: add a return type and an execution context t…

3345c28

…o the 'tx.execute' method

spline #954 refactoring: replace "Query.chainInput" flag with the "Qu…

cbb5fcc

…ery.LastResultPlaceholder"

spline #954 draft: AppTxBuilder

8b58606

spline #954 Producer API :: remove multi-events transaction.

0d90129

spline #954 Add FoxxPostTxBuilder

acf9d21

spline #954 POM: code formatting, removing redundant elements

217f6d4

spline #954 package.json: remove redundant dev dependencies

229ea86

spline #954 Add 'plans' and 'events' routes to Foxx

efb6f38

spline #954 + arangodb-foxx-api module

37d3e3f

spline #954 arangodb-foxx-services module to use TS model definitions…

3978c23

… from arangodb-foxx-api module

spline #954 arangodb-foxx-services Producer REST to delegate to Foxx.…

b24d040

… Remove AppTxBuilder.

spline #954 Foxx manifest: add more info about the service

7e4888b

spline #954 POM: propagate Maven version number to package.json for J…

274a789

…S based modules.

spline #954 Foxx: Store execution plan

2a1f782

spline #954 Foxx: Store execution event

6e4eac1

spline #954 log service execution time

ae847e1

spline #954 implement WRITE transaction

d8ab92a

spline #954 + naive rollback impl

b933517

spline #954 add READ transactions creation, fix WRITE transactions cr…

d5415e0

…eation, replace UUIDv4 TxId with ArangoDB autogen key.

spline #954 formatting and typing

3cb7ea2

spline #954 use READ transaction (partial impl)

4d58ea7

spline #954 fix lint errors

f5604a2

spline #954 fix lint errors

6ec248f

wajda added 8 commits October 4, 2022 20:07

spline #954 Admin: reinstall AQL functions on '--foxx-reinstall'. Cha…

92880f9

…nge AQL func JS file name pattern to '*.func.js'

spline #954 Add SPLINE::IS_VISIBLE_FROM_TX AQL function

d26c616

spline #954 Use IS_VISIBLE_FROM_TX AQL func

9f645fe

spline #954 Simplify tx related PRUNE & FILTER statements generation …

c32507c

…in AQL queries, with the help of AQLGenHelper

spline #954 Add Logger

586cec5

spline #954 Merge 'arangodb-aql-functions' module into 'arangodb-foxx…

10143aa

…-services'

spline #954 Fix manifest schema URL

ad605fe

spline #954 Addressing PR review, removing unneeded code

e583424

wajda force-pushed the feature/spline-954-acid-tx branch from e4a1838 to e583424 Compare October 4, 2022 18:08

spline #954 Fix forward lineage AQL

07bb430

wajda force-pushed the feature/spline-954-acid-tx branch from 2a9aa14 to 07bb430 Compare October 5, 2022 16:41

wajda added 5 commits October 5, 2022 20:59

spline #954 Fix ACID AQL code gen

ee4d995

spline #954 Add missing copyright header

0d26b72

spline #954 Add migration query

fbd4246

spline #954 POM: RAT plugin: exclude external source

09d34b5

spline #954 Foxx: Add Tx isolation to the getById() method

1162bf2

wajda force-pushed the feature/spline-954-acid-tx branch from 5a55284 to 1162bf2 Compare October 6, 2022 13:27

spline #954 Remove TODO

dc2ed5e

wajda marked this pull request as ready for review October 10, 2022 15:04

korel-san reviewed Oct 11, 2022

View reviewed changes

korel-san approved these changes Oct 11, 2022

View reviewed changes

dk1844 approved these changes Oct 11, 2022

View reviewed changes

wajda merged commit ab09c35 into develop Oct 14, 2022

wajda deleted the feature/spline-954-acid-tx branch October 14, 2022 15:41



		export function getExecutionEventFromEventKey(eventKey: DocumentKey): ExecutionEvent {
		export function getExecutionEventFromEventKey(eventKey: DocumentKey): Progress {

Conversation

wajda commented Sep 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wajda Oct 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Oct 10, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dk1844 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wajda commented Sep 28, 2022 •

edited

Loading

wajda Oct 13, 2022 •

edited

Loading