Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade domain model to accommodate attribute level lineage requirements #677

Closed
wajda opened this issue May 21, 2020 · 2 comments
Closed

Upgrade domain model to accommodate attribute level lineage requirements #677

wajda opened this issue May 21, 2020 · 2 comments
Assignees
@wajda
Labels
component: Persistence component: Producer
Milestone
0.6.0

Comments

@wajda
Copy link
Contributor

wajda commented May 21, 2020

Introduce schema, attribute and expression nodes.
@wajda wajda added this to the 0.6.0 milestone May 21, 2020
@wajda wajda self-assigned this May 21, 2020
@wajda wajda mentioned this issue May 21, 2020
Merged
wajda added a commit that referenced this issue Jun 25, 2020
@wajda
spline #677 Upgrade persistence model to accommodate attribute level …
e3fbccf 
…lineage requirements
wajda added a commit that referenced this issue Jun 25, 2020
@wajda
spline #677 Upgrade persistence model to accommodate attribute level …
648d70b 
…lineage requirements
wajda added a commit that referenced this issue Jun 26, 2020
@wajda
spline #677 Upgrade persistence model to accommodate attribute level …
7007f96 
…lineage requirements
wajda added a commit that referenced this issue Jun 26, 2020
@wajda
spline #677 Upgrade persistence model to accommodate attribute level …
caadc32 
…lineage requirements
wajda added a commit that referenced this issue Jun 26, 2020
@wajda
spline #677 Upgrade persistence model to accommodate attribute level …
cefe2cf 
…lineage requirements
wajda added a commit that referenced this issue Sep 23, 2020
@wajda
spline #677 Upgrade persistence model to accommodate attribute level …
d813e03 
…lineage requirements
wajda added a commit that referenced this issue Oct 8, 2020
@wajda
spline #677 Use Array instead of Seq in a generic type param definiti…
3ab7c83 
…on in the model entities, for correct Swagger generic type generation.
wajda added a commit that referenced this issue Oct 8, 2020
@wajda
spline #677 Upgrade ReDoc CLI
a2414e4
wajda added a commit that referenced this issue Oct 9, 2020
@wajda
spline #677 Upgrade rest-api-doc-generator to ver 1.0.1
35bd8f2
@pratapmmmec
Copy link

@wajda Wanna check if this model will give us the capability to build forward lineage in future without need of tweaking persistence model again. If yes, then I am assuming we only need to build API and UI to enable forward lineage in Spline.

@wajda
Copy link
Contributor Author

wajda commented Oct 20, 2020

This is still a work-in-progress feature, but yes, we are going there.
Spline 0.6 will come with a attribute level lineage support on the persistent model level. It should give a solid tool to create/query different representation of linage with different level of details depending on the use-case.
We'll also change Consumer API accordingly to expose those new abilities over the REST API and subsequently the UI

wajda added a commit that referenced this issue Nov 24, 2020
@wajda
spline #677 Upgrade persistence model to accommodate attribute level …
bf6c347 
…lineage requirements
wajda added a commit that referenced this issue Nov 24, 2020
@wajda
spline #677 Use Array instead of Seq in a generic type param definiti…
862119b 
…on in the model entities, for correct Swagger generic type generation.
wajda added a commit that referenced this issue Nov 24, 2020
@wajda
spline #677 Upgrade ReDoc CLI
77389b9
wajda added a commit that referenced this issue Nov 24, 2020
@wajda
spline #677 Upgrade rest-api-doc-generator to ver 1.0.1
abf724a
@wajda wajda changed the title Upgrade persistence model to accommodate attribute level lineage requirements Upgrade domain model to accommodate attribute level lineage requirements Nov 27, 2020
@wajda wajda mentioned this issue Nov 27, 2020
Closed
wajda added a commit that referenced this issue Nov 30, 2020
@wajda
spline #677 Upgrade persistence model to accommodate attribute level …
a8f7af5 
…lineage requirements
wajda added a commit that referenced this issue Nov 30, 2020
@wajda
spline #677 Use Array instead of Seq in a generic type param definiti…
67e1451 
…on in the model entities, for correct Swagger generic type generation.
wajda added a commit that referenced this issue Nov 30, 2020
@wajda
spline #677 Upgrade ReDoc CLI
034d74b
wajda added a commit that referenced this issue Nov 30, 2020
@wajda
spline #677 Upgrade rest-api-doc-generator to ver 1.0.1
847ba66
wajda added a commit that referenced this issue Nov 30, 2020
@wajda
spline #677 Upgrade rest-api-doc-generator to ver 1.0.2
ae8bb06
wajda added a commit that referenced this issue Nov 30, 2020
@wajda
spline #677 Replace "Operation.expression" with "ExpresionRef in params"
d2a1bf0
wajda added a commit that referenced this issue Nov 30, 2020
@wajda
spline #677 Replace "Operation.expression" with "ExpresionRef in params"
09f2fd6
wajda added a commit that referenced this issue Dec 10, 2020
@wajda
spline #677 Upgrade persistence model to accommodate attribute level …
4a75213 
…lineage requirements
wajda added a commit that referenced this issue Feb 16, 2021
@wajda
spline #677 Addressing PR comments
0ec6fe7
wajda added a commit that referenced this issue Feb 16, 2021
@wajda
spline #677 Addressing PR comments - remove unnecessary closure
0f5078d
wajda added a commit that referenced this issue Feb 16, 2021
@wajda
spline #677 Addressing PR comments - simplifying pattern-matching
6b568af
wajda added a commit that referenced this issue Feb 16, 2021
@wajda
spline #677 Addressing PR comments - improve AttrOrExprRef javadoc
1b7d341
wajda added a commit that referenced this issue Feb 19, 2021
@wajda
spline #677 Fix deserialization issue + scala-graph binary incompatib…
fe0569c 
…ility
wajda added a commit that referenced this issue Feb 19, 2021
@wajda
spline #677 pom.xml: remove unused properties
7fd5725
wajda added a commit that referenced this issue Feb 26, 2021
@wajda
spline #677 Consumer REST: Fix lineage-overview
d689e2f
wajda added a commit that referenced this issue Feb 26, 2021
@wajda
spline #677 Persistence model converter: Fix "Uses" edges extraction
6ee9019
wajda added a commit that referenced this issue Feb 28, 2021
@wajda
spline #677 Producer REST: Fix "Uses" type
8e67785
wajda added a commit that referenced this issue Mar 1, 2021
@wajda
spline #677 Persistence model: ++ expression arity and JSONPath
a43c0ea
wajda added a commit that referenced this issue Mar 1, 2021
@wajda
spline #677 Consumer REST: Fix "lineage-detailed"
cff3a2d
wajda added a commit that referenced this issue Mar 2, 2021
@wajda
spline #677 Consumer REST: Remove redundant http request parameter
f7fcb54
wajda added a commit that referenced this issue Mar 2, 2021
@wajda
spline #677 Fix resolving attribute dependencies (derivesFrom edge …
18bd9bf 
…type)
wajda added a commit that referenced this issue Mar 3, 2021
@wajda
spline #677 AQL: Add missing collections to the "WITH" statement requ…
586abcd 
…ired for clustered ArangoDB setup
wajda added a commit that referenced this issue Mar 3, 2021
@wajda
spline #677 Consumer REST: Fix "lineage-detailed" (add missing attrib…
b31153d 
…utes to extra)
wajda added a commit that referenced this issue Mar 3, 2021
@wajda
spline #677 Fix velocypack serialization (add missing default constr …
ca9e5ff 
…+ replace Set with Seq)
wajda added a commit that referenced this issue Mar 4, 2021
@wajda
spline #677 Fix attr lineage path in operation graph
a11ca44
wajda added a commit that referenced this issue Mar 4, 2021
@wajda
spline #677 Fix attr usage path in operation graph
a06221d
wajda added a commit that referenced this issue Mar 4, 2021
@wajda
spline #677 Fix attribute order in operation schemas
e15b793
wajda added a commit that referenced this issue Mar 4, 2021
@wajda
spline #677 Fix attr impact path in operation graph
cecbac7
wajda added a commit that referenced this issue Mar 4, 2021
@wajda
spline #677 AQL: Add missing collections to the "WITH" statement requ…
3d1aaf9 
…ired for clustered ArangoDB setup
wajda added a commit that referenced this issue Mar 4, 2021
@wajda
spline #677 Move DBVersion collection definition from NodeDef to the …
8bedc16 
…CollectionDef, as it's not a node
wajda added a commit that referenced this issue Mar 4, 2021
@wajda
spline #677 Remove attribute search functionality from the Consumer R…
381d686 
…EST API
wajda added a commit that referenced this issue Mar 5, 2021
@wajda
spline #677 Remove attribute search functionality from the Consumer R…
66f311d 
…EST API
wajda added a commit that referenced this issue Mar 5, 2021
@wajda
spline #677 Add attribute "name" to the attribute lineage Consumer RE…
97ddfe2 
…ST model
wajda added a commit that referenced this issue Mar 5, 2021
@wajda
spline #677 Add type aliases for entity IDs for better readability
3891c74
wajda added a commit that referenced this issue Mar 5, 2021
@wajda
spline #677 Introduce ExpressionRepository for fetching expression re…
7932eea 
…lated metainfo
cerveada added a commit that referenced this issue Mar 8, 2021
@wajda @cerveada
Feature/spline 677 persistence (#822)
abb6240 
* spline #677 Upgrade persistence model to accommodate attribute level lineage requirements

* spline #677 Use Array instead of Seq in a generic type param definition in the model entities, for correct Swagger generic type generation.

* spline #677 Upgrade ReDoc CLI

* spline #677 Replace "Operation.expression" with "ExpresionRef in params"

* spline #677 REST: Return 400 if incorrect content is received

* spline #677 Producer REST: Fix deserialization issues

* spline #677 Log failed AQL queries

* spline #677 Update "find operation by ID" query

* spline #677 Move a part of the persistent model that belongs to the Consumer layer to that layer.

* spline #677 Service layer is ready for test

* spline #677 Fix swagger gen

* spline #677 Fix ExecutionPlan deserialization

* spline #677 Fix NoSuchElementException. Add order to the "follows" and "computedBy" edges

* spline #677 upgrade 'arangodb-java-driver' and 'velocypack-module-scala`

* spline #677 Fix unique constraint violation in DataSources

* spline #677 add persistent attribute and expressions details

* spline #677 addressing TODOs: rename _type to `type`

* spline #677 addressing TODOs: implement CachingConverter.keyOf where applicable. Also remove unused methods and types

* spline #677 addressing TODOs: replace 'var' in 'RecursiveSchemaFinder' with mixing 'CachingConverter' trait

* spline #677 addressing TODOs: remove explicit side-effect from the operation output converter. Also fix some bugs along the way.

* spline #677 addressing TODOs: replace "optionally" with "having"

* spline #677 fix topological sort for length < 2

* spline #677 addressing TODOs: AQL: replace string concat with var binding

* spline #677 addressing TODOs: deduplicate

* spline #677 addressing TODOs: simplify collection operations

* spline #677 addressing TODOs: rearrange methods

* spline #677 addressing TODOs + refactoring

* spline #677 upgrade Commons to 0.0.21

* spline #677 addressing todos

* spline #677 typo

* spline #677 addressing todos

* spline #677 update javadoc

* spline #677 addressing todos

* spline #677 typo

* spline #677 remove unused DAGTraversals

* spline #677 Upgrade to Commons 0.0.22. Remove unnecessary ErrorMsg class

* Update producer-model-mapper/src/main/scala/za/co/absa/spline/producer/modelmapper/v1/spark/SparkSplineObjectConverter.scala

Co-authored-by: Adam Cervenka <adam.cervenka@absa.africa>

* spline #677 Addressing PR comments: removing unused stuff, fix code style

* spline #677 Addressing PR comments

* spline #677 Addressing PR comments - remove unnecessary closure

* spline #677 Addressing PR comments - simplifying pattern-matching

* spline #677 Addressing PR comments - improve AttrOrExprRef javadoc

* spline #677 Fix deserialization issue + scala-graph binary incompatibility

* spline #677 pom.xml: remove unused properties

* spline #677 Consumer REST: Fix lineage-overview

* spline #677 Persistence model converter: Fix "Uses" edges extraction

* spline #677 Producer REST: Fix "Uses" type

* spline #677 Persistence model: ++ expression arity and JSONPath

* spline #677 Consumer REST: Fix "lineage-detailed"

* spline #677 Consumer REST: Remove redundant http request parameter

* spline #677 Fix resolving attribute dependencies (`derivesFrom` edge type)

* spline #677 AQL: Add missing collections to the "WITH" statement required for clustered ArangoDB setup

* spline #677 Consumer REST: Fix "lineage-detailed" (add missing attributes to extra)

* spline #677 Fix velocypack serialization (add missing default constr + replace Set with Seq)

* spline #677 Fix attr lineage path in operation graph

* spline #677 Fix attr usage path in operation graph

* spline #677 Fix attribute order in operation schemas

* spline #677 Fix attr impact path in operation graph

* spline #677 AQL: Add missing collections to the "WITH" statement required for clustered ArangoDB setup

* spline #677 Move DBVersion collection definition from NodeDef to the CollectionDef, as it's not a node

* spline #677 Remove attribute search functionality from the Consumer REST API

* spline #677 Add attribute "name" to the attribute lineage Consumer REST model

* spline #677 Add type aliases for entity IDs for better readability

* spline #677 Introduce ExpressionRepository for fetching expression related metainfo

Co-authored-by: Adam Cervenka <adam.cervenka@absa.africa>
@wajda wajda closed this as completed Mar 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wajda wajda

Labels
component: Persistence component: Producer
Projects
Spline
Status: Done
Milestone
0.6.0
Development

No branches or pull requests
2 participants
@wajda @pratapmmmec