Some UI issues on Spline UI web tool 0.3.9

[iamgzy]

There are two major concerns while running Spline UI tool 0.3.9 to establish lineage information

1. We ran a long and big spark job and turns out the icons and boxes on Lineage graph are very much squeezed like some icons overlapped on each other. I think may be a parameter configuration matter for the Angular modules to keep space between the icons on horizontal level ?
2. No search function for column level information. Basically the search is only per output but when trying to find information on a specific column user has to click on the projections/filters/agg to find it

Saw a new version of Spline UI Web 0.4.0 :) Thank you for the incremental updates, just curious:
Is Spline UI Web jar 0.4.0 compatible to Spline 0.3.9 and may resolve above issue?

Thanks

[wajda]

Hi @iamgzy,
Thanks for the report.

1. It would be interesting to see a screenshot of that long and big lineage :)
   In 0.3 UI we used vis.js library for graph rendering. There are some tweaking knobs but not much. Especially for the hierarchical layout we found it to be difficult to create a decent one-for-all setup.
   We don't support 0.3 UI anymore, but I would encourage you to try to play with the config yourself if you feel like. It's always easier when you have good sample data to test on. I'll be happy to merge your PR if you come up with a fix.
   There are two graph components - one for the lineage overview and another for the detailed lineage. Both have slightly different settings.

• https://github.com/AbsaOSS/spline/blob/release/0.3/web/ui/src/app/dataset/lineage-overview/vis-options.ts

• https://github.com/AbsaOSS/spline/blob/release/0.3/web/ui/src/app/lineage/graph/vis-options.ts

  You can easily run the UI in a dev mode locally and point it to the running Spline Web server (for REST), so you could make changes in code and see immediate result. Let me know if you need more help on this.

2. What kind of information would you like to search? Can you describe a more detailed use case?
   The search topic is discussed from time to time, so I think it will gain more attention in next releases.

Regarding the Spline 0.4 backward compatibility, the answer is yes and no. It is not compatible directly, meaning that you cannot run say Spline 0.4 UI on the existing 0.3 Mongo database.
Spline 0.4 has a brand new architecture, API and everything. But we have created a migration CLI utility, so all your existing lineage data can be retained. You are welcome to test it :)
https://github.com/AbsaOSS/spline/tree/develop/migrator-tool
The detailed instruction is coming, but it's a simple one line command.
java -jar migrator-tool-0.4.0-SNAPSHOT.jar --help.
You'll need a Spline 0.4 server running, see the example README for details. (the doc is written for the upcoming 0.4.0 release which isn't publish to Maven central yet, so you'll need to build it from sources)

[iamgzy]

Thanks @wajda for the details. That's very helpful. What is the angular cli version that built up release 0.3.9? package file provided a range for devDependency but not specifically mentioned the version (cli-compiler as well). I found this page but seems only for develop version (0.4 maybe?). Is there a simple doc for testing configuration for 0.3.9 release?

What kind of information would you like to search?

by column name
Currently in 0.3.9 if we like to know a position of a column in a lineage graph, we have to click through bunch of nodes(projection or aggregation of filter) to find the column and click on the right detail list to look up the lineage in orange. It'd be the best to search by column name then UI can list all possible nodes, then when user click on each search result it would be just like click on the associated node in the graph.

[wajda]

Angular CLI wasn't used in Spline 0.3. You only need NPM.
To run UI in a local dev mode:

cd $SPLINE_PROJECT_DIR/web/ui
npm install
npm run dev

This will also start a mock rest server on the port 3004 where UI is pointed by default in the dev mode.

To repoint the UI to another URL where your real Spline server sits you need to modify webpack.config.js, section devServer.proxy. Put you URL into the target and comment out the pathRewrite altogether.
Now you can run the UI using the same command as above, or (as you don't need a mock rest server in this case) using a different command that runs nodejs http server only:

npm run dev-http-server

Happy hacking :)

[iamgzy]

Thanks much @wajda

1. Could you please enlighten what is the Spline server? In 0.3.9 is it the Mongo server meta data store or the Spring boot application(Spline Web) with embedded tomcat or jetty server? From webpack setting it looks like the proxy setting is meant for RESTful services, but both mongo server and Spline Web are some sort of REST server.

2. Does Spline 0.4.0 Still support the Atlas integration? (which is the most critical factor in the making the decision :) mentioned in the other post)

3. I got below errors while run npm run dev, are they benign so could be ignored?

[0] ERROR in ./src/app/lineage/details/expression/expression-inline.component.ts
[0] Module not found: Error: Can't resolve '../../../../generated-ts/expression-model' in 'C:\Users\superuser123\projects\vs.code\spline.0.3\spline\web\ui\src\app\lineage\details\expression'
[0]  @ ./src/app/lineage/details/expression/expression-inline.component.ts 30:25-77
[0]  @ ./src/app/lineage/details/details.module.ts
[0]  @ ./src/app/lineage/lineage.module.ts
[0]  @ ./src/app/app.module.ts
[0]  @ ./src/main.ts
[0]  @ multi (webpack)-dev-server/client?http://localhost:8080 ./src/main.ts
[0]
[0] ERROR in ./src/app/lineage/details/operation/operation-details.component.ts
[0] Module not found: Error: Can't resolve '../../../../generated-ts/lineage-model' in 'C:\Users\superuser123\projects\vs.code\spline.0.3\spline\web\ui\src\app\lineage\details\operation'
[0]  @ ./src/app/lineage/details/operation/operation-details.component.ts 28:22-71
[0]  @ ./src/app/lineage/details/details.module.ts
[0]  @ ./src/app/lineage/lineage.module.ts
[0]  @ ./src/app/app.module.ts
[0]  @ ./src/main.ts
[0]  @ multi (webpack)-dev-server/client?http://localhost:8080 ./src/main.ts
[0]
[0] ERROR in ./src/app/lineage/details/operation/operation-icon.component.ts
[0] Module not found: Error: Can't resolve '../../../../generated-ts/lineage-model' in 'C:\Users\superuser123\projects\vs.code\spline.0.3\spline\web\ui\src\app\lineage\details\operation'
[0]  @ ./src/app/lineage/details/operation/operation-icon.component.ts 28:22-71
[0]  @ ./src/app/lineage/details/details.module.ts
[0]  @ ./src/app/lineage/lineage.module.ts
[0]  @ ./src/app/app.module.ts
[0]  @ ./src/main.ts
[0]  @ multi (webpack)-dev-server/client?http://localhost:8080 ./src/main.ts
[0]
[0] ERROR in [at-loader] ./src/app/dashboard/dataset-browser/dataset-browser.service.ts:18:43 
[0]     TS2307: Cannot find module '../../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/dashboard/dataset-browser/dataset-browser.component.ts:20:43 
[0]     TS2307: Cannot find module '../../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/types.ts:17:26 
[0]     TS2307: Cannot find module '../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/types.ts:18:27 
[0]     TS2307: Cannot find module '../../generated-ts/expression-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/types.ts:19:25 
[0]     TS2307: Cannot find module '../../generated-ts/datatype-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/lineage.store.ts:18:77 
[0]     TS2307: Cannot find module '../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/lineage.store.ts:58:69 
[0]     TS7006: Parameter 'id' implicitly has an 'any' type.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/lineage.component.ts:19:52 
[0]     TS2307: Cannot find module '../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/lineage.service.ts:19:28 
[0]     TS2307: Cannot find module '../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/graph/graph.model.ts:18:26 
[0]     TS2307: Cannot find module '../../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/details/expression/expression-render.service.ts:26:8 
[0]     TS2307: Cannot find module '../../../../generated-ts/expression-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/details/expression/expression-render.service.ts:97:71 
[0]     TS7006: Parameter 'child' implicitly has an 'any' type.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/graph/graph-builder.ts:17:66 
[0]     TS2307: Cannot find module '../../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/graph/graph-builder.ts:24:52 
[0]     TS2307: Cannot find module '../../../generated-ts/operation-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/graph/graph-builder.ts:113:32 
[0]     TS7006: Parameter 'op' implicitly has an 'any' type.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/graph/graph-builder.ts:116:37 
[0]     TS7006: Parameter 'inDsId' implicitly has an 'any' type.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/graph/graph-builder.ts:149:63 
[0]     TS7006: Parameter 'attrId' implicitly has an 'any' type.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/graph/graph.component.ts:18:28 
[0]     TS2307: Cannot find module '../../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/lineage.resolver.ts:19:57 
[0]     TS2307: Cannot find module '../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/details/operation/operation-icon.component.ts:18:26 
[0]     TS2307: Cannot find module '../../../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/details/operation/operation-details.component.ts:18:38 
[0]     TS2307: Cannot find module '../../../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/details/operation/operation-details.component.ts:21:27 
[0]     TS2307: Cannot find module '../../../../generated-ts/operation-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/details/details-section-header.component.ts:18:26 
[0]     TS2307: Cannot find module '../../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/details/expression/expression-dialog.component.ts:19:27 
[0]     TS2307: Cannot find module '../../../../generated-ts/expression-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/details/expression/expression-inline.component.ts:20:27 
[0]     TS2307: Cannot find module '../../../../generated-ts/expression-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/details/attribute-list/attribute-list.component.ts:18:26 
[0]     TS2307: Cannot find module '../../../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/details/attribute-list/attribute-list.component.ts:19:42 
[0]     TS2307: Cannot find module '../../../../generated-ts/datatype-model'.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/details/attribute-list/attribute-list.component.ts:87:35 
[0]     TS7006: Parameter 'f' implicitly has an 'any' type.
[0]
[0] ERROR in [at-loader] ./src/app/lineage/details/attribute-list/data-type-view/data-type-view.component.ts:20:33 
[0]     TS2307: Cannot find module '../../../../../generated-ts/datatype-model'.
[0]
[0] ERROR in [at-loader] ./src/app/dataset/dataset.service.ts:19:57 
[0]     TS2307: Cannot find module '../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/dataset/dataset.service.ts:37:38 
[0]     TS7006: Parameter 'ds' implicitly has an 'any' type.
[0]
[0] ERROR in [at-loader] ./src/app/dataset/dataset.resolver.ts:20:43 
[0]     TS2307: Cannot find module '../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/dataset/lineage-overview/lineage-overview.resolver.ts:19:57 
[0]     TS2307: Cannot find module '../../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/dataset/lineage-overview/lineage-overview.model.ts:18:48 
[0]     TS2307: Cannot find module '../../../generated-ts/operation-model'.
[0]
[0] ERROR in [at-loader] ./src/app/dataset/lineage-overview/lineage-overview.component.ts:19:40 
[0]     TS2307: Cannot find module '../../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/dataset/lineage-overview/lineage-overview.component.ts:23:48 
[0]     TS2307: Cannot find module '../../../generated-ts/operation-model'.
[0]
[0] ERROR in [at-loader] ./src/app/dataset/lineage-overview/lienage-overview-graph.component.ts:18:28 
[0]     TS2307: Cannot find module '../../../generated-ts/lineage-model'.
[0]
[0] ERROR in [at-loader] ./src/app/dataset/lineage-overview/lienage-overview-graph.component.ts:23:48 
[0]     TS2307: Cannot find module '../../../generated-ts/operation-model'.
[0]
[0] ERROR in [at-loader] ./src/app/dataset/lineage-overview/lienage-overview-graph.component.ts:123:17 
[0]     TS2322: Type 'VisClusterNode<number | VisNode | { (...items: ConcatArray<VisNode>[]): VisNode[]; (...items: (VisNode | ConcatArray<VisNode>)[]): VisNode[]; } | ((separator?: string) => string) | ((start?: number, end?: number) => VisNode[]) | ... 24 more ... | (() => IterableIterator<...>)>[]' is not assignable to type 'VisClusterNode<VisNode>[]'.
[0]   Type 'VisClusterNode<number | VisNode | { (...items: ConcatArray<VisNode>[]): VisNode[]; (...items: (VisNode | ConcatArray<VisNode>)[]): VisNode[]; } | ((separator?: string) => string) | ((start?: number, end?: number) => VisNode[]) | ... 24 more ... | (() => IterableIterator<...>)>' is not assignable to type 'VisClusterNode<VisNode>'.
[0]     Type 'number | VisNode | { (...items: ConcatArray<VisNode>[]): VisNode[]; (...items: (VisNode | ConcatArray<VisNode>)[]): VisNode[]; } | ((separator?: string) => string) | ((start?: number, end?: number) => VisNode[]) | ... 24 more ... | (() => IterableIterator<...>)' is not assignable to type 'VisNode'.
[0]       Type 'number' is not assignable to type 'VisNode'.
[0]
[0] ERROR in [at-loader] ./src/app/dataset/lineage-overview/lienage-overview-graph.component.ts:128:112 
[0]     TS2339: Property 'label' does not exist on type 'number | VisNode | { (...items: ConcatArray<VisNode>[]): VisNode[]; (...items: (VisNode | ConcatArray<VisNode>)[]): VisNode[]; } | ((separator?: string) => string) | ((start?: number, end?: number) => VisNode[]) | ... 24 more ... | (() => IterableIterator<...>)'.
[0]   Property 'label' does not exist on type 'number'.
[0]
[0] ERROR in [at-loader] ./src/app/dataset/lineage-overview/lienage-overview-graph.component.ts:191:81 
[0]     TS7006: Parameter 'dsId' implicitly has an 'any' type.

[wajda]

1. Spline Web server (with embedded Tomcat).
2. Not yet. Yeah I understand and remember the discussion. It's in our todo list, but just not a highest priority. Need to give it time.
3. Ah, sorry, I completely forgot about that one. No, you cannot ignore compilation errors :) Frontend app needs TypeScript interfaces that are generated from the Spline model during the project build.
   Just run a full Maven build once to generate the sources, and then try NPM again.
   mvn install -DskipTests

[iamgzy]

1. Spline Web server (with embedded Tomcat).
2. Not yet. Yeah I understand and remember the discussion. It's in our todo list, but just not a highest priority. Need to give it time.
3. Ah, sorry, I completely forgot about that one. No, you cannot ignore compilation errors :) Frontend app needs TypeScript interfaces that are generated from the Spline model during the project build.
   Just run a full Maven build once to generate the sources, and then try NPM again.
   mvn install -DskipTests

Thanks. Wow I am lucky to catch up at this moment. Hope you can get some rest later.

Will try build the project and run it again :)

Spline Web server (with embedded Tomcat).

So I take it as localhost:808x which ever port defined for the Spline Web tool? But isn't it just for http UI? Can I ask what is the port and url for RESTful call within the Spline Web tool?

[wajda]

It's the same port number. Spline Web server does both things - it serves static content (JS, HTML, CSS etc) and it also exposes REST interface.
So if you run your Spline Web at say port 8080, then http://localhost:8080/ is where your browser gets the Spline UI app from, while http://localhost:8080/rest/... is where the Spline UI app goes for REST.

[iamgzy]

@wajda thanks. Can Spline 0.4.0 compatible with 2.6.5?

thanks again for the instruction. In brief I found physics -> hierarchicalRepulsion -> nodeDistance can make a difference but as you mentioned it is difficult to find universal solution. The problem is when extending nodeDistance the graph no longer looks like a tree structure but more or less a network graph topology. There are threads on line mentioned similar issue but all leads to customization depends on the graph layout. I am continuing on the research.

[wajda]

Do you mean HDP 2.6.5?
If so then it should be compatible as long as you use Spark 2.3+ (2.2 is only partially supported as well as is in Spline 0.3)

[wajda]

Closing as won't fix as Spline 0.3 version series is End-Of-Life.
Please create new tickets against Spline 0.4 is the issue persists.