Pipeline viz #1

fabalex7 · 2023-12-14T12:44:16Z

merging pipeline_viz (with cli support) into main

joluj

General comments:

use kebab-case instead of snake_case for file names (we do that everywhere else, so just for consistency).
use camelCase for variables, constants and functions.
refactor the code to work with the AST functions instead of the runtime types. This makes the code much more solid and more errors will be detected on runtime. Also, it then should be a bit more readable. I commented an excerpt of important functions for you, you'll probably don't need much more.

joluj · 2023-12-15T12:47:06Z

apps/pipeline-visualization/src/main.ts

+  setMermaidTheme,
+ } from './mermaid_utils';
+
+export async function myRunAction(


Use proper names

joluj · 2023-12-15T12:54:47Z

apps/pipeline-visualization/src/main.ts

+
+  const model = await extractAstNodeFn(services, loggerFactory);
+
+  const myblocks = getBlocksInTopologicalSorting(model.pipelines[0]!)


This will fail if there a no pipelines in the model and it will result in non-intuitive behavior if there are more than one pipelines defined.

Both cases do not make sense at the current state of Jayvee, but I'd prefer some sanity check there (i.e. assert that there is exactly one pipeline and fail otherwise).

joluj · 2023-12-15T12:55:37Z

apps/pipeline-visualization/src/main.ts

+  fileName: string,
+  mermaidOptions: MermaidOptions
+){
+  console.log('Hello World!');


Suggested change

console.log('Hello World!');

joluj · 2023-12-15T12:56:00Z

apps/pipeline-visualization/src/main.ts

+    debugTarget: undefined
+  }
+  const result = await myRunAction(fileName, options, mermaidOptions);
+  console.log("End World");


Suggested change

console.log("End World");

joluj · 2023-12-15T12:56:07Z

apps/pipeline-visualization/src/main.ts

+  return ExitCode.SUCCESS;
+}
+
+async function myCall(


joluj · 2023-12-15T17:33:45Z

apps/pipeline-visualization/src/mermaid_utils.ts

+    if (mermaidOptions.properties){
+      for (let property of block.body.properties) {
+        if (properties.includes(property.name)){
+          let pv: any = property.value // Is there a better way of doing this?


// Is there a better way of doing this?

Yes, using the AST :D

joluj · 2023-12-15T17:36:41Z

apps/pipeline-visualization/src/mermaid_utils.ts

+    if (block.name.includes("Extractor")){
+      classAssign.push(`class ${block.name} source;`)
+    }
+    if (block.name.includes("Loader")){
+      classAssign.push(`class ${block.name} sink;`)
+    }


The block names are arbitrary, i.e. it is possible to name an extractor MyCoolLoader. You should use our utility functions (in this case the attribute inputType of a BlockTypeWrapper could help)

joluj · 2023-12-15T17:38:17Z

example/cars2.jv

@@ -0,0 +1,221 @@
+// SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg


I'd remove this file from git.

joluj · 2023-12-15T17:38:37Z

example/exercise1.jv

@@ -0,0 +1,45 @@
+pipeline AirportsPipeline {


What benefit does this file bring?

joluj · 2023-12-15T17:39:42Z

example/mycars.jv

@@ -0,0 +1,83 @@
+// Define a new blocktype named CSVExtractor outside of the pipeline


Nice example of composite blocktypes. I'd be happy if we can use this example later, but it needs some polishing first (and renaming)

fabalex7 · 2024-01-11T08:41:17Z

included changes, please review again @joluj

joluj

Commented instead of approved since the Actions did not run. They will run when you push the next time.

joluj · 2024-01-11T13:48:39Z

apps/pipeline-visualization/src/mermaid_utils.ts

+  const listofPipes: Array<string[]> = [];
+  const listofBocks: Array<string> = [];


Suggested change

const listofPipes: Array<string[]> = [];

const listofBocks: Array<string> = [];

const listOfPipes: Array<string[]> = [];

const listOfBocks: Array<string> = [];

joluj · 2024-01-11T13:52:17Z

apps/pipeline-visualization/src/mermaid_utils.ts

+    if (mermaidOptions.properties) {
+      for (const property of block.body.properties) {
+        if (properties.includes(property.name)) {
+          if (isTextLiteral(property.value)) {


Keep in mind that there are other literals that are also easy and important to display, for example numeric or boolean literals. Can be done later though.

joluj · 2024-01-11T14:05:06Z

example/cars-composite.jv

@@ -0,0 +1,83 @@
+// Define a new blocktype named CSVExtractor outside of the pipeline


If the file is in the example folder it will be published as in the Jayvee documentation (https://jvalue.github.io/jayvee/docs/category/jayvee-examples).

Since this file is only relevant for testing purposes, I'd suggest moving it to apps/pipeline-visualization/test/assets/nested-composite-blocks.jv (create folders and rename).

joluj · 2024-01-15T07:13:19Z

The CI Pipeline fails because of missing licenses. In our project, every file must have a valid license.

For files that allow comments: Add them at the top (Example for Jayvee Files and TypeScript files).
For files that don't allow them: Create .license-file (Example directory with project.json and project.json.license).

The CLA Assistant fails which we can ignore in a fork.

fabalex7 added 12 commits October 24, 2023 15:48

created pipeline-viz app

3e38a02

added pipeline viz app

6ebd2bc

Merge branch 'main' into pipeline_viz

975dacc

fix tsconfig.base.json

4ee0a1f

added own runAction and interpretModel functions

308a120

added calling function

edf0a08

added print for all pipelines

81e6f9b

added debugging and additional examples

a3bae2c

create parametrized mermaid code for simple pipeline

ef65474

added composite functionality and example

32636a1

added properties functionality

ef39e8c

added CLI and file output

bf2a581

joluj reviewed Dec 15, 2023

View reviewed changes

fabalex7 added 12 commits December 17, 2023 11:33

added output CLI arguments

ef9160c

renamed processOptions and prettified

3902ac3

fix for escaping characters in properties

1723692

added example for debugger

94e1c1d

change default options for debugging

4fe0cdf

changing to isComposite wip

a7280a4

replaced composite and properties with AST functions

541c995

linter runs successfully

39fe442

refactored createMermaidRepresentation and createMermaidStyling

da75962

refactored processOptions and runModel into run.ts

3ec54f6

removed examples and reset debugger

e7cc93f

added exit code

0eb35d8

added prettier settings

951d242

joluj reviewed Jan 11, 2024

View reviewed changes

fabalex7 added 2 commits January 14, 2024 18:08

variable refinement; use convenient method hasInput

84fcb86

moved composite-blocks example to test assets

950b312

fabalex7 added 2 commits January 15, 2024 15:59

removed .vscode/settings.json

335e731

added licensing information

35c02f2

fabalex7 merged commit fb74ff1 into main Jan 16, 2024
1 of 2 checks passed

github-actions bot locked and limited conversation to collaborators Jan 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pipeline viz #1

Pipeline viz #1

fabalex7 commented Dec 14, 2023

joluj left a comment

joluj Dec 15, 2023

joluj Dec 15, 2023

joluj Dec 15, 2023

joluj Dec 15, 2023

joluj Dec 15, 2023

joluj Dec 15, 2023

joluj Dec 15, 2023

joluj Dec 15, 2023

joluj Dec 15, 2023

joluj Dec 15, 2023

fabalex7 commented Jan 11, 2024

joluj left a comment

joluj Jan 11, 2024

joluj Jan 11, 2024

joluj Jan 11, 2024

joluj commented Jan 15, 2024 •

edited

Loading


		const model = await extractAstNodeFn(services, loggerFactory);

		const myblocks = getBlocksInTopologicalSorting(model.pipelines[0]!)

		@@ -0,0 +1,221 @@
		// SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg

		@@ -0,0 +1,83 @@
		// Define a new blocktype named CSVExtractor outside of the pipeline

		const listofPipes: Array<string[]> = [];
		const listofBocks: Array<string> = [];

Pipeline viz #1

Pipeline viz #1

Conversation

fabalex7 commented Dec 14, 2023

joluj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabalex7 commented Jan 11, 2024

joluj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joluj commented Jan 15, 2024 • edited Loading

joluj commented Jan 15, 2024 •

edited

Loading