Step execution history is not showing the correct values

[klopfdreh]

Hi,

all values in the step execution history are currently showing the same value, but I know from our runs that they should differ (the process skip count should not be higher than 0, because all items were processed) - the count column is always at 40

This might be, because I am using scdf 2.9.4 with the new ui, but I guess there were no changes made to this part of the ui.

Versions:

Spring Cloud Data Flow 2.9.4
Spring Cloud Task 2.4.2
Spring Cloud Data Flow UI 3.3.0-SNAPSHOT (current version of master branch without angular 14 commit)

@claudiahub / @oodamien have you time to look at this?

[oodamien]

HI @klopfdreh,
Did you have a look on the API values?
I quickly checked the code UI, I didn't see any error so far.

[klopfdreh]

@oodamien - the api is providing this values - it is an issue of spring cloud task or spring cloud data flow backend instead. Thanks for pointing me into this direction:

[klopfdreh]

Hey @markpollack - I just wanted to ask why the label area/documentation was added - did I missed something or are the values correctly displayed? I was asking because our task did not skipped anything but showing up those values.

[cppwfs]

The first goal is to properly document the values present in the table. This will give a common vocabulary from which to ask questions or discuss changes in the calculations.
The skip values I will investigate.

[klopfdreh]

Ah ok, that makes sense to me. Maybe it helps when I explain what I understand.

So for me and most cases there would be no requirement to differentiate between max / min / mean. Commit Count could be a single number how many items were committed for example. Good to know would be if there is a deviation and how may items are affected from the last run. So deviation would also be good to know in a column.

So all count values could be listed as tuple numbers. The actual count and the deviation from the last run.

The only exception is duration and duration per read. Duration per read to me is how long it took to read an item with a reader which in this case could be displayed with min / max / mean and deviation. Duration in general to me is the time it took to complete the whole step execution which could also could be displayed as min / max / mean.

So all in all I would split the information in addition to check if they are correct. (Which they aren’t in our case 😄 )

Hope this helps.

[cppwfs]

Sorry for the delay in response.
The number of reads, writes, and commits can be viewed from the Job Executions Page as shown here:

As to the history Let's look at the example below:

In this example, we see that this job was run 4 times, and based on the flow, this step was executed 4 times (3 of which processed 6 records and one processed 20). Thus the changes in the min-max for the commits, reads, and writes. I agree that the count for each column is redundant based on how things are calculated and may be shown once at the top of the table.

I hope this helps.

[klopfdreh]

I think to show it once at top of the table would be better and reduce the confusion about what the count belongs to.

[cppwfs]

Added documentation for the Step History in the Data Flow Reference documentation.
Also created a story to deprecate the showing of counts for each column. It is enumerated here: #4971

@klopfdreh thank you for bringing this to our attention and writing up a good Issue.