Average event time

[yuyiguo]

I talked with Dima regarding the WMArchive UI. He was very interested to it. The tried the average event time that he cares a lot. And he found we have negative number for that. See below plot:
https://cmsweb-testbed.cern.ch/wmarchive/web/performance?metrics%5B%5D=jobstate&metrics%5B%5D=cpu.AvgEventTime&axes%5B%5D=host&axes%5B%5D=jobstate&axes%5B%5D=site&start_date=20161009&end_date=20161027&workflow=cerminar_Run2016B-v2-SingleElectron-23Sep2016_8020_161020_183522_4775&jobtype=Merge
@ticoann, is the default value for average event time is negative?

Then he tried this one:
https://cmsweb-testbed.cern.ch/wmarchive/web/performance?metrics%5B%5D=jobstate&metrics%5B%5D=cpu.AvgEventTime&axes%5B%5D=host&axes%5B%5D=jobstate&axes%5B%5D=site&start_date=20161009&end_date=20161027&workflow=cerminar_Run2016B-v2-SingleElectron-23Sep2016_8020_161020_183522_4775&jobtype=Processing
It has average event time for 10 second/event and the is the correct value. This plot made with job_type=Processing.

If one removes the the filer (that is for all job types), Then this is the plot:
https://cmsweb-testbed.cern.ch/wmarchive/web/performance?metrics%5B%5D=jobstate&metrics%5B%5D=cpu.AvgEventTime&axes%5B%5D=host&axes%5B%5D=jobstate&axes%5B%5D=site&start_date=20161009&end_date=20161027&workflow=cerminar_Run2016B-v2-SingleElectron-23Sep2016_8020_161020_183522_4775
It has 1 second/event. This is too small. Could be the data integration counting the number of events wrong? Or added the negative time?

[yuyiguo]

@ticoann confirmed that the default value event time is -1 if there is no value provided. It is obvious that WMArchive integration counted the default value -1. The purpose of -1 instead of 0 is because this give us a way to know if there is an event time with value 0 or none. The correct integration is not counting the -1's. @vkuznet will look into the integration code to see if there is a way to systemically ignore the default.

[vkuznet]

Hi,
I made a change to count only positive values. Could you please look-up data again on testbed and see if things change. I verified that one of the Dima's negative plot now show almost zero avg. time, see http://bit.ly/2fWu1hG

[vkuznet]

The code changes are here: #252

[ticoann]

I verified that one of the Dima's negative plot now show almost zero avg. time

This sounds too small. But its seems there is no 0 values. Are we using directly what is reported or changing the unit?

[vkuznet]

Seangchan,
there is no data manipulation in wmarchive, we see is what what we store.
I only applied a cut on values to be greater-equal than zero.
Valentin.

On 0, ticoann notifications@github.com wrote:

I verified that one of the Dima's negative plot now show almost zero avg. time

This sounds too small. But its seems there is no 0 values. Are we using directly what is reported or changing the unit?

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#245 (comment)

[yuyiguo]

@vkuznet
@ticoann
Below plot still have a 1-2 second/event and it seems to small according to Diam, but I think it is correct. Do you?

https://cmsweb-testbed.cern.ch/wmarchive/web/performance?metrics%5B%5D=jobstate&metrics%5B%5D=cpu.AvgEventTime&axes%5B%5D=host&axes%5B%5D=jobstate&axes%5B%5D=site&start_date=20161009&end_date=20161027&workflow=cerminar_Run2016B-v2-SingleElectron-23Sep2016_8020_161020_183522_4775

If I choose log collect as job type, there is no event time. If I choose merge type , this is almost zero event time. The processing type has a regular 10s/event. So if I average them together, log collect will not contribute, but average processing and merge will make it much small.

Eventthrough would be more useful than event time. But we don't have it. why?

[vkuznet]

Seangchan @ticoann please confirm that this is solved.