Problem with uniformity of DB12 results

[IgorPelevanyuk]

Recently I was analyzing the job execuiton and noticed quite a strange situation.

There are two problems with uniformity:

1. In the area around 54 DB12. It looks like some DB12 results are impossible to get. Like 52.8, 53.7, 54.4, 55.0 and so on.
2. In the area around 38 DB12. On the other hand, it looks like some values have considerably higher probability to appear.

So, I decided that the problem is with rounding and here what I got.

1. When we calculate DB12, we are using os.times() , but this function gives values rounded by 2 numbers after comma. Example: 4.58, 4.59, 4.6 and so on.
2. When DB12 benchmarks results are saved by dirac, they are also rounded by 1 number after comma (https://github.com/DIRACGrid/DIRAC/blob/c0afdba5ded29e72f1db144d71df3918313a28cb/src/DIRAC/WorkloadManagementSystem/scripts/dirac_wms_cpu_normalization.py#L66).

To simulate that I generated DB12 results which could be received from 50000 tests with test duration distributed uniformly:

df = pd.DataFrame({
    'duration': [3.00 + x * 0.0001 for x in range(50000)],
    'result': [round((250.0 / round(3.00 + x * 0.0001, 2)) / 1.0, 1) for x in range(50000)],
})

And draw a histogram counting different values of results (that is how CPUNormalizationFactor ius calculated for every job).

(Interactive plot with results: DB12_freq.html)
Obviously there is a problem with uniformity of the results. It is not only for high DB12 values, but also for lower one. It is just not so visible for low values of DB12.

[IgorPelevanyuk]

I also have a solution. Since we get DB12 result by using duration of test, and this duration always rounded by numbers after the comma, we can randomize that by the following formula:

cput = sum(end[:4]) - sum(start[:4]) # Code from DB12 benchmark
cput = cput - 0.005 + 0.01* random.random() # Proposal 

With that change in DB12 the results distribution would be the following:

And the interactive plot:

DB12_freq.html

You can repeat that by youself with the following code:

import plotly.express as px
import pandas as pd
import random

df = pd.DataFrame({
    'duration': [3.00 + x * 0.0001 for x in range(100000)],
    'result': [round((250.0 / (round(3.00 + x * 0.0001, 2) - 0.005 + 0.01 * random.random())) / 1.0, 1) for x in range(100000)],
})

fig = px.histogram(
    df,
    x='result', 
    title='Count different DB12 possible',
    text_auto=True,
    category_orders={"result": sorted(df['result'].unique())}, 
    nbins=2*len(df['result'].unique())
)

fig.update_layout(
    xaxis_title="DB12 values (accuracy 0.1)",
    yaxis_title="Count"
)

fig.show()
fig.write_html(
    "DB12_freq.html",
    include_plotlyjs=True,
    full_html=True,  
    auto_open=False 
)

[IgorPelevanyuk]

And another approach - Increase duration of DB12 calculation.
To understand how much time DB12 calculation took on a server, you may use formula:
duration = 250 / DB12

So, with modern CPUs that give you 40 DB12, it usually takes around 6 seconds.
The lower performance recorded in our infrastructure is around 14, so the duration of DB12 for that server is around 17 seconds.

If we consider that DB12 calculation duration around 1 minute is ok, than we can increase benchmark complexity 4 times!
That would mean another code change

DB12/src/db12/benchmark.py

Line 30 in f35072c

iters = int(1000 * 1000 * 12.5)

:

iters = int( 4 * 1000 * 1000 * 12.5)
calib = 4 * 250.0

And in that case we would get the following results:

Which is better, but not perfect.

[IgorPelevanyuk]

So my proposal is the following:

1. Increase complexity of DB12 calculations. 2 times
2. Add randomization, since os.times() is so inacurate.

[fstagni]

Hi Igor, thank you for your nice study. I think you can go ahead and commit your proposals.