JOSS Short Paper Review Comments

[lkosmid]

No paper mentioning WCET is complete without reference to its seminal paper: https://doi.org/10.1145/1347375.1347389 Please add the citation after the first appearence of WCET.

The MLE and PWM abbreviations are mentioned without explaining their meaning. Please provide their definition. Moreover, you need to provide a citations for them, as you have already done for the Kolmogorov-Smirnov and Anderson-Darning tests. A citation is needed also for Block-Maxima, Peak-over-threshold, Generalised Extreme Value and Generalised Pareto Distribution.

Also, seminal papers and state-of-the-art resources for MBPTA are missing:

• https://doi.org/10.1109/ECRTS.2012.31 was the first paper to introduce MBPTA, which is based on EVT. Moreover, an appropriate citation for EVT should be provided.

• https://www.tdx.cat/handle/10803/460819?locale-attribute=en my PhD thesis explains that in order to safely use MBPTA certain properties needs to be present. This is that:

  a) the execution times processed by EVT need to be independent and identically distributed random variables (i.i.d.). This is a requirement coming from EVT theory and has been relaxed later by Santinelli et al under certain circumstances.

  b) the events that take place during analysis and affect the execution time of the program (eg. cache misses) must be equal or worse than the ones that take place at operation. This is a requirement of MBPTA in order to ensure that the pWCET is not underestimated, and it is described in the literature as
  representativeness.

  It is very important to note that those properties are not present in existing platforms, so MBPTA cannot be applied on them without additional measures. In my thesis I proposed hardware and software solutions that make this possible. A simple solution is to make the events that affect the execution time, eg. cache misses, random from execution to execution, so that both properties are satisfied by construction: 1. the cache misses do not depend from one execution to another -> independence, 2. the events follow the same random distribution from execution to execution -> identical distribution, 3. due to identical distribution, the events that happen at analysis are the same with the ones at operation.

  Currently the paper only mentions the requirement a) but not b). I think it is very important that the paper mentions both requirements of MBPTA, to prevent that people are using chronovise (and therefore MBPTA) blindly, without ensuring its correct application, because this can result in unsafe pWCET estimates.

• https://doi.org/10.1145/3065924 identifies and solves a problem of classical MBPTA, proposing MBPTA-CV; MBPTA-CV takes values only from the tail of the distribution, which are used to model the tail of the distribution with EVT. In contrast, original MBPTA uses values from the entire distribution processed with Block-Maxima or Peak-Over Threashold, which may not belong to the tail of the distribution, resulting in a Generalised Extreme Value (GEV) or Generalised Pareto (GP) distribution with shape that is unsuitable for pWCET use. Thefore, MBPTA-CV can obtain pWCET in cases that original MBPTA fails, while it requires the collection of fewer samples.

• https://doi.org/10.1145/3185154 This independent comparison study performed a comparison between the Exponential Model employed by MBPTA-CV and the Block-Maxima or Peak-Over Threashold used by original MBPTA. It shows that the Exponential Model provides sound results, while Block-Maxima or Peak-Over Threashold can provide unsafe pWCETs (underestimation).

In addition to the above missing references, it is worth to note that MBPTA-CV is available as open source from https://doi.org/10.5281/zenodo.1065776 Therefore, the claim in the paper that source code for MBPTA is not available has to be revised. However, I still think that chronovise is a useful tool for MBPTA, because it can provide a common platform for the comparison of the different approaches.

In this aspect, I would like to see MBPTA-CV (Exponential Model and selecting data only from the tail) to be implemented in chronovise, as well as at least the EQMAE (Estimated Quantile's Mean Absolute Error) minimisation criterion proposed in https://doi.org/10.1145/3185154 although the implementation of the
Generalised Maximum Likelihood Estimation (GMLE) and L-moments would be nice to have, too. The reason is that those missing features are considered the current state of the art in MBPTA, and therefore they must be included in chronovise in order to increase its use from the research and engineering community. I believe that implementing those additional features will not be very difficult, since chronovise seems to be extensible.

Finally, please fix some entries in the references section (Ieee, Sies, Rtas) and make sure that all entries follow the same format (some entries have the full first names of the authors while some others only the first letter, some entries have first the surname of the author etc).

[federeghe]

Thank you for the detailed review. I will reply inline.

No paper mentioning WCET is complete without reference to its seminal paper: https://doi.org/10.1145/1347375.1347389 Please add the citation after the first appearence of WCET.

The MLE and PWM abbreviations are mentioned without explaining their meaning. Please provide their definition. Moreover, you need to provide a citations for them, as you have already done for the Kolmogorov-Smirnov and Anderson-Darning tests. A citation is needed also for Block-Maxima, Peak-over-threshold, Generalised Extreme Value and Generalised Pareto Distribution.

We added the citations for WCET, MLE and PWM and we expanded the acronym. For BM/PoT/GED/GPD we added a couple of books as a general citation for EVT.

Also, seminal papers and state-of-the-art resources for MBPTA are missing:

https://doi.org/10.1109/ECRTS.2012.31 was the first paper to introduce MBPTA, which is based on EVT. Moreover, an appropriate citation for EVT should be provided.

Added

https://www.tdx.cat/handle/10803/460819?locale-attribute=en my PhD thesis explains that in order to safely use MBPTA certain properties needs to be present. This is that:

a) the execution times processed by EVT need to be independent and identically distributed random variables (i.i.d.). This is a requirement coming from EVT theory and has been relaxed later by Santinelli et al under certain circumstances.

b) the events that take place during analysis and affect the execution time of the program (eg. cache misses) must be equal or worse than the ones that take place at operation. This is a requirement of MBPTA in order to ensure that the pWCET is not underestimated, and it is described in the literature as
representativeness.

It is very important to note that those properties are not present in existing platforms, so MBPTA cannot be applied on them without additional measures. In my thesis I proposed hardware and software solutions that make this possible. A simple solution is to make the events that affect the execution time, eg. cache misses, random from execution to execution, so that both properties are satisfied by construction: 1. the cache misses do not depend from one execution to another -> independence, 2. the events follow the same random distribution from execution to execution -> identical distribution, 3. due to identical distribution, the events that happen at analysis are the same with the ones at operation.

Currently the paper only mentions the requirement a) but not b). I think it is very important that the paper mentions both requirements of MBPTA, to prevent that people are using chronovise (and therefore MBPTA) blindly, without ensuring its correct application, because this can result in unsafe pWCET estimates.

Thank you for the detailed explanation. We added a couple of sentences for (b) and the citation, but the in-depth discussion of requirements cannot fit in the very short paper, considering that we should provide also the overall EVT theory basics in case. We emphasized the fact the requirements must be satisfied to obtain a reliable pWCET.

https://doi.org/10.1145/3065924 identifies and solves a problem of classical MBPTA, proposing MBPTA-CV; MBPTA-CV takes values only from the tail of the distribution, which are used to model the tail of the distribution with EVT. In contrast, original MBPTA uses values from the entire distribution processed with Block-Maxima or Peak-Over Threashold, which may not belong to the tail of the distribution, resulting in a Generalised Extreme Value (GEV) or Generalised Pareto (GP) distribution with shape that is unsuitable for pWCET use. Thefore, MBPTA-CV can obtain pWCET in cases that original MBPTA fails, while it requires the collection of fewer samples.

We implemented the CV version of EVT, using the exponential model (that is just a specialized case of the GPD).

https://doi.org/10.1145/3185154 This independent comparison study performed a comparison between the Exponential Model employed by MBPTA-CV and the Block-Maxima or Peak-Over Threashold used by original MBPTA. It shows that the Exponential Model provides sound results, while Block-Maxima or Peak-Over Threashold can provide unsafe pWCETs (underestimation).
(...)
In this aspect, I would like to see MBPTA-CV (Exponential Model and selecting data only from the tail) to be implemented in chronovise, as well as at least the EQMAE (Estimated Quantile's Mean Absolute Error) minimisation criterion proposed in https://doi.org/10.1145/3185154 although the implementation of the
Generalised Maximum Likelihood Estimation (GMLE) and L-moments would be nice to have, too. The reason is that those missing features are considered the current state of the art in MBPTA, and therefore they must be included in chronovise in order to increase its use from the research and engineering community. I believe that implementing those additional features will not be very difficult, since chronovise seems to be extensible.

This is a very recent work (Apr 2018) and for this reason we decided to do not include directly in chronovise. This for one main reason: the proposed method EQMAE breaks the normal EVT flow, because it requires to run the distribution estimation several times. We are not saying that EQMAE is not interesting, but since it changes a lot the process flow, we prefer to wait and see the impact on the community. The idea is chronovise is to provide a simple way for the researcher to implement their own algorithms, providing only a common structure to EVT process flow. I created #12 #13 #14 for future reference.

In addition to the above missing references, it is worth to note that MBPTA-CV is available as open source from https://doi.org/10.5281/zenodo.1065776 Therefore, the claim in the paper that source code for MBPTA is not available has to be revised. However, I still think that chronovise is a useful tool for MBPTA, because it can provide a common platform for the comparison of the different approaches.

Thank you, reference added.

Finally, please fix some entries in the references section (Ieee, Sies, Rtas) and make sure that all entries follow the same format (some entries have the full first names of the authors while some others only the first letter, some entries have first the surname of the author etc).

Fixed. The first names of a couple of references are still missing because they are old articles and we cannot find the original names. Anyway, we provided the DOI for all articles having it.

[lkosmid]

Thank you for the paper changes. In general my comments are addressed.

Some minor additional changes in order to close this issue:

• Probabilistic hard real-time has been proposed -> Probabilistic approaches for hard real-time systems have been proposed
• MBPTA is a probabilistic real-time branch -> MBPTA is a probabilistic analysis branch for real-time systems
• must fullfil the requirements of EVT. As discussed in the previous comment from the two requirements only the 1st come from EVT. I propose to change the text as follows: must fullfil certain requirements. In particular ... The first requirement comes from EVT, it can be checked with suitable statistical tests [MBPTA paper] and can be relaxed under some circumstances [Santinelli et al], while the latter... and to the system (hardware/software) properties. Both requirements are necessary ...
• The few previous software are cited in -> The few existing software [Lu],[Lesage] ...
• ... a variant of classical EVT procedures -> a variant of classical MBPTA analysis...
• pure users -> end-users
• Dsd in your paper reference-> DSD
• Rtas -> RTAS
• remove &dash from the WCET paper reference
• MBPTA-Cv reference -> MBPTA-CV

Regarding the EQMAE I understand your argument, so I agree that you don't have to implement it for the paper to be accepted. How about #12 and #13? If they do not break chronovise's flow, they should be easy to implement and will increase the impact of your software.

[federeghe]

Thank you for the paper changes. In general my comments are addressed.

Some minor additional changes in order to close this issue:

Probabilistic hard real-time has been proposed -> Probabilistic approaches for hard real-time systems have been proposed
MBPTA is a probabilistic real-time branch -> MBPTA is a probabilistic analysis branch for real-time systems
must fullfil the requirements of EVT. As discussed in the previous comment from the two requirements only the 1st come from EVT. I propose to change the text as follows: must fullfil certain requirements. In particular ... The first requirement comes from EVT, it can be checked with suitable statistical tests [MBPTA paper] and can be relaxed under some circumstances [Santinelli et al], while the latter... and to the system (hardware/software) properties. Both requirements are necessary ...
The few previous software are cited in -> The few existing software [Lu],[Lesage] ...
... a variant of classical EVT procedures -> a variant of classical MBPTA analysis...
pure users -> end-users
Dsd in your paper reference-> DSD
Rtas -> RTAS
remove &dash from the WCET paper reference
MBPTA-Cv reference -> MBPTA-CV

Thank you, all comments fixed.

Regarding the EQMAE I understand your argument, so I agree that you don't have to implement it for the paper to be accepted. How about #12 and #13? If they do not break chronovise's flow, they should be easy to implement and will increase the impact of your software.

Ok, PWM is another name for L-moments, so I closed #14 and wrote it in the paper. I implemented GMLE also and closed #13, thank you for suggestions.

[lkosmid]

Thank you for the document and code updates, we are almost there :)
Last issues:

On the other hands -> On the other hand
There is are still some capitalisations missing in the references:
Ieee in the RTSS 2012 refernce -> IEEE
Ieee and Sies in SIES 2013 reference-> IEEE and SIES respectively
another Ieee in RTCSA 2010 -> IEEE
Ieee in RTAS 2017 -> IEEE

[federeghe]

Unfortunately it's not simple to notice them without having the possibility to compile markdown+bibtex locally (or at least I didn't find an easy way to do it). Thank you for detecting them.

Fixed 👍

[lkosmid]

Thank you @federeghe for the quick fix. There is a last Sies remaining and a typo (aimis->aims).

[federeghe]

Thanks, fixed