Repository navigation
Home
Bottom line - these are guidelines not rules!
While there will be many circumstances where application of the guidelines is straightforward, we expect that there is a diverse set of circumstances where users may require some guidance to troubleshoot usage. Below is a growing list of Frequently Asked Questions (FAQ) that we will gradually extend. If you do not see your answers here then contact us!
Q: I have developed a hybrid simulation model. Do the STRESS guidelines apply?
Hybrid simulation is a rapidly growing area of Modelling & Simulation. The STRESS Guidelines are for DES, ABS and SD. How do we apply them to... HS? We recommend a common sense approach. If the parts of a HS are clearly mappable to DES, ABS and/or SD then use multiple STRESS documents to describe these parts (i.e. the following HS model is composed of a DES and an ABS component captured in the following two STRESS records). If this is difficult then we feel that you have discovered a new STRESS Guideline. Get in touch and we will help you develop it and store it in this repository.
Q: Writing is a creative process. Can I structure my journal paper/conference paper/report, etc. in my own way?
The Guidelines are not prescriptive in how to structure an article. The guidelines simply specify a minimum checklist to aid reproducibility of an author’s model. Authors can structure their articles however they wish. We encourage authors to make use of Supplementary material and additional files (where possible). We also ask authors to remember that reviewers tend to want a manuscript that is easy to follow and appraise. Simple reporting that is clear and concise for scientific writing is good practice.
Q: Our model is a very large “mega-model” of an entire city. Do the STRESS guidelines apply?
The applicability of the guidelines depends on whether the contribution is the model and scientific result or the programming framework you have developed. For frameworks, it may be more appropriate to use standard documentation approaches from software engineering (see the STRESS paper for details). If it is a specific model with a specific result that you wish to disseminate (for example, how a transport policy affects overall city congestion) then it is a scientific paper that should follow the appropriate STRESS Guidelines. We acknowledge that for very complex and large models some adaption may be required. Again, get in touch as we might need a new STRESS Guideline for "mega-models".
Q: Not all of the elements of STRESS are applicable in my case.
The STRESS Guidelines are not rules but guidelines to strengthen reporting of simulation models. There is no issue if authors find that certain sections of STRESS are not applicable. Reviewers may query omissions, and authors should be able to justify them. Any queries then please contact us.
Q. There are some unique features of my model that STRESS does not cover. Should I document them?
The spirit of the Guidelines is reproducibility and we expect that the guidelines set the minimum information needed to report most models from ABS, SD and DES. If an author believes that their model requires additional detail in order to aid reproducibility, then they should include it for completeness of reporting.
Q. I have published my model code – why do I need to include the other details in STRESS?
We recommend that authors consider publishing code as an enhancement to reporting not the other way around. Some journals, such as System Dynamics Review, require authors to submit the model that they are reporting. Publication of open code will strengthen reproducibility (we note that not all authors may wish to share their code for commercial reasons). However, other researchers and practitioners may not have access to the commercial software used or may not have the right programming skills. Even when researchers are familiar with the programming languages used the code itself might be difficult to follow for a variety of reasons.
Q: My model is stochastic. What is a reproducible result?
There are several levels of reproducibility that might be achieved within the reporting of a model and its results: ordinality, precision and output uncertainty. Ordinality refers to the order of results, i.e., for a given model output, other researchers can confirm that scenario x is statistically better than scenario y and both x and y are better than scenario z. The precision of a result refers to the point estimate for a model output of a specific model configuration or the difference in output between two model configurations (scenarios). Within a stochastic model, for a given output and model configuration, the point estimate should be reproducible exactly or within the given confidence interval of reported results. It follows from precision that any sensitivity analysis conducted to assess uncertainty in model outputs due to uncertainty in model inputs should be reproduced in the recreated model.
Q: I am modelling using DEVS. Do the guidelines apply?
Discrete-event system specification (DEVS) has its own self-documenting formalisation. The STRESS Guidelines are aimed at improving the completeness of reporting for DES, SD or ABS models that have been developed using less specialised approaches (i.e., simulation software, programming languages or general purpose programming languages). We see DEVS has the potential to form one part of simulation reporting within STRESS. These guidelines complement DEVS by incorporating other details needed for full reproducibility of the modelling study.
Q: Why are the extra details needed? If I want to recreate a model I can contact the authors of the work.
Yes that is fine and if necessary there is no reason not to contact the authors, but it assumes that the author(s) are still contactable, available, willing to respond and can remember. The published write-up is the permanent public record of the work. The reporting may not be perfect, but if the guidelines have been followed it will reduce the reliance on the authors and if necessary it will help an author answer the questions perhaps five years after publication (which might be seven years or more after the work was actually done). Use of Open Science approaches will also help support reproducibility.
Q: Will STRESS limit the write-up of “projects” i.e., the story of what happened in a simulation study
The reporting guidelines apply to the model itself and the model’s results. The wider aspects of the modelling process/practice and its context are also of importance to the scientific and practical communities. This is a separate scientific area from STRESS and it is linked to Behavioural Operational Research. Such studies should follow the rigour of an appropriate Behavioural Operational Research methodology and perhaps an appropriate quantitative or qualitative reporting guideline.
Q. My model and operations system is confidential and I cannot include all of the details recommended by STRESS. However, there are lessons from the work that are relevant to the simulation community.
In such cases, reviewers of the work wish to know that a rigorous approach to model development and analysis has been followed. One option to do this is to appoint an independent third party to quality assure the work whilst still maintaining confidentiality of proprietary information. It is recommended that a summary of the quality assurance is submitted along with the model. If the novel aspects of the work cannot be fully understood and verified without knowing the confidential information then it is better not to publish.
Q: I have concerns about other using my work without crediting the original authors of the work.
We recommend that authors publish their work under a creative commons licence. Authors are now licensers of their research and can choose a licence that suits their needs. A popular licence used by many open access publishers is the attribution licence CC BY (https://creativecommons.org/licences/). This means that anyone can reuse the work either in part or in whole for any purpose, for both commercial and non-commercial licence but must credit the original authors. Other licences mandate that licensees also make their published work available under the same licence terms as the original.
Q: I plan to publish the details of my model through a third party or academic institutions website and reference it in the manuscript. Do I still need to follow the STRESS Guidelines?
We appreciate that many models are large and complex. This has the potential to lengthen journal articles. We strongly recommend that, where available, authors make use of journal facilities for online supplementary material as opposed to third party websites. Third party websites may change, break or be taken down without an author’s knowledge. This has the potential to affect review and reuse at a later date. Journal articles are a permanent public record of the model.
Q: Model verification and validation is not included in the guidelines. Should I include it in my write-up?
The short answer is Yes. It is recommended and preferred that details of model validation and verification (V&V) are included in both academic and industry reports of models. The STRESS test aims to increase the reproducibility of models and V&V is not a requirement to do so. Nonetheless, STRESS helps with reporting of V&V. It requires verification data to be reported, either the data used to produce the scientific results or hypothetical test verification data. Reporting V&V in full would require details of the tests, either statistical or based on expert judgement. It may also be necessary to provide more than one set of test data as a single set may not provide sufficient confidence to other researchers and practitioners who wish to reuse a model due to “not invented here syndrome”. So, V&V data and results can provide confidence that a model has been reproduced to sufficient accuracy. It is recommended that authors consult V&V literature.
Q: I have reused or adapted a published or publicly available model that has followed the STRESS Guidelines. How do I report the model I am using in my paper?
It is only necessary to report the adaptations to the model, data or analysis that you have conducted. These modifications should follow the STRESS guidelines. Provide a reference to the original paper. The same applies to industry models, although industry models that adapt models from research may wish to include a copy of the academic paper in an appendix.
Early days but we hope this will grow rapidly.
- Taylor, S.J.E., Anagnostou, A., Monks, T., Currie, C., Onggo, B.S., Kunc, M., Robinson, S. (2018). Applying the STRESS Guidelines for Reproducibility in Modeling & Simulation: Application to a Disease Modeling Case Study. In Proceedings of the 2018 Winter Simulation Conference, to appear.
An example of how STRESS-ABS can be used to document and agent based simulation. The STRESS record for the Disease Modelling Case Study is at http://dx.doi.org/10.15169/sci-gaia:1494421530.94.