Why are there multiple FDS version runs in Verification and Validation guides?

[Er9y714]

As reading an article today I have learned (never noticed before) that there are results from multiple FDS versions in Validation and Verification Guides of FDS . For example in Validation Guide of v6.11.0 the majority of the results are from FDS v6.10.1. Whereas, 6.11.0 is majority in Verification guide but also consists of 6.10.1 and 6.9.1.

As far I as remember, all those V&V simulations are run and checked for each major release. Including earlier versions to a certain release document looks counter intuitive.

I was wondering if there is any reason having a mixture of versions in the guides?

Thank you,

[drjfloyd]

The validation cases take a very long time to run. If you look at the git hashes in the Validation Guide you will see that they vary and that they do not equal the git hash of the release version of 6.10.1. These represent various states of the FDS source since the release of FDS 6.10.1. For example, 6.10.1 was released April 2025 with the git hash of 12efa16, and the git hash for the plots for the NIST FSE 2008 validation cases is 20dc4c7 which is the state of the source in Dec 2025.

There are a handful of verification cases which are not part of the continuous integration testing done each night as they take too long to run to do overnight. These are periodically cycled through with the Validation Cases.

[Er9y714]

I see, thank you for the clarification.

[bwklein]

I hadn't really looked into it before, but I assumed that prior to a release, that all V&V cases were run and reviewed for discrepancies from the prior release. I know it cannot be done for the nightly tests, but a comprehensive run of the entire suite prior to a release seems like a reasonable QA check. Something I have mentioned in the past was to have a table of prior release metrics for the suite, to quickly check for significant changes from the last version. These metrics would be case/quantity specific, but would help track changes over time to the results of FDS, knowing if the results are getting better or worse as time goes on. At a minimum, it could be the table of uncertainties and biases in the Validation guide.

[drjfloyd]

With the exception of a very small number of cases, every Verification case is run every night and we do not do a release unless all of those nightly cases pass. Every case has one or more metrics being examined. There are 1000s of verificaiton comparisions. The few cases that are not part of the nightly build are primarily fundamental flow phenomena that are regions of the code that are not frequently changed. Even those cases were run between 6.10.1 and 6.11.0.

All of the validation cases were run between 6.10.1 and 6.11.0. Running the entire suite takes weeks if the full NIST cluster is devoted to nothing but running validation cases. This is the same cluster used for development. This is not a commercial operation where there is incomming revenue to spin up cloud computing at will for running a massive compute load. A balance needs to be struck between being able to continue doing work and support releases.

[bwklein]

I appreciate that most of the Verification cases are run nightly, and that a full V&V run was completed between 6.10.1 and 6.11.0. I take issue with the claim that commercial operations have more resources to waste on frivolous or unnecessary compute services, I know we don't. Also, this isn't a attack on the dev team, my comment was to express that the V&V process wasn't aligned with expectations (mine and others) for a release package. I fully understand how time and resource consuming this activity can be.

How does any single developer or the project team in general have confidence that everything is still fine for every test case, with multiple developers changing different parts of the code, unless you run the tests? How does the team go about selecting the correct subset of cases to run if the entire suite isn't processed? The fact that the suite was run recently demonstrates that a balance can be found, and the activity is worthwhile.

It seems from the response that two things are true.

1. Verification Cases are the primary set of test cases for the functionality of FDS. Which makes sense as they are designed to exercise specific features and capabilities of the model.
2. The team is confident that a good process in place by which the best subset of Validation Cases are selected and run prior to a release as a secondary check on how code changes might impact results compared to experimental data.

According to the Configuration Management plan, 'For major new releases of the model, a complete suite of test cases should be compared to those from previous versions of the model. ' As there hasn't been a major release of FDS in over 12.5 years, you are already doing more than stated in the SQAP for minor version 6.11.0. 😄

I don't think in general the V&V document matters, nobody is reading it for fun on the weekends. Until something about the model is called into question, and then the relevant parts of the document become a big deal to those who are excited about something. The fallback is always to just take the cases you think are relevant, and run them in whatever release you have questions about, which makes out of date information in the last version of the V&V Guide even more irrelevant in that moment.