Software validation

You may think of a PKPD software as a calculator. You expect that operations in a calculator are correctly performed internally, so that the machine returns the correct answer. However, as a calculator, the answer is only accurate if it is correctly operated, and the problem at hand is correctly translated into sequential operations that give the correct answer. For instance, how many seconds are there in a day? The correct answer is 246060=86400, and not 24+60+60=144. A validated calculator will give both answers, 86400 and 144 (and not any other number), depending on how it is operated. The result is only as good as the ability of the calculator operator to translate a real-world problem into a tractable mathematical problem that correctly approximates reality ("Essentially, all models are wrong, but some are useful" (George Box, a famous statistician)).

PK, PD, PKPD, PBPK models are nothing more than (more complicated) sequential operations aiming at the analysis and explanation of the observed experimental behaviour of some substance in the body. They normally use differential equations (DE) that is a mathematical tool that helps to explain time variations and physical interactions. Although more complicated, the operator principle is the same.

Operational/Performance qualification of Scipion

Every Scipion installation must be validated to make sure that there is no problem with this version of the software and hardware. We must be sure that the software is returning the right answer if it is given certain data (in our example above if we input 24 times 60 times 60, the calculator must answer 86400). This is called Operational/Performance qualification (OQ/PQ or OPQ). This answer depends on the software as well as the hardware in which it is installed. Finding the parameters of a DE system that better fits some observed data involves several million arithmetic operations (our example of 86400 involved just three of them) and small differences in the hardware, combined with the decisions taken by the search algorithm based on the numbers it sees, result in slightly different answers for the DE parameters. In our 86400 example, it is as if the calculator is returning 86399.999985, the result is still correct (actually with your pocket calculator there is an interesting experiment: A) divide 4 by 3 (answer=1.333333); B) subtract 1 from the result (answer=0.33333); C) multiply the result by 4 (answer: 1.333333, we should be back at the starting point of step B); D) repeat steps B and C 30-40 times, the calculator behaviour is not what you would expect; this experiment has involved no more than 100 operations, at most, fitting DE parameters require several millions).

Additionally, the fitting problem is not uniquely determined and several different sets of DE parameters explain almost equally well the observed data (the statistical theory of sensitivity and identifiability is rich and very interesting for the advanced modeller). In our 86400 example, 242460 is 86400, but 2459.9560.05 is also 86400.

Finally, the observed data is noisy (when we say that 30 minutes after the oral dose the drug concentration in plasma was 1 umol/L, our precision of the 30 minutes and 1 umol/L is limited; we do not mean 30 minutes=1,800,000,000 microseconds after the oral dose and 1.00000000000000000 umol/L with no analytical error). It is as if in our 86400 example, we have to look for three numbers whose product is some observed value. This observed value is sometimes 86401, other times 86396, 86407.4, ...

All this results in a variability of the DE parameters. The answer is considered to be correct as long as the parameters fall within an acceptable, predefined range. In our 86400 example, the answer is considered to be correct if the number of seconds per minute is estimated to be between 59.5 and 60.5.

Xcipion comes with a suite of tests whose results are known for free. These tests construct the inputs for multiple protocols and verify that the answer is the expected one. To run these tests, go into the xcipion installation directory and run

./scipion tests >& tests.log
grep -i error tests.log
cat tests.log

The test process can go up to 2h depending on the machine. The grep command should not have any message and the cat should show all the logging from the executions in green (no red message should be shown). If there are errors in the test, please, contact info@kinestat.com.

You may inspect the results of any of these tests by the command

./scipion project TestGabrielssonXXXXWorkflow &

where XXXX is the label of the tests (typical labels are PK01, PK02, ... PD01, PD02, ...). You may also open xcipion and open the test project you prefer.

./scipion & 

The installation process itself must also be validated (Installation Qualification).

Useful resources

https://www.certara.com/2011/03/08/software-validation-do-i-have-to-do-it/