General PostProcessors and ROM training performance improvement

[alfoa]

---

Issue Description

Almost any PostProcessors and the TimeDep ROM training is extremely slow.
For example, the PostProcessor HistorySampling is basically not usable with datasets > 1 gb, completely defeating the purpose of the PP.
Now that we moved to the new DataObject, we have to improve all the performance forbiding the usage of the method adDataset("dict"). It is too slow...

Please attach the input file(s) that generate this error. The simpler the input, the faster we can find the issue.

---

For Change Control Board: Issue Review

This review should occur before any development is performed as a response to this issue.

• 1. Is it tagged with a type: defect or improvement?
• 2. Is it tagged with a priority: critical, normal or minor?
• 3. If it will impact requirements or requirements tests, is it tagged with requirements?
• 4. If it is a defect, can it cause wrong results for users? If so an email needs to be sent to the users.
• 5. Is a rationale provided? (Such as explaining why the improvement is needed or why current code is wrong.)

---

For Change Control Board: Issue Closure

This review should occur when the issue is imminently going to be closed.

• 1. If the issue is a defect, is the defect fixed?
• 2. If the issue is a defect, is the defect tested for in the regression test system? (If not explain why not.)
• 3. If the issue can impact users, has an email to the users group been written (the email should specify if the defect impacts stable or master)?
• 4. If the issue is a defect, does it impact the latest stable branch? If yes, is there any issue tagged with stable (create if needed)?
• 5. If the issue is being closed without a merge request, has an explanation of why it is being closed been provided?

[wangcj05]

For the record, with 5 GigaByte input data, the performance of BasicStatistics PP is:

1) Increase the realizations from 2020 to 10100, with 169 variables, mean, variance and covariance are computed for 169 variables:
a. 2020|169 cost per time step 6s
b. 4040|169 cost per time step 11s
c. 10100|169 cost per time step 24s

2. with 3 variables:
 a. 4040|3 cost per time step 0.005s
 b. 8080|3 cost per time step 0.0077s

3. with 85 variables
a. 4040|85 cost per time step 2.75s

Thus the computational cost is proportional to number of time steps, number of realizations, and the square of number of variables when covariance is computed.

[wangcj05]

archived