Validation slow for typed dimension tables in Solvency2

[alexdejong1723]

When validating XBRL files containing dynamic tables like in Solvency2 or CRDIV, the validation takes significantly more time than other validation tools. I created a test to reproduce this issue using the S.06.02.01 tables in Solvency2:

I added two tests. One with 10 lines and one with 1000 lines. The file containing 10 lines is validated relatively quickly, but the one containing 1000 lines will take a significant amount of time (after 30 minutes I stopped the process). Meanwhile, a validation tool like Altova validates that file in a couple of seconds.

2.6.0.zip

I have used version 1.2022.05.27 64-bit for these tests.

[derekgengenbacher-wf]

@alexdejong1723 We will begin an initial investigation into the slow performance on this next week. Thanks for the heads up and we will let you know the result of our investigation and resulting improvements.

[derekgengenbacher-wf]

@alexdejong1723 A quick update on our investigation. We have zeroed in on a filtering function that is operating really slow. We are analyzing what we can do to fix/rewrite/change the function to improve performance.

[vivet]

When do you expect this to be fixed / implemented?

[austinmatherne-wk]

@vivet we don't have an exact timeline yet, but I'll make sure this issue is updated when we do.

[austinmatherne-wk]

As of the latest release of Arelle 2.5.8 we've managed to cut the formula execution time for this taxonomy in half. Still not nearly as fast as Altova, but likely as fast as we can get at the moment given the current platform limitations (Python). We have some longer term plans that could potentially close the gap.