Insights into performance gains of tuning min/max validators

[kosty]

Context

While there is a wide range of tradeoffs between code readability/maintainability and performance it would be interesting to look at how much speed is given up with different levels of readability.

This PR offers a simple test-case-like setup that allows do have some ball-park numbers to make discussion more concrete. Also by no means is this test suite complete or precise.

Methodology

Base case

Since there was no isolated environment to do the benchmarking on, I opted for benchmarking with respect to some "base" cases. Base cases were chosen as comparing two DoubleNode and comparing two LongNode.

Average time

Time is measured as difference between start and end of ThresholdMixin execution, and averaged over number of executions. Then average time for a particular case is compared to average time of a corresponding base case

Cases

Each combination of threshold and value is measured separately. Measurement for each combination is treated equally, which might not be true for a particular production setup, i.e. if numbers in schema/payload were always generated without quotes and for values within 64-bit range cases with double/double and integer/integer comparison would account for majority of use-cases.

Test values

Same numbers were used across multiple tests. Numbers were set to be within 64bit range. See Insights section for more.

Considered cases

Which cases should we consider when talking about performance/readability spectrum? Before going into case let us describe some questions each of the cases would have to answer to be functionally correct:

• What json schma "type" was provided ?
  Generally two are assumed: "integer" and "number"

• What threshold value was provided?
  Here we can talk of CPU-word long case (aka long/double) and a number that are outside CPU-word length (aka BigInteger/BigDecimal).
  Case of quoted numeric value can easily be assumed as one of the above two cases, as threshold assignment happens at schema creation and does not generally affect the validation runtime.

• What value node was provided for comparison?
  There seems to be 5 major categories, 4 similar to ones in threshold aspect plus a quoted value. Since string parsing now contributes to validation runtime we need to include this separately.

case 1: blind type coercing

Essentially it would functionally correct to coerce any threshold to a BigDecimal do the very same thing to comparison value and then compare two BigDecimal values. Very readable, simple and clean. ThresholdMixin is not really a necessity here. even less code! Performance comparable (might even slightly improve) compared to current setup.

case 2: special treatment for "integer" type, current setup

Assume that "number" type comparison is performed similarly to blind coercing and give special treatment to "integer". This already brings in ThresholdMixin into picture and given that threshold value can still be expressed as a floating point calls for a several sub-cases. Readability degraded. Same performance. Mixin could be refactored into it's own class, making code slightly less cluttered.

case 3: special treatment for value types

Here each "integer" and "number" each get their own mixin. At runtime mixin differentiates between values under/above 64-bit. Major speedup ~2x performance gain. Readability degraded similar to current case.

case 4: Special treatment to "types" and threshold values

Here a dedicated mixin is created based on combination of expected type and actual value provided for minimum/maximum. Number of mixins grows up to 4-6 (depending on implementation), runtime code for each mixin still needs to differentiate between different value types. Readability is greatly reduced. Performance gain ~30% over the special treatment for each of value types case

Insights

After some examination it looks like most of performance penalty is payed when a double/long value is serialized to string and then de-serialized back into some sort of numeric representation. Similar to this snippet

BigDecimal value = new BigDecimal(node.asText());

Both "big" number and textual values seem to perform better for such code. Also performance gains become more apparent as size of the value grows. Likely because bigger values usually take more symbols in a serialized form which then means a longer input for de-serializing the value.

Personal note

Not sure what is the major use case for light-4j frameworks. From where I am coming it looks like overwhelming majority of inputs is quoted numerics (long/double values serialized with quotes around them), with a minor share of values under 32-bits that come in unquoted form.

Reference

Sample result from my laptop (3.1 GHz Intel Core i7)

Base execution time (comparing two DoubleNodes) 36-59 ns
Base execution time (comparing two LongeNodes) 36-59 ns

||                         ||  value 64bit || value over 64bit || quoted value ||
| case 1 (float inputs)     | 176 ns        | 109 ns            | 192 ns        |
| case 1 (intg inputs       | 194.58 ns     | 536 ns            | 93.7 ns       |
| case 2 (float inputs)     | 357.8 ns      | 253.6 ns          | 179 ns        |
| case 3 (float inputs)     | 44 ns         | 50 ns             | 107 ns        | 
| case 3 (big float inputs) | 201.6 ns      | 47.66 ns          | 105 ns        | 
| case 4                    | 39.95 ns      | 62 ns             | 107 ns        |

or in terms of base performance

||                         || value 64bit || value over 64bit || quoted value ||
| case 1 (float inputs)     | x2.96        | x1.8              | x3.2          |
| case 1 (intg inputs       | x4.97        | x13.7             | x2.4 .        |
| case 2 (float inputs)     | x9.2         | x6.54             | x4.6          |
| case 3 (float inputs)     | x1.2         | x1.4              | x2.9          | 
| case 3 (big float inputs) | x5.5         | x1.3              | x2.9          | 
| case 4                    | x1.05        | x1.6              | x2.8          |

[stevehu]

@kosty The numbers are very interesting. I always thought integer should be the fastest but it is not the case. I need to dig into it deeper to understand how the library handles these use cases internally. This performance test gives us a lot of insight into how the current implementation behaves and it can be used as a based line to compare with any further enhancements. Thanks a lot for the effort.