Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
chore: Add basic collision probabilities script (#29)
## Context
We currently have very little information on how likely collisions are for each of copycat's API methods. For some methods (e.g. bool), these are trivial or unimportant. For data types that typically need to be unique (e.g. `uuid` or `email`), this is important information to know: it allows us to know where in the API we need to improve to make collisions less likely, and lets our users know how likely collisions will be for their use case.
## Approach
```
* For each API method in the shortlist:
* Until the stopping conditions are met (see below):
* Run the method with a new uuid v4 as input until the first collision is found, or until we've iterated 999,999 times
* Record the data set size at which the first collision happened as a datapoint
* Output the obtained stats (e.g. mean, stddev, margin of error)
// The stopping conditions
* Both of these must be true:
* We've run the method a minimum (100); or
* Either
* The margin of error is under a lower threshold (`0.05`); or
* The margin of error is under a higher threshold (`0.10`), but the sum (sum of first collision dataset sizes obtained so far) is over a higher threshold (999,999)
```
Since the task here is number-crunching, we do this using a worker farm to leverage multiple cpu cores to get results faster.
## Why complicate the logic for the stopping conditions?
Ideally, we could only check two things: that our runs (the set of first collision data set sizes) is over a minimum threshold, and that our margin of error is under a maximum threshold.
Unfortunately, this isn't very computationally feasible for methods with large numbers for their first collision dataset size - it would just take too long to reach a lower margin of error. So we make a compromise, and allow for a higher margin of error (`0.10` rather than `0.05`) for methods with high numbers for their first collision data set size.
For example, the first collision dataset size for `int` is very large - so in this case, it isn't very feasible to run this until we obtained `0.05`. Let's say we ran it 100 times, and in total, all of the runs (i.e. all of the first collision data set sizes) summed up to over 999,999. We then decide that we're happy enough with a margin of error of `0.10`, and stop there.
## Why not analyse for the theoretical probabilities instead?
We're taking an empirical approach here, rather than analysing the code to work out the theoretical probabilities. The answer is that it is difficult for us to do this for each and every copycat API method in question, and know for certain these calculations are accurate - there are just too many different variables at play influencing the probabilities, and we wouldn't really know for sure if we've accounted for all of them. The only way we can know how the collision probabilities look in practice, is to actually run the code.
## Limitations
* I must admit that I know very little about the areas I'm wondering around in with this PR: both hash collision probabilities, and stats. If there's something important I'm missing with the approach I'm taking, I'd like to know!
* The approach required taking some shortcuts, since calculating the probabilities within better margins of error for some of the methods would take infeasibly long. See _Why such complex logic for the stopping conditions?_ above for more on this.
## How to read the results
For each api method:
*`mean` is roughly: the average data set size at which a collision happened (arithmetic mean)
* `stddev` is the standard deviation
* `min` is the worst-case run - the smallest data set size at which first collision happened
* `max` is the best-case run - the largest data set size at which first collision happened
* `runs` is the number of times we tested the api method before deciding the stats are "good enough"
* `sum` is the sum of all the runs - the total number of times the method was called
* `moe`, the "margin of error", basically means: we are 95% certain the "real" average dataset size at which the first collision happens is within `x` % of `mean` - for example, a `moe` of `0.05` means we are 95% certain the "real" average data set size at which the first collision happens is within `5` % of `mean`
* `hasCollided`, if we reached the maximum dataset size that we test collisions for (999,999)
## Results
Incomplete list (have yet to tweak the script to actually finish running in a reasonable time for the other methods):
```
{"methodName":"dateString","mean":"414.61","stddev":"214.23","moe":"0.05","runs":411,"min":5,"max":1412,"sum":170404,"hasCollided":true}
{"methodName":"fullName","mean":"1574.32","stddev":"784.96","moe":"0.05","runs":383,"min":75,"max":4853,"sum":602964,"hasCollided":true}
{"methodName":"streetAddress","mean":"22367.15","stddev":"12490.97","moe":"0.10","runs":122,"min":1154,"max":58605,"sum":2728792,"hasCollided":true}
{"methodName":"float","mean":"186526.06","stddev":"103245.91","moe":"0.10","runs":118,"min":12602,"max":550474,"sum":22010075,"hasCollided":true}
{"methodName":"ipv4","mean":"79714.37","stddev":"41439.56","moe":"0.10","runs":106,"min":6581,"max":205074,"sum":8449723,"hasCollided":true}
{"methodName":"email","mean":"119657.21","stddev":"59025.15","moe":"0.10","runs":100,"min":5254,"max":275462,"sum":11965721,"hasCollided":true}
```- Loading branch information
