Consider augmenting regex tests with "polyglot regex corpus"

[stephentoub]

This is a corpus of over 500k regexes in 8 different dialects:
https://github.com/SBULeeLab/LinguaFranca-FSE19/blob/master/data/production-regexes/uniq-regexes-8.json
from the paper:
https://people.cs.vt.edu/~davisjam/downloads/publications/DavisMichaelCoghlanServantLee-LinguaFranca-ESECFSE19.pdf
We should consider using them to augment our testing of Regex.

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This is a corpus of over 500k regexes from 8 different dialects:
https://github.com/SBULeeLab/LinguaFranca-FSE19/blob/master/data/production-regexes/uniq-regexes-8.json
from the paper:
https://people.cs.vt.edu/~davisjam/downloads/publications/DavisMichaelCoghlanServantLee-LinguaFranca-ESECFSE19.pdf
We should consider using them to augment our testing of Regex.

Author:	stephentoub
Assignees:	-
Labels:	area-System.Text.RegularExpressions
Milestone:	-

[danmoseley]

@stephentoub this repo has an MIT license, but it seems these regex were harvested from all over. Do we have confidence that they all have permissive licensing?

[stephentoub]

I've not investigated.

[danmoseley]

@davisjam -- I know you and @stephentoub briefly discussed your paper a few years back, I wonder whether there's anything you can say about the licenses which might apply to your 500K harvested regexes. For us to copy them into tests here, they need to be under a permissive license.

I'm guessing they're collected from projects with all kinds of licenses (or no declared license) so there's no way to identify which are permissively licensed?

Another thought -- were there any projects you found that had 100's or 1000's of regexes, where we could go check the license explicitly? We did grab a few such as an AT&T corpus, but perhaps there's another we didn't see yet.

[davisjam]

@danmoseley They were harvested without an eye to license. However, I'm in the process of updating the corpus and we can grab the licenses along the way. Do you have a specific definition of "permissive"?

[danmoseley]

Do you have a specific definition of "permissive"?

It's essentially excluding copy-left licenses. Licenses like MIT, BSD and Apache are acceptable.

@richlander do we have a definition or list of licenses that we consider compatible with our project?