Skip to content
This repository has been archived by the owner on Jan 3, 2022. It is now read-only.

Commit

Permalink
Merge pull request #23 from fastwildcard/11-benchmarks
Browse files Browse the repository at this point in the history
Reworked benchmarks to behave more like real world
  • Loading branch information
alexangas committed Oct 18, 2018
2 parents 04a1580 + e1157cd commit 5a4999f
Show file tree
Hide file tree
Showing 14 changed files with 263 additions and 126 deletions.
2 changes: 2 additions & 0 deletions FastWildcard.sln
Expand Up @@ -29,9 +29,11 @@ Global
{5F59A854-96EF-4337-85D2-D67F4F985AFD}.Release|Any CPU.ActiveCfg = Release|Any CPU
{5F59A854-96EF-4337-85D2-D67F4F985AFD}.Release|Any CPU.Build.0 = Release|Any CPU
{5C737FA0-EBE2-48E4-A03A-B9327931311D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{5C737FA0-EBE2-48E4-A03A-B9327931311D}.Debug|Any CPU.Build.0 = Debug|Any CPU
{5C737FA0-EBE2-48E4-A03A-B9327931311D}.Release|Any CPU.ActiveCfg = Release|Any CPU
{5C737FA0-EBE2-48E4-A03A-B9327931311D}.Release|Any CPU.Build.0 = Release|Any CPU
{AD18367B-D8F9-4992-A32C-4F2D2DB99794}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{AD18367B-D8F9-4992-A32C-4F2D2DB99794}.Debug|Any CPU.Build.0 = Debug|Any CPU
{AD18367B-D8F9-4992-A32C-4F2D2DB99794}.Release|Any CPU.ActiveCfg = Release|Any CPU
{AD18367B-D8F9-4992-A32C-4F2D2DB99794}.Release|Any CPU.Build.0 = Release|Any CPU
{57D8C898-2699-4661-A95C-665542903887}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
Expand Down
78 changes: 12 additions & 66 deletions README.md
Expand Up @@ -6,85 +6,31 @@
[![codecov](https://codecov.io/gh/fastwildcard/fastwildcard/branch/master/graph/badge.svg)](https://codecov.io/gh/fastwildcard/fastwildcard)
[![Semantic Versioning](https://img.shields.io/badge/semver-2.0.0-3D9FE0.svg)](http://semver.org/)

A fast and robust wildcard matching library for .NET Standard.
A wildcard matching library for .NET Core 2.x, and .NET Standard 1.3/2.0.

* _*Robust:*_ extensively unit tested to be reliable and predictable - no edge cases!
* _*Fast:*_ developed with regular benchmarking and performance analysis to provide the best speed possible!

## Pattern syntax

* '*' matches 0 or more characters
* '?' matches 1 character

For examples, please review the `FastWildcard.Tests` project.
For examples, please review some [`FastWildcard.Tests`](blob/master/tests/FastWildcard.Tests/IsMatchTests.cs)!

## Benchmarks

### Cross-framework

Test case details:

* randomly generated string of 25 alphanumeric characters
* 2 single character '?' wildcards anywhere in the string
* 2 multi character '*' wildcards anywhere in the string (may overwrite single character wildcard)

The same string was executed against each target.
FastWildcard version 2.0.1 was used.

Test execution details:

``` ini

BenchmarkDotNet=v0.10.14, OS=Windows 10.0.17134
Intel Core i7-7500U CPU 2.70GHz (Kaby Lake), 1 CPU, 4 logical and 2 physical cores
.NET Core SDK=2.1.200
[Host] : .NET Core 2.0.7 (CoreCLR 4.6.26328.01, CoreFX 4.6.26403.03), 64bit RyuJIT
Clr : .NET Framework 4.7.1 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3101.0
Core : .NET Core 2.0.7 (CoreCLR 4.6.26328.01, CoreFX 4.6.26403.03), 64bit RyuJIT


```
![Cross-library comparison](reports/FastWildcard.Performance.Benchmarks.LibraryComparison-report.png)

| Method | Job | Runtime | Mean | Error | StdDev | Median |
|-------------- |----- |-------- |----------:|----------:|----------:|----------:|
| FastWildcard | Clr | Clr | 6.701 ns | 0.1356 ns | 0.1058 ns | 6.710 ns |
| Regex | Clr | Clr | 57.737 ns | 1.1910 ns | 1.7458 ns | 57.375 ns |
| RegexCompiled | Clr | Clr | 79.653 ns | 0.9964 ns | 0.9320 ns | 79.400 ns |
| FastWildcard | Core | Core | 8.865 ns | 1.1381 ns | 1.0089 ns | 8.619 ns |
| Regex | Core | Core | 62.321 ns | 1.2019 ns | 1.0655 ns | 62.290 ns |
| RegexCompiled | Core | Core | 62.960 ns | 1.2891 ns | 3.0133 ns | 61.906 ns |
Notes:

### Cross-library
* Executed against FastWildcard 3.0.0 on an Azure Standard F2s virtual machine
* Compiled RegEx is excluded as under the full CLR its execution time is extremely high
* Values for WildcardMatch are an average as it encountered execution errors

Test case details:

* randomly generated string of 25 alphanumeric characters
* 2 single character '?' wildcards within the first 11 characters
* 2 multi character '*' wildcards within the last 12 characters

The same string was executed against each target.
FastWildcard version 2.0.1 was used.

Test execution details:

``` ini

BenchmarkDotNet=v0.10.14, OS=Windows 10.0.17134
Intel Core i7-7500U CPU 2.70GHz (Kaby Lake), 1 CPU, 4 logical and 2 physical cores
.NET Core SDK=2.1.200
[Host] : .NET Core 2.0.7 (CoreCLR 4.6.26328.01, CoreFX 4.6.26403.03), 64bit RyuJIT
Clr : .NET Framework 4.7.1 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3101.0

Job=Clr Runtime=Clr

```

| Method | Mean | Error | StdDev | Median |
|-------------- |----------:|----------:|----------:|----------:|
| FastWildcard | 7.016 ns | 0.3243 ns | 0.7894 ns | 6.635 ns |
| WildcardMatch | 11.285 ns | 0.2844 ns | 0.4169 ns | 11.175 ns |
| Regex | 57.502 ns | 1.1226 ns | 1.0501 ns | 57.316 ns |
| RegexCompiled | 80.180 ns | 1.5398 ns | 1.4404 ns | 79.957 ns |
Complete [test execution log](reports/FastWildcard.Performance.Benchmarks.LibraryComparison.log) and [reporting spreadsheet](reports/FastWildcard.Performance.Benchmarks.LibraryComparison-report.xlsx).
Or run [FastWildcard.Performance](tree/master/tests/FastWildcard.Performance) yourself!

## Support details

Supports .NET Standard 1.3 and higher.

Follows semantic versioning.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
2 changes: 1 addition & 1 deletion src/FastWildcard/FastWildcard.csproj
Expand Up @@ -21,8 +21,8 @@
<summary>A fast and robust wildcard matching library for .NET.</summary>
<description>
A wildcard matching library for .NET Core 2.x, and .NET Standard 1.3/2.0.
Fast: developed with regular benchmarking and performance analysis to provide the best speed possible!
Robust: extensively unit tested to be reliable and predictable - no edge cases!
Fast: developed with regular benchmarking and performance analysis to provide the best speed possible!
</description>
<language>en-US</language>
<releaseNotes>Please see https://github.com/fastwildcard/fastwildcard/releases/ for release notes.</releaseNotes>
Expand Down
@@ -1,17 +1,19 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFrameworks>netcoreapp2.0;netcoreapp2.1;net461</TargetFrameworks>
<TargetFrameworks>netcoreapp2.1;netcoreapp2.0;net461</TargetFrameworks>
</PropertyGroup>

<ItemGroup>
<ProjectReference Include="..\FastWildcard.Performance\FastWildcard.Performance.csproj" />
</ItemGroup>

<ItemGroup>
<PackageReference Include="BenchmarkDotNet" Version="0.11.0" />
<PackageReference Include="Bogus" Version="22.3.2" />
<PackageReference Include="FluentAssertions" Version="5.4.1" />
<PackageReference Include="AutoFixture.Xunit2" Version="4.5.0" />
<PackageReference Include="BenchmarkDotNet" Version="0.11.1" />
<PackageReference Include="Bogus" Version="24.3.0" />
<PackageReference Include="FluentAssertions" Version="5.4.2" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.9.0" />
<PackageReference Include="xunit" Version="2.4.0" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.0" />
</ItemGroup>
Expand Down
65 changes: 65 additions & 0 deletions tests/FastWildcard.Performance.Tests/IterationBuilderTests.cs
@@ -0,0 +1,65 @@
using System;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using AutoFixture.Xunit2;
using FastWildcard.Performance.Benchmarks;
using FluentAssertions;
using Xunit;

namespace FastWildcard.Performance.Tests
{
public class IterationBuilderTests
{
[Theory, AutoData]
public void BuildPattern_InvalidParameter_ThrowsException(
[Range(-100, 0)] int patternLength,
[Range(-100, -1)] int singleCharacterCount,
[Range(-100, -1)] int multiCharacterCount
)
{
Action action = () => IterationBuilder.BuildPattern(patternLength, singleCharacterCount, multiCharacterCount);

action.Should().Throw<ArgumentOutOfRangeException>();
}

[Theory, AutoData]
public void BuildPattern_OfLength_ReturnsStringWithLength(
[Range(1, 100)] int patternLength,
[Range(0, 100)] int singleCharacterCount,
[Range(0, 100)] int multiCharacterCount
)
{
var (result, newSingleCount, newMultiCount) = IterationBuilder.BuildPattern(patternLength, singleCharacterCount, multiCharacterCount);

result.Should().NotBeNullOrWhiteSpace()
.And.HaveLength(patternLength)
.And.Match(s => s.Count(ch => ch == '?') == newSingleCount)
.And.Match(s => s.Count(ch => ch == '*') == newMultiCount);
}

[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
public void BuildTestString_InvalidParameter_ThrowsException(string pattern)
{
Action action = () => IterationBuilder.BuildTestString(pattern);

action.Should().Throw<ArgumentException>();
}

[Theory, AutoData]
public void BuildTestString_OfLength_ReturnsStringWithLength(
[Range(1, 100)] int patternLength,
[Range(0, 100)] int singleCharacterCount,
[Range(0, 100)] int multiCharacterCount
)
{
var (pattern, _, _) = IterationBuilder.BuildPattern(patternLength, singleCharacterCount, multiCharacterCount);

var result = IterationBuilder.BuildTestString(pattern);

result.Should().NotBeNullOrWhiteSpace();
}
}
}
94 changes: 94 additions & 0 deletions tests/FastWildcard.Performance/Benchmarks/IterationBuilder.cs
@@ -0,0 +1,94 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace FastWildcard.Performance.Benchmarks
{
public static class IterationBuilder
{
public static (string result, int newSingleCount, int newMultiCount) BuildPattern(int patternLength, int singleCountSuggestion, int multiCountSuggestion)
{
const int wildcardPercentage = 10;

if (patternLength <= 0)
throw new ArgumentOutOfRangeException(nameof(patternLength));
if (singleCountSuggestion < 0)
throw new ArgumentOutOfRangeException(nameof(singleCountSuggestion));
if (multiCountSuggestion < 0)
throw new ArgumentOutOfRangeException(nameof(multiCountSuggestion));

var adjustedSingleCount = Math.Min(singleCountSuggestion, patternLength) / wildcardPercentage;
var adjustedMultiCount = Math.Min(multiCountSuggestion, patternLength) / wildcardPercentage;

var patternBuilder = new StringBuilder(new Bogus.Randomizer().AlphaNumeric(patternLength));

var random = new Random();

var singleCharacterLocations = new List<int>();
if (singleCountSuggestion > 0)
{
singleCharacterLocations = Enumerable.Range(0, adjustedSingleCount)
.Select(x => random.Next(0, patternLength))
.ToList();
singleCharacterLocations.ForEach(x => patternBuilder[x] = '?');
}

var multiCharacterLocations = new List<int>();
if (multiCountSuggestion > 0)
{
multiCharacterLocations = Enumerable.Range(0, adjustedMultiCount)
.Select(x => random.Next(0, patternLength))
.Where(x => !singleCharacterLocations.Contains(x))
.ToList();
multiCharacterLocations.ForEach(x => patternBuilder[x] = '*');
}

var pattern = patternBuilder.ToString();
return (pattern, singleCharacterLocations.Count, multiCharacterLocations.Count);
}

public static string BuildTestString(string pattern)
{
if (string.IsNullOrWhiteSpace(pattern))
throw new ArgumentException("Value cannot be null or whitespace.", nameof(pattern));

var randomizer = new Bogus.Randomizer();

const int noMatchPercentage = 50;
if (WeightedMatch(noMatchPercentage))
{
return randomizer.String();
}

const int charMatchPercentage = 95;
var strBuilder = new StringBuilder(pattern.Length * 2);
foreach (var patternCh in pattern)
{
if (patternCh == '?')
{
strBuilder.Append(randomizer.AlphaNumeric(1));
}
else if (patternCh == '*')
{
strBuilder.Append(randomizer.Words());
}
else
{
if (WeightedMatch(charMatchPercentage))
{
strBuilder.Append(patternCh);
}
}
}

var str = strBuilder.ToString();
return str;

bool WeightedMatch(int percentage)
{
return randomizer.Bool(percentage / 100f);
}
}
}
}
@@ -1,22 +1,23 @@
using System;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;
using BenchmarkDotNet.Attributes;
using FastWildcard.Performance.Matchers;

namespace FastWildcard.Performance.Benchmarks
{
[ClrJob]
public class StandardLength
[MemoryDiagnoser]
public class LibraryComparison
{
private const int StringLength = 25;
private const int SingleCharacterMatchCount = 2;
private const int SingleCharacterStart = 0;
private const int SingleCharacterEnd = 10;
private const int MultiCharacterMatchCount = 2;
private const int MultiCharacterStart = 12;
private const int MultiCharacterEnd = 24;
[Params(10, 50)]
public int PatternLength { get; set; }

[Params(2, 5)]
public int SingleCharacterCount { get; set; }

[Params(1, 3)]
public int MultiCharacterCount { get; set; }

private string _pattern;
private string _str;
private FastWildcardMatcher _fastWildcardMatcher;
Expand All @@ -29,30 +30,16 @@ public class StandardLength
[IterationSetup]
public void IterationSetup()
{
var patternBuilder = new StringBuilder(new Bogus.Randomizer().AlphaNumeric(StringLength));
(_pattern, _, _) = IterationBuilder.BuildPattern(PatternLength, SingleCharacterCount, MultiCharacterCount);

var random = new Random();

var singleCharacterLocations = Enumerable.Range(0, SingleCharacterMatchCount)
.Select(x => random.Next(SingleCharacterStart, SingleCharacterEnd))
.ToList();
singleCharacterLocations.ForEach(x => patternBuilder[x] = '?');

var multiCharacterLocations = Enumerable.Range(0, MultiCharacterMatchCount)
.Select(x => random.Next(MultiCharacterStart, MultiCharacterEnd))
.ToList();
multiCharacterLocations.ForEach(x => patternBuilder[x] = '*');

_pattern = patternBuilder.ToString();
_str = IterationBuilder.BuildTestString(_pattern);

_fastWildcardMatcher = new FastWildcardMatcher();
_regexMatcher = new RegexMatcher(_pattern, RegexOptions.None);
_regexMatcherCompiled = new RegexMatcher(_pattern, RegexOptions.Compiled);
#if !NETCOREAPP
_wildcardMatchMatcher = new WildcardMatchMatcher();
#endif

_str = new Bogus.Randomizer().AlphaNumeric(StringLength);
}

[Benchmark]
Expand Down

0 comments on commit 5a4999f

Please sign in to comment.