Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixing filtered update performance #3233

Merged
merged 3 commits into from Jun 5, 2018

Conversation

Projects
None yet
5 participants
@viktor-svub
Copy link
Contributor

viktor-svub commented May 31, 2018

  • update with regex filter, matching 83 packages of 210, resulted in runtime of about 30 minutes, instead of the expected cca 30 seconds of full update
  • most of the time was spent in the filter function, constructing and compiling the regex for each nuget package dependency
  • issue was completely fixed by extracting the regex object creation to be done only single time

viktor-svub added some commits May 30, 2018

@@ -96,31 +96,26 @@ type QualifiedPackageName =
let packageName = PackageName packageName
QualifiedPackageName (groupName, packageName)

type PackageMatch(ex:String) =
member this.Expression = Regex("^" + ex + "$", RegexOptions.CultureInvariant ||| RegexOptions.IgnoreCase)

This comment has been minimized.

@matthid

matthid May 31, 2018

Member

Are you sure this no longer creates regex objects on every access?

This comment has been minimized.

@viktor-svub

viktor-svub May 31, 2018

Author Contributor

pretty much :)
as far I understand F# this should be constructor syntax, filling the Expression member with result of the expression after equals
and I'm seeing the described improvement on my machine, getting filtered update in approximately same time as full one
edit: including Debug build, so it's not just a result of compiler optimization, and it's indeed most likely called only on the filter instance creation

This comment has been minimized.

@0x53A

0x53A Jun 1, 2018

Contributor

Decompiled into C#:

		[CompilationMapping(SourceConstructFlags.ObjectType)]
		[Serializable]
		public class PackageMatch
		{
			public PackageMatch(string ex) : this()
			{
				this.ex = ex;
			}

			public Regex Expression
			{
				get
				{
					return new Regex("^" + this.ex + "$", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant);
				}
			}

			internal string ex;
		}

This comment has been minimized.

@0x53A

0x53A Jun 1, 2018

Contributor

I know there is a syntax to initialize it only once, I think it is something with val. Tbh, in that regard F# is really confusing.

This comment has been minimized.

@0x53A

0x53A Jun 1, 2018

Contributor

And would it maybe make sense to add ``RegexOptions.Compiled`? https://stackoverflow.com/questions/513412/how-does-regexoptions-compiled-work#7707369

This comment has been minimized.

@baronfel

baronfel Jun 1, 2018

Contributor

Here's a sharplab.io example: https://sharplab.io/#v2:DYLgZgzgPg9gDgUwHYAIDKBPCAXBBbAWAChjsNEUBhACgQA8QUcAnASyQHMBKFAXmJQo8+AEYJmKAG4BDYCgCidOMwQQIrGKl4p6KANQoARCOnNDAwYNLkEVAPoAlBAGMYeOAFdctBk2xtOHn4iQWE8MQlsAAtWCAA6RWVVdU0+HTp9IxMzIA===

You can see here that thanks to member val the property value is computed once on construction and then just reused, whereas for C_Recompute the property is recomputed each access.

This comment has been minimized.

@baronfel

baronfel Jun 1, 2018

Contributor

Note that we don't have access to System.Text.RegularExpressions in sharplab so that's why I didn't repro your scenario exactly.

This comment has been minimized.

@viktor-svub

viktor-svub Jun 1, 2018

Author Contributor

Oh my... :) Ok, I'll revise it to create the instance only once for sure.
At this moments it seems possible that all the slowdown was caused just by RegexOptions.Compiled while creating instance per comparison in either case.

@matthid

This comment has been minimized.

Copy link
Member

matthid commented May 31, 2018

Thanks for looking into this, always wondered why this is slower then full update

@viktor-svub

This comment has been minimized.

Copy link
Contributor Author

viktor-svub commented May 31, 2018

OK, I have zero idea what happened in the Travis CI ...
Error in '/usr/bin/mono': realloc(): invalid next size: 0x00000000025e0a70

@@ -96,31 +96,26 @@ type QualifiedPackageName =
let packageName = PackageName packageName
QualifiedPackageName (groupName, packageName)

type PackageMatch(ex:String) =
member val Expression = Regex("^" + ex + "$", RegexOptions.CultureInvariant ||| RegexOptions.IgnoreCase)

This comment has been minimized.

@matthid

matthid Jun 4, 2018

Member

I'm ok with this. Personally I would probably prefer:

type PackageMatch(ex:String) =
    let regex = Regex("^" + ex + "$", RegexOptions.CultureInvariant ||| RegexOptions.IgnoreCase) 
    member _.Expression = regex

or even

type PackageMatch =
   { Expression : Regex }
module PackageMatch =
   let ofString ex =
     { Expression = Regex("^" + ex + "$", RegexOptions.CultureInvariant ||| RegexOptions.IgnoreCase) }

@forki forki merged commit e49b28a into fsprojects:master Jun 5, 2018

1 of 2 checks passed

continuous-integration/appveyor/pr AppVeyor build failed
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.