Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Faceted queries now done in parallel #117

Closed
wants to merge 1,116 commits into
from

Conversation

Projects
None yet
7 participants

Sorry, the line endings got borked, the only change is https://github.com/mattwarren/ravendb/blob/1.2/Raven.Database/Queries/FacetedQueryRunner.cs#L35

On my machine, I see the following speed-ups using this new code (basically parallel takes 1/2 the time of sequential):

Query x => (x.Megapixels > 5.0)
Sequential: Took 2884.3698 ms
Parallel: Took 548.3615 ms

Query x => (x.Megapixels > 5.0)
Sequential: Took 434.9242 ms
Parallel: Took 201.7947 ms

Query x => ((x.Megapixels > 5.0) AndAlso (x.Cost < 500))
Sequential: Took 1175.9053 ms
Parallel: Took 328.5939 ms

Query x => ((x.Megapixels > 5.0) AndAlso (x.Cost < 500))
Sequential: Took 649.9667 ms
Parallel: Took 303.6177 ms

Query x => (((x.Megapixels > 5.0) AndAlso (x.Cost < 500)) AndAlso (x.DateOfListing > new DateTime(2000, 1, 1)))
Sequential: Took 1260.7684 ms
Parallel: Took 482.0374 ms

Query x => (((x.Megapixels > 5.0) AndAlso (x.Cost < 500)) AndAlso (x.DateOfListing > new DateTime(2000, 1, 1)))
Sequential: Took 1007.0183 ms
Parallel: Took 464.6603 ms

Query x => ((((x.Megapixels > 5.0) AndAlso (x.Cost < 500)) AndAlso (x.DateOfListing > new DateTime(2000, 1, 1))) AndAlso(x.Manufacturer == "Nikon"))
Sequential: Took 1100.8792 ms
Parallel: Took 500.6481 ms

Query x => ((((x.Megapixels > 5.0) AndAlso (x.Cost < 500)) AndAlso (x.DateOfListing > new DateTime(2000, 1, 1))) AndAlso(x.Manufacturer == "Nikon"))
Sequential: Took 1045.592 ms
Parallel: Took 473.6903 ms

Query x => (((((x.Megapixels > 5.0) AndAlso (x.Cost < 500)) AndAlso (x.DateOfListing > new DateTime(2000, 1, 1))) AndAlso (x.Manufacturer == "Nikon")) AndAlso (x.Model == "blah"))
Sequential: Took 1220.7945 ms
Parallel: Took 547.5513 ms

Query x => (((((x.Megapixels > 5.0) AndAlso (x.Cost < 500)) AndAlso (x.DateOfListing > new DateTime(2000, 1, 1))) AndAlso (x.Manufacturer == "Nikon")) AndAlso (x.Model == "blah"))
Sequential: Took 1166.1339 ms
Parallel: Took 522.5051 ms

samueldjack and others added some commits Jun 7, 2012

@samueldjack samueldjack Fixed RavenDB-345: Exception is now displayed if there is an error in…
… the JSON syntax when trying to save the document.
b1d6e21
@samueldjack samueldjack Fixed RavenDB-316: When deleting a document in the Edit Document page…
…, and document-set navigation is available, now navigates to the next document in the set.

Also, updated the breadcrumb bar so that it reflects how the user arrived at the document set.
0ff4402
@samueldjack samueldjack Made sure that over-long index names get trimmed when displayed in br…
…ead-crumb bars
5ef3c7d
@samueldjack samueldjack Implemented fade-trimming on index names in the indexes list view. 03e2d80
@samueldjack samueldjack Tweaked some styles 4d74e51
@samueldjack samueldjack Tweaks to Combobox styles 1bd3206
@ayende ayende Removing json.net dependency that is no longer required 99645a4
@ayende ayende Allow to use empty JsonProperty in queries 8b0c062
@ayende ayende Do not translate the names of fetched properties - They are loaded fr…
…om the document, not the index.
1f0e3ff
@ayende ayende Merge branch 'linq-long-count' of https://github.com/abombss/ravendb
…into 1.2
2056121
@ayende ayende We don't need to load even 1 doc to get the total results 4513505
@ayende ayende Making sure we can get count on dynamic queries when using page size = 0 0b91175
@samueldjack samueldjack Merge branch '1.2' of https://github.com/ayende/ravendb ba01ba1
@samueldjack samueldjack Made sure disabled toolbar buttons don't show a white background.
Fixed the borders of linq/query editors to be consistent with textboxes
ab64a85
@samueldjack samueldjack Improved the styling of the Edit document page, especially the search…
… controls.
dda2b76
@ayende ayende moving to a separate file 5c59b1a
@samueldjack samueldjack Working on RavenDB-315: Reformated toolbars in Edit Index page 73a1be2
@samueldjack samueldjack Working on RavenDB-315: Tweaked the toolbars on IndexesListView ac62a9d
@samueldjack samueldjack Working on RavenDB-315: Tweaked styling of the QueryView page, and ad…
…justed the toolbars
a865e98
@samueldjack samueldjack Tweaks to the header of the Logs,Tasks ,Statistics,IndexesErrors page…
…s. Made sure that all pages have titles
73b35e1
@samueldjack samueldjack Restyled the Index Errors page - now uses a DataGrid 7af0f35
@samueldjack samueldjack Made the index error notification more prominent at the bottom of the…
… edit index page.
92926c0
@samueldjack samueldjack Re-styled the Terms page a309ea6
@mattwarren mattwarren Initial work on Advanced Facet API (strongly typed, lambdas) c314a44
@mattwarren mattwarren Initial (failing) unit test bf9992e
@mattwarren mattwarren Spaces -> Tabs fix and better Generic Arg names 72dbac8
@ayende ayende Merge branch 'master' of https://github.com/samueldjack/ravendb into 1.2 b6fb7ab
@ayende ayende Fixing build on 3.5 0fec520
@ayende ayende Merge branch '1.2' of github.com:ayende/ravendb into 1.2 b2e73aa
@fitzchak fitzchak Convert linq expression to methods chine 872510d
@fitzchak fitzchak Support v1 of the server, which sent responses has strings instead of…
… objects, as v1.2 servers
3d3ed24
@fitzchak fitzchak Merge branch '1.2' of github.com:ayende/ravendb into 1.2 614065f
@ayende ayende Making sure that we can work with complex values recursively fdb177a
@fitzchak fitzchak Refactor FactIfSqlServerIsAvailable so it will be able to run from th…
…e stress test
210bc58
@fitzchak fitzchak Better test naming 92a3c1d
@fitzchak fitzchak minor 2e184cf
@fitzchak fitzchak Add StressTest for SQL Replication a147f2d
@ayende ayende Fixing tests cd6ad3d
@ayende ayende Merge branch '1.2' of github.com:ayende/ravendb into 1.2 e8d97aa
@fitzchak fitzchak Fix a bug where some parameters (int and DBNull) was missed 3504af1
@fitzchak fitzchak Revert "Add StressTest for SQL Replication"
Reason: not a race condition. Bug found in 1.2 branch.

This reverts commit a147f2d.
9e2795c
@ayende ayende Fixing test failure 39c5bed
@ayende ayende Merge branch '1.2' of https://github.com/fitzchak/ravendb into 1.2 8f5ef23
@ayende ayende Merge branch 'master' of https://github.com/fitzchak/ravendb into 1.2 2b6bd98
@ayende ayende Fixing an issue where Cutoff Etag wasn't clear properly 79f9231
@fitzchak fitzchak We can test ConcurrentlyOpenedTenantsUsingEsent with 100 tenants, bec…
…ause we also stress test this test.
7788fe0
@ayende ayende Can also handle databases that have . - _ in their names 205de10
@ayende ayende Make sure to use the default credentials 58edd44
@ayende ayende Minor eda5711
@ayende ayende Resolving RavenDB-255 - Will now consider the user account running Ra…
…venDB as an admin as well
35d0c15
@ayende ayende Adding admin statistics 2a4480d
@ayende ayende Adding support for filtering bundles, now can specify them using Rave…
…n/ActiveBundles as semi column separated value.
1f24c32
@ayende ayende Fixing how we can filter bundles 9116905
@ayende ayende Making sure that the replication bundle can be opt in c27a5ee
@ayende ayende Recording the moving of a file 80f8dfa
@ayende ayende Fixing build of 3.5 a6010c6
@fitzchak fitzchak Let's see how much useful that is 8b78ba4
@ayende ayende Fixing merge 62ec20e
@synhershko synhershko TODO e9f09a6
@synhershko synhershko Adding failing test d24c198

samueldjack and others added some commits Jun 27, 2012

@samueldjack samueldjack Fixed bug which prevented SyntaxEditor parsing being done on a backgr…
…ound thread
779916a
@samueldjack samueldjack Made sure open curly is always visible when collapsing region 974a3d0
@ayende ayende Adding support for complex includes (Include(x=>x.Friends.Select(y=>y…
….Id)) )
c83abe1
@ayende ayende REMOVING SUPPORT FOR .NET 3.5 bc97042
@ayende ayende force removal of write.lock in crash recovery scenarios d3c1c4c
@ayende ayende Better error messages 9fddc6d
@ayende ayende Severity is actually an enum string a2ae295
@ayende ayende Don't try to force the lcid on the index (fails sometimes) 404d4f3
@samueldjack samueldjack Re-implemented JSON prettifying feature using an Actipro ITextFormatt…
…er so that it works even if there are syntax errors.
d5bbb7a
@ayende ayende Use old API to work on Win XP 2494335
@samueldjack samueldjack Fixed race condition f488db3
@samueldjack samueldjack Implemented RavenDB-373: There's now a button that enables the auto-c…
…ollapse mode.
d8c6276
@ayende ayende Support interfaces for AddMapToAll 1e1f14d
@ayende ayende Merge branch '1.2' of github.com:fitzchak/ravendb into 1.2 cbb68ef
@ayende ayende Minor 8a7f5cd
@pierslawson @ayende pierslawson + ayende Test handling of string and value type Id fields
In this thread
https://groups.google.com/forum/?fromgroups#!topic/ravendb/1gNqNW26bhk I
describe how I thought how the Include via Linq and Load via Linq
operations seemed contradictory in how they treated a string based Id.
On further investigation I convinced myself I had just misunderstood how
o use string based Ids. Oren thought it might be  abug and asked me to
reproduce it in  a failing test. I have added one test that passes
becuase it is using value type Ids and one that fails because it is
using string based Ids.

Conflicts:

	Raven.Tests/Bugs/Queries/Includes.cs
1b8ab9b
@samueldjack samueldjack Merge branch '1.2' of https://github.com/ayende/ravendb 742bfa0
@ayende ayende Fixing test 603f105
@samueldjack samueldjack Fixed bug where default document size was not being applied when relo…
…ading app.
a482d99
@samueldjack samueldjack Fixed bug where documents were sometimes not auto-collapsed 4f3bc52
@samueldjack samueldjack Implemented RavenDB-368: The query page now has a toggle button calle…
…d "Show Fields" which will show the stored index fields in the results rather than the document.
79b97e6
@ayende ayende Adding passing test 07da187
@ayende ayende Merge branch 'master' of git://github.com/samueldjack/ravendb into 1.2 36ef7ff
@ayende ayende Fixing merge 7d08bc4
@ayende ayende Made sure that we can parse old style date queries 4dbe8ef
@ayende ayende Removing dead code from SchemaCreator.cs
Testing raw write perf
25353e6
@pierslawson pierslawson Passing tests checking Include behaviour 67b1168
@ayende ayende Merge branch '1.2' of https://github.com/pierslawson/ravendb into 1.2 c689a88
@ayende ayende Fixed date parsing in lazy requests
Fixed query modified when not needed
e57902f
@ayende ayende Proper arg checking 12679ee
@ayende ayende Fixing an issue with local transaction identifier always being though…
…t of as the same
e69d892
@ayende ayende MAJOR speed improvement in the way we are doing Sort By Distance usin…
…g Spatial Queries
685c3a2

Nice fix, that's a big speed boost.

Just one thing, if I look at the code here https://github.com/ayende/ravendb/blob/1.2/Raven.Database/Indexing/Sorting/SpatialDistanceSortField.cs#L38, I can't see where the ConditionalWeakTable cache gets used? Am I missing something, is it meant to still be in there?

It's not, and we should be leveraging the cache held in Spatial4n instead of applying our own here. That requires some API changes there hence wasn't implemented yet.

Owner

ayende replied Jul 1, 2012

ayende and others added some commits Jul 1, 2012

@ayende ayende Removing stupid vestigal field 2660912
@ayende ayende Fixing RavenDB-389 - Moving Replication bundle to the core 1df924e
@ayende ayende Fixing RavenDB-81 bdcddc2
@ayende ayende BREAKING CHANGE
We now require an explicit mention of the bundle inside Raven/ActiveBundles to register a builtin bundle, it is not enough to just be inside the Plugins folder
521a47a
@ayende ayende Fixing RavenDB-360 - Content-Type is no longer shown in the metadata 3b72250
@ayende ayende RavenDB-359 - Changing "default" to "system" as the default database …
…name
e60739e
@ayende ayende RavenDB-388 - Auto reset the index if the index version does not matc…
…h the server expected version.
0af8270
@ayende ayende Fixing RavenDB-384 - attachment starts with 425636c
@ayende ayende minor 7350234
@ayende ayende making sure that we don't filter the content type for attachments ac91033
@ayende ayende fixing tests 8932b67
@ayende ayende making sure to build for AnyCPU e9d7b9b
@ayende ayende fixing build 6a2fd8b
@ayende ayende fixing replication tests 0ee046d
@ayende ayende Also limiting documents indexing count by actual size as well acf206f
@ayende ayende More smuggler options e4904e2
@ayende ayende Reserve just a little bit more space 28b762e
@ayende ayende More robust smuggler behavior ada5753
@ayende ayende Fixing tests 8683cca
@ayende ayende Allow to report pushed size 2abb805
@ayende ayende Better smuggler output 2d3a708
@ayende ayende Better output 5a9c94f
@ayende ayende Much better smuggler reporting d3da9c7
@ayende ayende Limit the amount of memory that can be loaded from storage for a sing…
…le indexing run.

Take into account such factors as current memory utilization, document sizes, etc.
5fff7f7
@ayende ayende Fixing build fdee3a5
@mattwarren mattwarren Faceted queries are now done in parallel using BackgroundTaskExecuter…
… (so it respects the user settings of how many cores to use)
47400ac

@mattwarren mattwarren closed this Jul 4, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment