Segmentation Fault auf dem Mac #2
Assignees
Comments
|
Ich habe auch mal frisch gebaut. Leider tritt der Fehler nicht auf:
Kannst du mal ein neues Verzeichnis /tmp/vocbase2 anlegen und es damit probieren? vielen Dank |
|
Aha! Der Fehler tritt nur auf, wenn ich ihm kein Directory für die Datenbank mitgebe! War also ein Benutzer Fehler! |
|
Hm, bei mir kommt: fceller@hahnwald: Mit welchen Parametern startest du? |
|
|
Irgendwas ist auch deinem Mac anders. Kannst du es mal mit GDB ausprobieren?
(gdb) run Program exited with code 01. |
|
|
Kannst du nach dem SEGFAULT noch ein (gdb) where ausführen? |
|
|
Ok. Kannst du die Verzeichnisse in /var/lib/avocado/ mal weg-werfen? Ich hatte - glaube ich - noch eine Änderung am Datenformat gemacht. Das könnte die Ursache für das Problem sein. |
|
Ja, dass hat es gelöst! Vielen Dank :) Kannst das Ticket schließen (ich darf nicht ;) )! |
|
Done |
5 tasks
27 tasks
5 tasks
31 tasks
Closed
This was referenced Dec 10, 2019
ghost
mentioned this issue
mchacki
added a commit
that referenced
this issue
Mar 10, 2020
* Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather * [WIP] Changed Api for MultiDepExecutors in ExecBlockImpl (not yet in the actual executors) * Moved AqlCallSet into a separate file * Changed SortingGather to use the new API * Changed ParallelUnsortedGather to use the new API * Changed UnsortedGather to use the new API * Moved AqlCall operator<< into .cpp file * Implement operator<< for AqlCallSet * Fix boolean mix-up * Fixed state machine: go to UPSTREAM when the AqlCallSet is not empty * Fixed assertion * Bugfix * SortingGather bugfixes * Added init() method to fix an assertion and cleanup * Removed unused variable * Fixed constrained sort * Fixed constrained sort #2 * Fix boolean mix-up * Remove old interface * Use call parameter for upstream request in produceRows * Remove more old interface code * Add skip methods to MultiAqlItemBlockInputRange * Skip in UnsortedGather * skip for UnsortedGather * Fix skip and upstream calls for UnsortedGather * skipRowsRange change * Remove useless comments * Moved multi-dep related code from ExeBlockImpl to MultiFetcher * Cleanup in SortingGather, implemented parallel fullCount * Try to fix a windows compile error * Simplify and extend skipRowsRange for UnsortedGatherExecutor * Made ParallelUnsortedGather actually parallel * Removed erroneous assertion * Undid erroneous change * Fixed MacOs compile. Also disabled tests for non-relevant AqlCallStacks. They will be removed * Fixed initialize Cursor for multi dependency blocks * Fixed fullCount case in parallel unsorted gather * Fixed fullCount upstream call of ParallelUnsortedGatherExecutor * Fixed fullCount in SortingGather * Windows \o/ if you cannot work properly with constexpr and static asserts, we do not let you do it! * Do not advance in Unsorted gather if there are still rows to skip * Add more comparison operators for AqlCall limits * Send clientCall limits to upstream in SortingGather * Improved fullCount in SortingGatherExectur * Disabled a cluster profile test. We now ask the RemoteNode more often if it already has data. It is a bit unclear to me if this is now better performance wise (<< i think so) or triggers undesired side effects * Helpless attempt to work around issues in stonage Visual Studio Compiler we are using. * Clearly adding an operator on a well defined type causes ambigousness on random basic types using the operator Co-authored-by: Michael Hackstein <michael@arangodb.com> Co-authored-by: Markus Pfeiffer <markus@arangodb.com>
mchacki
added a commit
that referenced
this issue
Mar 13, 2020
) * Added first draft of AqlCallObject * Added AqlItemBlockInputRange * Added Header file for AqlCallStack with comments. No implementation yet. * Added AqlCallStack first draft implementation. Made everything compile * Added empty dummy stubs for execute() in ExecutionBlock * Implementation of new API in FilterExecutor. Rough implementation in ExecutionBlockImpl (not complete, only skip path) everything compiles, but is not tested * Fixed off by one error in InputRange. Fixed FilterExecutor * Added some more tests if executor cannot fulfill atMost * Added heplper functions to the AqlCall struct to avoid mangeling around with variants everywhere. * Add a member DataRange to ExecutionBlockImpl. Also now all Fetcher expose a DataRange, this might not be the correct one every where * Added preliminary minimal implementation of execute to DependencyProxy and SingleRowFetcher. Both untested and yet incomplete, they will only work for MVP. * added first implementation of count collect datarange produceRows function + test * Revert "added first implementation of count collect datarange produceRows function + test" This reverts commit 236825280bb5fd03dfe01febfc4dcf1a2d2e2c2f. * First draft of execute implementation, not yet handling ShadowRows though. This state is broken in the sense that a server does not boot with it and catch tests are broken * First working draft of execute() call in ExecutionBlockImpl. Tests are locally green, however there is at least one query still red (on ShadowRows) * Removed non finished implementation from this Branch. It moved to seperate branch * working - not done yet * Revert "working - not done yet" This reverts commit aa275b48ffc2c8fc9a3c364e2241321ec9c9293f. * first implementation of a ShadowRow fetching interface on AqlItemBlockInputRange * single row fetcher, execute + tests * Fixed SingleRowFetcherTest. Also an AqlItemBlockInputRange can now handle ShadowRows properly * Another fix on handling of shadow rows in InputRanges * Improved human readable output of SortLimit test, while validating the result * Removed debug profile execution of a query * Use simpleVPack printing for Trace of Execute, better to read by humans * Added tests for AqlItemBlockInputRange and adjusted implementation accordingly. * Fixed issue where ExecutionBlockImpl returns DONE, although there are still rows in the DataRange * Final modification of ShadowRows in FilterExecutor. All but profiling tests pass. This needs some discussion about overfetching of data, and propagation of hardLimit * Removed debug log includes * Moved responsibility for the client call into the output row. This can precisely count the elements it has written, and forms a single source of truth. The output row now works as the requested block by client that needs to be filled. Now the FilterExecutorTests are red. * Fixed FilterExecutorTest * Expose required features to AqlCall controlled by the output aql item row. Adapted executionBlockImpl to report there * Fixed bug with AqlCallStack pop operation that actually caused invalid memory access * outputRow now honors limits whan called for isFull. * Fixed isFull() in OutputAqlItemRow * Use the inbound call in skipRows for executors, in order to do a more precise upstream call * Make linker happy with operator< on CallLimit * Implement operator+ on AqlCall::Limit, also use std::visit for readability. * FilterExecutor getSome now propagates offset + min(limits) to upstream. This is pessimistic and avoids overfetching in all cases. NOTE: FilterExecutor should be moved to skipvariant using getSome * Fixed profiler tests for filter. These cannot be guaranteed anymore and it was an undesired, yet uncritical side effect that the tests have been green before * Fixed JSLint * Remove special casing for FilterExecutor * use if constexpr to distinguish between ported executors and non-ported executors enabling step-by-step porting and testing * Add skeleton skip code * Replace C&P accident * Simplify and implement skip for new executor interface * Fixup skipping for filter executor * Rename some methods for clarity and consistency * AqlItemBlockInputRange - hasMore is now hasDataRow - next is now nextDataRow - peek is now peekDataRow - state is now upstreamState - additional method upstreamHasMore * Introduce function to allocate an output block * Fix skipSome simulation bug * Some small cleanups * Replaced assertion on atMost on the output size. Otherwise we got into issues with atMost changes between waiting calls (can occur in our intermediate state now) * Attempt at fixing execute * Fix merge conflict * Feature/aql subquery execution block impl execute implementation exec block impl tests (#10838) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation shortest path executor (#10780) * Hook up new ShortestPathExecutor * Small fixes * Removed now unused code. * Moved ShortestPathExecutor tests to use Execute only. * Added a test for shortestPathExecutor where the output is smaller than the amount of entries within the path. * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Fixed Windows compile issue * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Windows compile fix * Attempt at fixing ShortestPathExecutorTest for Windows * Windows compile fix * Added a skip test and fixed the producing executor to recreate the list once for every input line. * Frankenstein my tests into the branch * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Hack * Removed designated initializers. Thanks for not supporting it MSVC! * Update ShortestPathExecutorTest * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Umbau 2.0 * Reshape output and skip a bit * Some test fixes * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Square 1 * Input validation * ShortestPathExecutor with fullCount and tests Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation exec block impl tests shadow rows (#10970) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Added tests for ExecutionBlockImpl, when there is a WAITING passed through. There is no test yet for in between waitings. * Allow to configure WAITING variant in WAITING block mock. * Use the WaitingBlockMock to be independent of Singleton in test * Added WAITING + Passthrough tests * Added a test to path through shadowRows without data. * Waiting ExecutionBlock Mock can now return/skip partial blocks. However it does NOT CARE for ShadowRows * Added tests and fixed an issue in ShadowRow forwarding * Added tests for shadowRow forwarding * Added the intenral state of Execute as member of ExecutionBlockImpl. We need to retain parts of this state accross returns * Fixed all Waiting and multiple call tests * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Fixed invalid memory access in test code. Do never return a lambda that references a stack variable... * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip * Allways -> Always. * Added tests for pass-by of non-relevant Subqueries. Fixed a bug found on the way in WaitingBlockMock. Now some other tests fail \o/ * Fixed failed tests. Now tests correspond to correct behaviour of WaitingExecutionBlockMock * Intermediate version, doe snot work * Fixed the empty subquery tests Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation sorted collect (#11020) * SortedCollect executor and some tests. * More tests. * More const and skip implemented. * Fixed some bugs found by tests. * Refactored skip. * Broken state with test framework. * Extracted testing framework into separate file and adjusted it to use WaitingBlockMock. However it is not yet fully implemented * Added another SortedExecutor test without fullcount. This passes. Also assert on skip and State * Removed obsolete assertion in skip, added another nonFullcount test. * More tests. Fixed some bugs in the test framework. * More cleanup. * Remove version control marker. * Removed wrong assertion. * Apply suggestions from code review Co-Authored-By: Michael Hackstein <michael@arangodb.com> * Remove dead code. * Removed fetcher. Co-authored-by: Michael Hackstein <michael@arangodb.com> * ReturnExecutor New style (#10831) * ReturnExecutor implemented. Tests misssing. * Apply suggestions from code review Co-Authored-By: Tobias Gödderz <tobias@arangodb.com> * Implemented skipRows in Return. Also fixed ExecutionBlock references in static asserts * Implemented first new style test for Return. * Moved the traceExecute calls into baseclass. * Added tracing around execute in WaitingExecutionBlockMock. Fixed a test bug, if the AqlCall covers all of the first block, but only a part of the Second block. You would only get the part of the second block. * Added more tests for standard behaviour of ReturnExecutor * Added an assertion. * Simplified call to ReturnExecutor * Prepare tests for doCount. Also added test for different input register. * Added accessor for _stats in Engine. * Included count tests in ReturnExecutorTest. * Removed old obsolete test. * Removed obsolete code-path * Fixed unrelated bug in FilterExecutor. In some cases it did lie on skipRows that it has more, even if it does not have and the upstream does not have as well. Thank you jenkins for hitting into this race :+1: Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Tweak ExecutorTestHelper to test pipelines * Revert "Tweak ExecutorTestHelper to test pipelines" This reverts commit 6359f0c926cad30b4efd7ce263fb3b64563621b8. * Added Stats return value to skipRowsRange (#11081) * Added Stats return value to skipRowsRange * Fixed catch test. Somehow i merged the incorrect part Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Additional Assertion in ExecutionBlockImpl (#11077) * Added a assertion arround skipRowsInRange that ensures that the Call and the skippedLocal do not get out-of-sync * Improved the assertion * Feature/aql subquery operations stack id executor (#10986) * added inputRange skip and produce * added inputRange skip and produce * Added shouldSkip helper to AqlCall * Added implementation if execute to Constfetcher. Not specifically tested yet. * Moved IDExecutor to newStyle executor * Added some documentation to SingleRowFetcher * Adapted IdExecutor tests. * Added an assertion in ConstFetcher * Fixed initialize cursor in ExecutionBlockImpl * Added tests for Initilaize cursor in IdExecutor. There is a special implementation for this... * Fixed linker issues, because of variables in cpp files named identical and not beeing static * Started refactoring the Scatter/Gather Executor logic. Added an intermediate class that does not use a template and can be casted to. This state compiles but will not work. * Made BlocksWithClientsImpl a template, based on an executor * Added a dummy implementation to fetch a new block. Distribute not yet implemented. * Added superoptimistic first implementation of ScatterExecutor return. Only works if there are no shadowRows. * Implemented logic to slice Blocks in Scatter according to the given call. Also added handfull of test for it, not complete though * Fixed a Scatter Executor handling of softLimit. Added another test to ansure diverse calls of clients to Scatter * Added random ordering test framework on ScatterExecutor * Added some tests for ScatterExecutor including subqueries * Removed todo, as it has been solved by now * Implemented Scatter by using IDExecutor. Tests not green yet * Implementation of ConstFetcher that allows to inject fullBlocks of AqlItemRows, and not only single rows. Also handles ShadowRows now in a correct way * Do not jump over relevant shadow rows in ConstFetcher. * Fixed tests. * Moved information of DistributeExecutor into the Infos. This resulted in adapting the Constructor API. I used this situation to do a big cleanup of ancient code. From this point there is no way back. * Implemented the last bits of DistributeExecutor * 'Implemented' execute by fallback to old API in RemoteExecutor * Fixed an EdgeCase with Scatter Executor, where the data was modified in place right after scatter. This has side-effects on peer executors. So we for now do a full copy of every block here. * Removed invalid assertions, also fixed RegisterInput to DistributeExecutor * Removed obsolete TODOs * Update arangod/Aql/IdExecutor.cpp Co-Authored-By: Lars Maier <lars@arangodb.com> Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Lars Maier <lars@arangodb.com> * Replaced the operator== and operator!= with `isSameBlockAndIndex`. (#11088) * New style IndexExecutor. (#11029) * New style IndexExecutor. * Small changes. * Fixing some logic for skipping. * Small changes. * TRI_ASSERT(FALSE); * Removed _fetcher member. (again) * Feature/aql subquery execution block impl execute implementation debug logging (#11112) * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * Feature/aql subquery operations stack hashed collect executor (#11103) * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * added inputRange skip and produce * added inputRange skip and produce * Added shouldSkip helper to AqlCall * Added implementation if execute to Constfetcher. Not specifically tested yet. * Moved IDExecutor to newStyle executor * Added some documentation to SingleRowFetcher * Adapted IdExecutor tests. * Added an assertion in ConstFetcher * Fixed initialize cursor in ExecutionBlockImpl * Added tests for Initilaize cursor in IdExecutor. There is a special implementation for this... * Fixed linker issues, because of variables in cpp files named identical and not beeing static * Started refactoring the Scatter/Gather Executor logic. Added an intermediate class that does not use a template and can be casted to. This state compiles but will not work. * Made BlocksWithClientsImpl a template, based on an executor * Added a dummy implementation to fetch a new block. Distribute not yet implemented. * Added superoptimistic first implementation of ScatterExecutor return. Only works if there are no shadowRows. * Implemented logic to slice Blocks in Scatter according to the given call. Also added handfull of test for it, not complete though * Fixed a Scatter Executor handling of softLimit. Added another test to ansure diverse calls of clients to Scatter * Added random ordering test framework on ScatterExecutor * Added some tests for ScatterExecutor including subqueries * Removed todo, as it has been solved by now * Implemented Scatter by using IDExecutor. Tests not green yet * Implementation of ConstFetcher that allows to inject fullBlocks of AqlItemRows, and not only single rows. Also handles ShadowRows now in a correct way * Do not jump over relevant shadow rows in ConstFetcher. * Fixed tests. * Moved information of DistributeExecutor into the Infos. This resulted in adapting the Constructor API. I used this situation to do a big cleanup of ancient code. From this point there is no way back. * Implemented the last bits of DistributeExecutor * 'Implemented' execute by fallback to old API in RemoteExecutor * Fixed an EdgeCase with Scatter Executor, where the data was modified in place right after scatter. This has side-effects on peer executors. So we for now do a full copy of every block here. * Removed invalid assertions, also fixed RegisterInput to DistributeExecutor * Removed obsolete TODOs * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * Made the branch compile again * Update arangod/Aql/IdExecutor.cpp Co-Authored-By: Lars Maier <lars@arangodb.com> * Intermediate commit. Empty might not compile * Added some external helper functions to compare ItemBlocks in different variants * Added a brief description for the bunch of input parameters in HashedCollect * Allow unordered output tests in TestHelper class. * Added first HashedCollectExecutor test using the helper * Allow to use never wait in getSome / skipSome variant of WaitingExecutionBlockMock. * Added more HashedCollect tests (multi group, count) and started with framework for AggregateMethods * Fixed simple Aggregate Test * Improved output for unordered Row in Output test. * Added test around collecting of Arrays and Objects * Also allow a fullCount call in the old Execute Wrapper * Almost Completed register and call behaviour test suite * Added a todo for a missing test * Improved setup of AqlTestCase framework, not only start the server once. This speeds up AQL test by at least a factor of 10. * implemented newStyle HashedCollect executor. * Cleanup of old code. * Added test cases where the upstream producer lies to HashedCollect. * Let HashedCollect and Impl life with the fact that upstream lies. * Improved gtest with multiple calls. * Simplify input consume logic of HashedCollect Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Lars Maier <lars@arangodb.com> * Feature/aql subquery operations stack enumerate list (#10988) * produce inputRange * produce inputRange test * produce inputRange test 2 * apply new api * enable new api * enumeratelistexecutor changes * added hackis shouldSkip method * add new execution style for enumerate lsit * added skip for enumerate list * edit of enumerate list executor * fixed enumlist executor * optimized skip loop * more tests, using new framework * allow empty output checks * added new style tests, removed old tests * removed old produceRows method * fixed tests * added one missing exit case * added another exit in SHADOWROWS mode, as we need to take a look at fullCount there * Revert "added another exit in SHADOWROWS mode, as we need to take a look at fullCount there" This reverts commit e74b7cede78f81b8fece034fc921a969e91dda12. * do not take care of fullCount in executor itself, impl will do * removed obsolete comment * rm _skipped because unused, return upstreamstate * added invalid value test, added border edge tests for softLimit, hardLimit and fullCount * api change * added test for inner states inside the executor * added new tests * fixed assert * removed two not needed ifs * asserted a case in our test helper * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * undo this commit before merge * undo this commit before merge * Merged Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation inject stack into fetcher (#11124) * Added api to pass through the stack to the dependencyProxy. THis is just temporary and doomed to be removed as soon as API is completed. * Let depdendency Proxy forward the Stack. This may not work in all cases. * Default implement Assignment constructor. * Removed Query jenkins debug output again * Fixed forwarding of stack in skipSome * Tweak ExecutorTestHelper to test pipelines (#11079) * Tweak ExecutorTestHelper to test pipelines * Modifications * Fixup tests * Address some comments * Fix. * AQL execute in LimitExecutor (#10886) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Fixed merge * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Reverted accidental change * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Assert no input rows in skip * Began writing a test framework for LimitExecutor * Removed unused code * Continued test framework * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip * Make AqlItemBlockInputRange hold a skipped count * Added == for LimitStats, moved code to .cpp file * Added Stats return value to skipRowsRange * WIP: Implement LimitExecutor test framework * Fixed namespace * Added missing advanceRow call * Added parametrized test cases * Dump * Made test output more readable * Fixed merge conflicts * Set engine in createFakeQuery() * [WIP] Dump: not working * Fixed merge conflicts * Switch from boost::variant to std::variant * Fix comparison of differently sized blocks * Fixed usage of temporary * Fixed skip handling in ExecutionBlockImpl * Add upstream fullCount in LimitExecutor * Fix assertion that was too narrow * Again, fixed skip handling in ExecutionBlockImpl * Extended ExecutorTestHelper, mainly a loop to handle passthrough Executors * Fixed expected skipped count * Fixed bugs in LimitExecutor * Another bugfix in LimitExecutor * Fixed expected output calculation * Removed some comments * Improve test speed by not recreating MockAqlServer for every case * Allow additional variants in ExecutorTestHelper * Tried to fix a bug in LimitExecutor, not quite working * A little cleanup * Toggle lieAboutHasMore * Reverted broken fix, fixed WaitingExecutionBlockMock * Fixed merge conflicts * Add LimitStats to expected checks in tests * Fixed another leftover merge conflict * Fixed fullCount stats * Removed old code from LimitExecutor * Updated ExecutionBlockImpl to allow removal of old executor methods * Removed old tests * Simplified test code * Added comments and constructors to AqlCall * Instantiate test cases with multiple different calls * Fixed a bug in run(), made looping optional * Fixed a bug in DependencyProxy * Fixed a bug and an assertion in LimitExecutor * Added assertions and a fix in WaitingExecutionBlockMock * Enabled stats tests again * Made the executor compile under Mac * Let AqlHelper compare profiled stats as well * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * Modified skip logic in LimitExecutor to simplify what has to be reported to downStream and what is handled by internal skip. Also allow to do skip and fullCount in one go, if limit is 0 * Adapted the Expected skipped output in Limit test * Forward fullCount from client through LIMIT. Simplyfied produceLogic. * Reduced the amount of Tests in LimitExecutor. Need to decide on a suitable subset * Removed lieAboutHasMore by simply adding an empty block to the end that is blocking return in the WaitingExecutor * Fixed SingleRowFetcher to allow a Range with only skip information. * Intermed version of ExecutionBlockImpl. Needs to be continued * Allow hardLimit to pass getSome check * Intermed commit * Unified FastForward and FullCount code path * Produce and Skip go to FastForward if the Executor reports done and the output row is full. This way we can handle more FUllCount cases straight away. * Fixed the expected value for FullCount stats in test. This is rather awkward to automatically compute... * Fixed lying of the ExecutionBlockMock. Did not lie correctly if the limit was reached by softLimit. * Make the rules for supporting the OldAPI more relaxed. * Added a missmatch of outputColumns and internal output columns in TestHelper * Fixed AqlItemBlockRangeTests * Implemented unreachable code, otherwise our Jenkins does not compile it ;( * Fixed SortedCollect, it might count the last group twice on fullCount, if produce ends exactly in this group * Use AqlTestCase in ReturnExecutorTest * Return Executor needs to report skipped from upstream properly * Fixed fullCount call behaviour * Fixed expected numberSkipped in LimitExecutor test * Added some DEBUG level log messages in FastForward * Fixed reporting of skip in skipSome case and removed dead code * Fixed Impl test, we cannot let the WaitingBlockMock do counting on ShadowRows, it is not implemented there. * Removed unused function * Pointeless comment * Fixed skipSome of Filter * Avoid endless Loop in indexExecutor fullCount, where it tried to skipped 0 documents forever * Fixed return state of IndexExecutor * Fake DONE on hardLimit for oldAPI, apply only on toplevel. Improve shortcutting behviour to jump into FastForward everytime HARDLimit is reached, even if output is full. * Added an assertion that multiple UPSTREAM call rounds do not mess up the callStack by leaving the _upstreamCall on top. Now the executor itself pops off the call, this is subject to change if we manage the Stack differently * Was erlaube skip? Adjsuted tests to honor that we can do skip and limit now in one go, for all executors that are updated already * Fixed ConstFetcher, to not modify the CallStack, it only has a reference, where the other Fetchers copy it to the Executors, and modify it there * Do not overfetch on passthrough if the block is full. * We cannot create DONE on oldStyle if we need fullcount... * Unfortunately we teporarily need more calls in fullCount case, as long as one of the Executors above is not yet on execute * Mhm gcc and clang do not aggree on which types are equal... * Fixed compile issue. Co-authored-by: Michael Hackstein <michael@arangodb.com> Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> * Feature/aql subquery operations stack calculation exec enumerate exec (#11098) * removed unused import * re-ordered asserts to see which register was wrongly used * added implementation of produceRows with inputRange * added first test to test new produceRows including inputRange * rm skip from calc executor * rm not correct assert, added second test * added produceRows inputrange hasmore test * added new produceRows + test * added skipRowsRange enumerate collection executor * added skipRowsRange enumerate collection executor test * added assert * adde fake query helper, one without auto trx but with prepare plan, the other one without prepare but with trx - also removed fakeit mocks from enumeratecollectionexecutor tests * fixed skipRowsRange in enumerateCollectionExecutor, added test as well * removed todo * added skipAll * added EnumerateCollection as new style executor * added produce and skip rows, not ready yet * added new tests, still not ready, need unsorted compare operators in helper, waiting * rm logs * intermidiate commit - may not work * removed unused import * added implementation of produceRows with inputRange * added first test to test new produceRows including inputRange * rm skip from calc executor * rm not correct assert, added second test * added produceRows inputrange hasmore test * added new produceRows + test * added skipRowsRange enumerate collection executor * added skipRowsRange enumerate collection executor test * added assert * adde fake query helper, one without auto trx but with prepare plan, the other one without prepare but with trx - also removed fakeit mocks from enumeratecollectionexecutor tests * fixed skipRowsRange in enumerateCollectionExecutor, added test as well * removed todo * added skipAll * added EnumerateCollection as new style executor * added produce and skip rows, not ready yet * added new tests, still not ready, need unsorted compare operators in helper, waiting * rm logs * intermidiate commit - may not work * make branch compile again * mv assert * protected server * make enum collection tests work again * fixed ExecBlockImpl * todo optimization * fix gtest calc executor * header file enumcollexec, helper method * fix shrink method in aqlitemblock * impl * nodiscard * calc executor * rm empty line * fix calc exec * tmp re-enable old method * tmp make tests green again, needs cleanup later * use uint64_t instead of u_int64_t * removed unused code * Update arangod/Aql/CalculationExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * Update arangod/Aql/CalculationExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * remove implementation code of old api * Update arangod/Aql/EnumerateCollectionExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * buildable tests again * rm unused code * added assert * make tests compile again ... * disable few more tests, need to be optimized after subquery end start merges Co-authored-by: Michael Hackstein <michael@arangodb.com> * Quickfix for compilation error due to merge * Feature/aql subquery execution block impl execute implementation k shortest path executor (#10940) * Add new produceRow API code for KShortestPathsExecutor * Add KShortestPathsExecutor test to build * Add KShortestPathsExecutor test * Test and bugfix for KShortestPathExecutor * Update k-shortest-paths executor for new interface * Update tests for k-shortest-paths executor * Make ShortestPathExecutor and KShortestPathExecutor more similar * Fix merge messup * Refactor KShortestPathsExecutorInfos * Move KShortestPathsExecutor to new executor interface * Fix tests * Resolve naming conflict * Address some review comments * Fix skip * Fix for new signatures * Fix KShortestPathsExecutor * Fix KShortestPaths regression Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation traversal executor (#11023) * Move to new executor interface * Fix tests and some bugs * new skip signature * new skip signature * Fix Traversal Executor Skipping * Fix compiler complaint Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move SubqueryStartExecutor to new interface (#11025) * Move SubqueryStartExecutor to new interface * Diddle * Tests pass for SubqueryStartExecutor * Delete most of the SubqueryEndExecutor tests They are irrelevant now as they only tested shadowrow behaviour. Shadow Rows are now handled by ExecutionBlockImpl. * Skeleton tests * Junk * Cleanup and make work * Tweak ExecutorTestHelper to test pipelines * SubqueryStart/SubqueryEnd * Blafummel. * Modifications * Fixup tests * Address some comments * Fix. * Fix tests * Various fixes. * Made the branch compile on Mac * Intermediate commit, does not compile, required for merge * Fixed one of the SplicedSubqueryTests * Seperated ShadowRow handling for SubqueryStart/End executors to get a better overview and understanding on how it should behave. * Allow to use a fullCallStack in ExecutorTestHelper. * Added asserts on SubqueryEnd that is not allowed to be called with Skipped from upstream * Use AqlTestCase in SubqueryStartExecutorTest. Transformed first test to TestHelper pipeline * Let ExecutorTestHelper also test for shadowRows * Adapted SingleInput=>Data+Shadow test to testing Helepr and make it pass * Transformed existing SubqueryStart tests to Helper, fixed a test where there was no place for the ShadowRow to be written * Fixed Skip in SubqueryStartExecutor * Allow the Pipeline to add Consumer and dependency without fiddling with the Deque from outside * Added Skip adn ShadowRowForwarding tests to SubqueryStartExecutor * Fixed fullCount of SubqueryStart to be called properly. * Renamed isModificationSubquery -> isModificatioNode again * Consume more than one subquery en-block * Fixed debugReporting in SplicedSubqueryTest. Disabled the skip in subquery tests for now * Removed AQL debug log level from testing config * First version of Subquery Skip bypassing. Including 1 passing test. * Added disabled tests for skip, and proposed to do it later * Fixed reporting of FILTERED in fullCount of EnumerateCollectionExecutor * Enable calculationExecutor execute interface * Fixed compile issues * To Velocypack the modification Subquery value * Fixed illegal assert and reactivated failure tests * Fixed compile issue * Deactivated Calculation again * Added Compatibility Mode to AqlCallStack * Removed IdExecutor<void> the only feature it has can be moved into IDexecutor in general. This still has to be done * When falling back in BlocksWithClients use compatibility stack * ConstFetcher now correctly reports if it skipped all input * Fixed a bug in ExecutionBlockImpl, which would go into FetchMore state, even if the Executor still has data and is happy to produce more, but the output is full. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute remote node api flag (#11159) * Clarified comment * Added api property to RemoteNode * Added missing include * Feature/aql subquery operations stack i research view executor (#11140) * added arangosearch view executor to new style * added new produceRows and skipRowsRange * ires * test jenkins * modify aql call * cleanup branch * Update arangod/Aql/IResearchViewExecutor.cpp Co-Authored-By: Dronplane <andrei.lobov@arangodb.com> * applied suggested changes * also enable IResearchViewMergeExecutor * Fixed an issue in skipRange, where the input was not moved forward in all cases. * Removed unused local variables Co-authored-by: Dronplane <andrei.lobov@arangodb.com> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move SingleRemoteModificationExecutor to new interface (#11166) * Move SingleRemoteModificationExecutor to new interface * Activate SingleRemoteModificationExecutor * Inspect the result of remote modification Co-authored-by: Michael Hackstein <michael@arangodb.com> * New style DistinctCollect (#11096) * First implementation of new style executor for DistinctCollect. * Tests for distinct collect. * Remove debug output. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation noresults (#11180) * Use ExecutorHelper in tests * Implemented no Results fetcher * Moved Constrained sort to execute API (#11173) * Moved Constrained sort to execute API * The LIMIT in ConstrainedSort can actually fetch TOO much, if there is too small input from upstream * Improved profiler tests, we need less calls now \o/ * Fixed unlimited call to limit executor * Feature/aql subquery operations stack aql item block input matrix (#11167) * added new file to cmakelist * added new method skipAllRemainingDataRows * added first implementation of execute - AllRowsFetcher * impl is now using skipAllRemainingDataRows method, also execute methods now using Fetcher::DataRange instead of inputRange * impl is now using skipAllRemainingDataRows method, also execute methods now using Fetcher::DataRange instead of inputRange - header file * AllRowsFetcher now using AqlItemBlockInputMatrix as DataRange, first implementation of execute * first implementation of AqlItemBlockInputMatrix * more cleanup * optimized execut elogic * optimized logic in inputmatrix, still a few todos left * added SortExecutor as newStyle * added produceRows and skipRows * logs (to be removed), skipAllRemainingRows * logs * inputMatrix * changed return type of skipAllRemainingDataRows * impl logs * logs, added clear method * sort test * typo, log * latest changes * cleanup, assert * rm logs * added more todos * todo * new method names, compile again * Reset ExecutorReturnedDone for SubqueryStart and SubqueryEnd. * InputMatrix needs to return state DONE on every input request. It guarantees that we have all input * Removed invalid Assertion, and removed early return if output is full. * Removed DebugLogging from CalculationTest * Fixed skipping in SortExecutor * Fixed upstreamState return of AqlItemMatrix. It is only finalState iff there is no data * Let the InputMatrix drop all rows after FullCount * Allow bypassing with hardLimit 0 in InputMatrix * Disabled the old-code assertion. This way the SortExecutor tests we have will be red, but not crash, allowing the jenkins to produce meaningfull results * Fix assert in WAITING case * Add rudimentary sort executor GTests * Fixed assertion on InputMatrix that starts right away with a shadowRow * Removed declared but not implemented function * changed from: peekDataRowAndState -> to: peekDataRow Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute in restaqlhandler (#11175) * Updated comment * Added two static strings * Avoid unintentional streaming operator of string_view as a container * Added parser for execute rest call * Fixed ambiguous overload in ResultT * Added execute and executeForClient to ExecutionEngine * Implemented executeForClient in RestAqlHandler * Add default value * Updated comment * Changed error attribute * Fixed duplicate log id * Try to make MSVC work * Moved AqlExecuteResult to a separate file * Tried to fix MSVC compile error * Revert "Tried to fix MSVC compile error" This reverts commit f6f43b2c7e4bfe17dafe65f48bcd5c0b6e5c69a9. * Next try to fix MSVC * Revert "Next try to fix MSVC" This reverts commit 528c4c795d10ee0aa6686d143be1f6faa6e9b553. * Revert "Try to make MSVC work" This reverts commit ba7d9c072fccb969da1cdf045eadf49c297da8e9. * Work around MSVC shortcoming * Support executors with multiple dependencies (#11181) * Use DataRange in some places * Add MultiAqlItemBlockInputRange * First implementation of life, universe, and everything * implement skip for UnsortedGather * Some more wiring * Attempt at hacking up execute for DependencyProxy * Compilation fixes! * Fixed AqlCall compilation and unified execute and executeForDependency * Fixed WAITIGN assertion * Sonderlocken. * Foo * Make compile * Make tests pass. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move ModificationExecutors to new interface (#11165) * Move ModificationExecutors to new interface * added inputRange method to inputMatrix class, added switch between DataRange Input <-> Matrix for ModificationExecutors * forgot to enable allrows modification executor in fastForwardType * removed obsolete comment * removed obsolete comment Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery operations stack materialize executor (#11192) * removed obsolete comment, added produceRows and skipRowsRange, enabled MaterializeExecutor as newStyleExecutor * forgot to use advanceRow, fetch input during skipRowsRange * removed old implementations * removed comment, fixed a logical loop error * skip could be simplified as well * temp workaround Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute remote (#11197) * Clarified comment * Updated comment * Added two static strings * Avoid unintentional streaming operator of string_view as a container * Added parser for execute rest call * Fixed ambiguous overload in ResultT * Added execute and executeForClient to ExecutionEngine * Implemented executeForClient in RestAqlHandler * Add default value * Updated comment * Changed error attribute * Added API toggle to RemoteExecutor * Fixed compile error * Fixed duplicate log id * Try to make MSVC work * Moved AqlExecuteResult to a separate file * Moved AqlExecuteResult to a separate file * Implemtented execute in Remote; except for (de)serialization, which is still missing * Tried to fix MSVC compile error * Revert "Tried to fix MSVC compile error" This reverts commit f6f43b2c7e4bfe17dafe65f48bcd5c0b6e5c69a9. * Next try to fix MSVC * Implemented (de)serialization for RemoteExecutor::execute * Fixes * Added minimal AqlCall (de)serialization tests * Bugfix * Added minimal (de)serialization tests for AqlCallStack and AqlExecuteResult * Fixed test * Changed Serialization format a little * Bugfix * Initialize SingletonBlock immediately with an empty row * Revert "Next try to fix MSVC" This reverts commit 528c4c795d10ee0aa6686d143be1f6faa6e9b553. * Revert "Try to make MSVC work" This reverts commit ba7d9c072fccb969da1cdf045eadf49c297da8e9. * Work around MSVC shortcoming * Work around MSVC shortcoming * Attempt to fix windows compile issue Co-authored-by: Michael Hackstein <michael@arangodb.com> * Added test for distinct collect with random order. (#11184) * Feature/aql subquery execution block impl execute implementation subquery executor (#11196) * Intermediate commit, does not compile. * New style for SubqueryExecutor. * Removed bad assert. * Reset subsqueryInitialised in skip. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation count collect (#11172) * Move SubqueryStartExecutor to new interface * Diddle * Tests pass for SubqueryStartExecutor * Delete most of the SubqueryEndExecutor tests They are irrelevant now as they only tested shadowrow behaviour. Shadow Rows are now handled by ExecutionBlockImpl. * Skeleton tests * Junk * Cleanup and make work * Tweak ExecutorTestHelper to test pipelines * SubqueryStart/SubqueryEnd * Blafummel. * Modifications * Fixup tests * Address some comments * Fix. * Fix tests * Various fixes. * Made the branch compile on Mac * Intermediate commit, does not compile, required for merge * Fixed one of the SplicedSubqueryTests * Seperated ShadowRow handling for SubqueryStart/End executors to get a better overview and understanding on how it should behave. * Allow to use a fullCallStack in ExecutorTestHelper. * Added asserts on SubqueryEnd that is not allowed to be called with Skipped from upstream * Use AqlTestCase in SubqueryStartExecutorTest. Transformed first test to TestHelper pipeline * Let ExecutorTestHelper also test for shadowRows * Adapted SingleInput=>Data+Shadow test to testing Helepr and make it pass * Transformed existing SubqueryStart tests to Helper, fixed a test where there was no place for the ShadowRow to be written * Fixed Skip in SubqueryStartExecutor * Allow the Pipeline to add Consumer and dependency without fiddling with the Deque from outside * Added Skip adn ShadowRowForwarding tests to SubqueryStartExecutor * Fixed fullCount of SubqueryStart to be called properly. * Renamed isModificationSubquery -> isModificatioNode again * Consume more than one subquery en-block * Fixed debugReporting in SplicedSubqueryTest. Disabled the skip in subquery tests for now * Removed AQL debug log level from testing config * First version of Subquery Skip bypassing. Including 1 passing test. * Added disabled tests for skip, and proposed to do it later * Fixed reporting of FILTERED in fullCount of EnumerateCollectionExecutor * Enable calculationExecutor execute interface * Fixed compile issues * To Velocypack the modification Subquery value * Fixed illegal assert and reactivated failure tests * Fixed compile issue * Deactivated Calculation again * Added a testSuite for countCollectExecutor * Fixed Subquery Test for CountCollect * Added CountCollectImplementation * Added Compatibility Mode to AqlCallStack * Removed IdExecutor<void> the only feature it has can be moved into IDexecutor in general. This still has to be done * When falling back in BlocksWithClients use compatibility stack * Fixed EnumerateCollectionExecutor * Fixed fullCountBehaviour of EnumerateList. Also removed a unlimited,fullCount combination in IndexExecutor * Fixed endless loop while skipAl in OperationCursor * Fixed View Executor, which sends a no-limit no-offset upstream. * Fixed forwarding in NoResultsExecutor Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> * Activate CalculationExecutors (#11177) * Fixed compile bug-due to hidden merge conflict * Feature/aql subquery execution block impl execute implementation base bugfixes (#11201) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Fix memory leak in traversal (#11210) * AqlSubqueryExecutionBlockImplExecuteImplementation use moveValueInto instead of cloneValueInto (#11213) * Fix memory leak in traversal * Use move instead of clone for AqlValues * Feature/aql subquery execution block impl execute implementation gather executors (#11202) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Feature/aql subquery execute parallel gather (#11215) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute parallel gather 2 (#11221) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather * [WIP] Changed Api for MultiDepExecutors in ExecBlockImpl (not yet in the actual executors) * Moved AqlCallSet into a separate file * Changed SortingGather to use the new API * Changed ParallelUnsortedGather to use the new API * Changed UnsortedGather to use the new API * Moved AqlCall operator<< into .cpp file * Implement operator<< for AqlCallSet * Fix boolean mix-up * Fixed state machine: go to UPSTREAM when the AqlCallSet is not empty * Fixed assertion * Bugfix * SortingGather bugfixes * Added init() method to fix an assertion and cleanup * Removed unused variable * Fixed constrained sort * Fixed constrained sort #2 * Fix boolean mix-up * Remove old interface * Use call parameter for upstream request in produceRows * Remove more old interface code * Add skip methods to MultiAqlItemBlockInputRange * Skip in UnsortedGather * skip for UnsortedGather * Fix skip and upstream calls for UnsortedGather * skipRowsRange change * Remove useless comments * Moved multi-dep related code from ExeBlockImpl to MultiFetcher * Cleanup in SortingGather, implemented parallel fullCount * Try to fix a windows compile error * Simplify and extend skipRowsRange for UnsortedGatherExecutor * Made ParallelUnsortedGather actually parallel * Removed erroneous assertion * Undid erroneous change * Fixed MacOs compile. Also disabled tests for non-relevant AqlCallStacks. They will be removed * Fixed initialize Cursor for multi dependency blocks * Fixed fullCount case in parallel unsorted gather * Fixed fullCount upstream call of ParallelUnsortedGatherExecutor * Fixed fullCount in SortingGather * Windows \o/ if you cannot work properly with constexpr and static asserts, we do not let you do it! * Do not advance in Unsorted gather if there are still rows to skip * Add more comparison operators for AqlCall limits * Send clientCall limits to upstream in SortingGather * Improved fullCount in SortingGatherExectur * Disabled a cluster profile test. We now ask the RemoteNode more often if it already has data. It is a bit unclear to me if this is now better performance wise (<< i think so) or triggers undesired side effects * Helpless attempt to work around issues in stonage Visual Studio Compiler we are using. * Clearly adding an operator on a well defined type causes ambigousness on random basic types using the operator Co-authored-by: Michael Hackstein <michael@arangodb.com> Co-authored-by: Markus Pfeiffer <markus@arangodb.com> * Feature/aql subquery execution block impl execute implementation bypass skip (#11203) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Added an implementation for a SkipResult, in order to simplify exchange of it. * Moved production API -> SkipResult * Made tests compile with new SkipResult * Added a test using skip and limit on subqueries * Prepared to use subqueries in SkipResult * Let subqueries modify the SkipResult subquery stack * Fixed return state of GatherNode * Activate all spliced subquery tests \o/ * Let SubqueryEnd honor the client request * Added a Maintainer only test for the stack, if it is 36 compatible * Added first part of side-effect executors. They now send upstream a fetch-all stack. Need to fix downstream reporting, and call the executor with hardLimit 0 * Add a fake FASTFORWARD call into a subquery-skipped ModificationExecutor. * Added helper shadow row function for SideEffect executors * Let the Modification Executor also produce data, even if no FullCount is used. * Revert "Let the Modification Executor also produce data, even if no FullCount is used." This reverts commit b1c6af5674de445f9e3d6343feb0e3cad422588c. * Revert "Revert "Let the Mo…
mchacki
added a commit
that referenced
this issue
Mar 20, 2020
…cted number of rows (#11274) * Added first draft of AqlCallObject * Added AqlItemBlockInputRange * Added Header file for AqlCallStack with comments. No implementation yet. * Added AqlCallStack first draft implementation. Made everything compile * Added empty dummy stubs for execute() in ExecutionBlock * Implementation of new API in FilterExecutor. Rough implementation in ExecutionBlockImpl (not complete, only skip path) everything compiles, but is not tested * Fixed off by one error in InputRange. Fixed FilterExecutor * Added some more tests if executor cannot fulfill atMost * Added heplper functions to the AqlCall struct to avoid mangeling around with variants everywhere. * Add a member DataRange to ExecutionBlockImpl. Also now all Fetcher expose a DataRange, this might not be the correct one every where * Added preliminary minimal implementation of execute to DependencyProxy and SingleRowFetcher. Both untested and yet incomplete, they will only work for MVP. * added first implementation of count collect datarange produceRows function + test * Revert "added first implementation of count collect datarange produceRows function + test" This reverts commit 236825280bb5fd03dfe01febfc4dcf1a2d2e2c2f. * First draft of execute implementation, not yet handling ShadowRows though. This state is broken in the sense that a server does not boot with it and catch tests are broken * First working draft of execute() call in ExecutionBlockImpl. Tests are locally green, however there is at least one query still red (on ShadowRows) * Removed non finished implementation from this Branch. It moved to seperate branch * working - not done yet * Revert "working - not done yet" This reverts commit aa275b48ffc2c8fc9a3c364e2241321ec9c9293f. * first implementation of a ShadowRow fetching interface on AqlItemBlockInputRange * single row fetcher, execute + tests * Fixed SingleRowFetcherTest. Also an AqlItemBlockInputRange can now handle ShadowRows properly * Another fix on handling of shadow rows in InputRanges * Improved human readable output of SortLimit test, while validating the result * Removed debug profile execution of a query * Use simpleVPack printing for Trace of Execute, better to read by humans * Added tests for AqlItemBlockInputRange and adjusted implementation accordingly. * Fixed issue where ExecutionBlockImpl returns DONE, although there are still rows in the DataRange * Final modification of ShadowRows in FilterExecutor. All but profiling tests pass. This needs some discussion about overfetching of data, and propagation of hardLimit * Removed debug log includes * Moved responsibility for the client call into the output row. This can precisely count the elements it has written, and forms a single source of truth. The output row now works as the requested block by client that needs to be filled. Now the FilterExecutorTests are red. * Fixed FilterExecutorTest * Expose required features to AqlCall controlled by the output aql item row. Adapted executionBlockImpl to report there * Fixed bug with AqlCallStack pop operation that actually caused invalid memory access * outputRow now honors limits whan called for isFull. * Fixed isFull() in OutputAqlItemRow * Use the inbound call in skipRows for executors, in order to do a more precise upstream call * Make linker happy with operator< on CallLimit * Implement operator+ on AqlCall::Limit, also use std::visit for readability. * FilterExecutor getSome now propagates offset + min(limits) to upstream. This is pessimistic and avoids overfetching in all cases. NOTE: FilterExecutor should be moved to skipvariant using getSome * Fixed profiler tests for filter. These cannot be guaranteed anymore and it was an undesired, yet uncritical side effect that the tests have been green before * Fixed JSLint * Remove special casing for FilterExecutor * use if constexpr to distinguish between ported executors and non-ported executors enabling step-by-step porting and testing * Add skeleton skip code * Replace C&P accident * Simplify and implement skip for new executor interface * Fixup skipping for filter executor * Rename some methods for clarity and consistency * AqlItemBlockInputRange - hasMore is now hasDataRow - next is now nextDataRow - peek is now peekDataRow - state is now upstreamState - additional method upstreamHasMore * Introduce function to allocate an output block * Fix skipSome simulation bug * Some small cleanups * Replaced assertion on atMost on the output size. Otherwise we got into issues with atMost changes between waiting calls (can occur in our intermediate state now) * Attempt at fixing execute * Fix merge conflict * Feature/aql subquery execution block impl execute implementation exec block impl tests (#10838) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation shortest path executor (#10780) * Hook up new ShortestPathExecutor * Small fixes * Removed now unused code. * Moved ShortestPathExecutor tests to use Execute only. * Added a test for shortestPathExecutor where the output is smaller than the amount of entries within the path. * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Fixed Windows compile issue * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Windows compile fix * Attempt at fixing ShortestPathExecutorTest for Windows * Windows compile fix * Added a skip test and fixed the producing executor to recreate the list once for every input line. * Frankenstein my tests into the branch * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Hack * Removed designated initializers. Thanks for not supporting it MSVC! * Update ShortestPathExecutorTest * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Umbau 2.0 * Reshape output and skip a bit * Some test fixes * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Square 1 * Input validation * ShortestPathExecutor with fullCount and tests Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation exec block impl tests shadow rows (#10970) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Added tests for ExecutionBlockImpl, when there is a WAITING passed through. There is no test yet for in between waitings. * Allow to configure WAITING variant in WAITING block mock. * Use the WaitingBlockMock to be independent of Singleton in test * Added WAITING + Passthrough tests * Added a test to path through shadowRows without data. * Waiting ExecutionBlock Mock can now return/skip partial blocks. However it does NOT CARE for ShadowRows * Added tests and fixed an issue in ShadowRow forwarding * Added tests for shadowRow forwarding * Added the intenral state of Execute as member of ExecutionBlockImpl. We need to retain parts of this state accross returns * Fixed all Waiting and multiple call tests * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Fixed invalid memory access in test code. Do never return a lambda that references a stack variable... * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip * Allways -> Always. * Added tests for pass-by of non-relevant Subqueries. Fixed a bug found on the way in WaitingBlockMock. Now some other tests fail \o/ * Fixed failed tests. Now tests correspond to correct behaviour of WaitingExecutionBlockMock * Intermediate version, doe snot work * Fixed the empty subquery tests Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation sorted collect (#11020) * SortedCollect executor and some tests. * More tests. * More const and skip implemented. * Fixed some bugs found by tests. * Refactored skip. * Broken state with test framework. * Extracted testing framework into separate file and adjusted it to use WaitingBlockMock. However it is not yet fully implemented * Added another SortedExecutor test without fullcount. This passes. Also assert on skip and State * Removed obsolete assertion in skip, added another nonFullcount test. * More tests. Fixed some bugs in the test framework. * More cleanup. * Remove version control marker. * Removed wrong assertion. * Apply suggestions from code review Co-Authored-By: Michael Hackstein <michael@arangodb.com> * Remove dead code. * Removed fetcher. Co-authored-by: Michael Hackstein <michael@arangodb.com> * ReturnExecutor New style (#10831) * ReturnExecutor implemented. Tests misssing. * Apply suggestions from code review Co-Authored-By: Tobias Gödderz <tobias@arangodb.com> * Implemented skipRows in Return. Also fixed ExecutionBlock references in static asserts * Implemented first new style test for Return. * Moved the traceExecute calls into baseclass. * Added tracing around execute in WaitingExecutionBlockMock. Fixed a test bug, if the AqlCall covers all of the first block, but only a part of the Second block. You would only get the part of the second block. * Added more tests for standard behaviour of ReturnExecutor * Added an assertion. * Simplified call to ReturnExecutor * Prepare tests for doCount. Also added test for different input register. * Added accessor for _stats in Engine. * Included count tests in ReturnExecutorTest. * Removed old obsolete test. * Removed obsolete code-path * Fixed unrelated bug in FilterExecutor. In some cases it did lie on skipRows that it has more, even if it does not have and the upstream does not have as well. Thank you jenkins for hitting into this race :+1: Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Tweak ExecutorTestHelper to test pipelines * Revert "Tweak ExecutorTestHelper to test pipelines" This reverts commit 6359f0c926cad30b4efd7ce263fb3b64563621b8. * Added Stats return value to skipRowsRange (#11081) * Added Stats return value to skipRowsRange * Fixed catch test. Somehow i merged the incorrect part Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Additional Assertion in ExecutionBlockImpl (#11077) * Added a assertion arround skipRowsInRange that ensures that the Call and the skippedLocal do not get out-of-sync * Improved the assertion * Feature/aql subquery operations stack id executor (#10986) * added inputRange skip and produce * added inputRange skip and produce * Added shouldSkip helper to AqlCall * Added implementation if execute to Constfetcher. Not specifically tested yet. * Moved IDExecutor to newStyle executor * Added some documentation to SingleRowFetcher * Adapted IdExecutor tests. * Added an assertion in ConstFetcher * Fixed initialize cursor in ExecutionBlockImpl * Added tests for Initilaize cursor in IdExecutor. There is a special implementation for this... * Fixed linker issues, because of variables in cpp files named identical and not beeing static * Started refactoring the Scatter/Gather Executor logic. Added an intermediate class that does not use a template and can be casted to. This state compiles but will not work. * Made BlocksWithClientsImpl a template, based on an executor * Added a dummy implementation to fetch a new block. Distribute not yet implemented. * Added superoptimistic first implementation of ScatterExecutor return. Only works if there are no shadowRows. * Implemented logic to slice Blocks in Scatter according to the given call. Also added handfull of test for it, not complete though * Fixed a Scatter Executor handling of softLimit. Added another test to ansure diverse calls of clients to Scatter * Added random ordering test framework on ScatterExecutor * Added some tests for ScatterExecutor including subqueries * Removed todo, as it has been solved by now * Implemented Scatter by using IDExecutor. Tests not green yet * Implementation of ConstFetcher that allows to inject fullBlocks of AqlItemRows, and not only single rows. Also handles ShadowRows now in a correct way * Do not jump over relevant shadow rows in ConstFetcher. * Fixed tests. * Moved information of DistributeExecutor into the Infos. This resulted in adapting the Constructor API. I used this situation to do a big cleanup of ancient code. From this point there is no way back. * Implemented the last bits of DistributeExecutor * 'Implemented' execute by fallback to old API in RemoteExecutor * Fixed an EdgeCase with Scatter Executor, where the data was modified in place right after scatter. This has side-effects on peer executors. So we for now do a full copy of every block here. * Removed invalid assertions, also fixed RegisterInput to DistributeExecutor * Removed obsolete TODOs * Update arangod/Aql/IdExecutor.cpp Co-Authored-By: Lars Maier <lars@arangodb.com> Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Lars Maier <lars@arangodb.com> * Replaced the operator== and operator!= with `isSameBlockAndIndex`. (#11088) * New style IndexExecutor. (#11029) * New style IndexExecutor. * Small changes. * Fixing some logic for skipping. * Small changes. * TRI_ASSERT(FALSE); * Removed _fetcher member. (again) * Feature/aql subquery execution block impl execute implementation debug logging (#11112) * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * Feature/aql subquery operations stack hashed collect executor (#11103) * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * added inputRange skip and produce * added inputRange skip and produce * Added shouldSkip helper to AqlCall * Added implementation if execute to Constfetcher. Not specifically tested yet. * Moved IDExecutor to newStyle executor * Added some documentation to SingleRowFetcher * Adapted IdExecutor tests. * Added an assertion in ConstFetcher * Fixed initialize cursor in ExecutionBlockImpl * Added tests for Initilaize cursor in IdExecutor. There is a special implementation for this... * Fixed linker issues, because of variables in cpp files named identical and not beeing static * Started refactoring the Scatter/Gather Executor logic. Added an intermediate class that does not use a template and can be casted to. This state compiles but will not work. * Made BlocksWithClientsImpl a template, based on an executor * Added a dummy implementation to fetch a new block. Distribute not yet implemented. * Added superoptimistic first implementation of ScatterExecutor return. Only works if there are no shadowRows. * Implemented logic to slice Blocks in Scatter according to the given call. Also added handfull of test for it, not complete though * Fixed a Scatter Executor handling of softLimit. Added another test to ansure diverse calls of clients to Scatter * Added random ordering test framework on ScatterExecutor * Added some tests for ScatterExecutor including subqueries * Removed todo, as it has been solved by now * Implemented Scatter by using IDExecutor. Tests not green yet * Implementation of ConstFetcher that allows to inject fullBlocks of AqlItemRows, and not only single rows. Also handles ShadowRows now in a correct way * Do not jump over relevant shadow rows in ConstFetcher. * Fixed tests. * Moved information of DistributeExecutor into the Infos. This resulted in adapting the Constructor API. I used this situation to do a big cleanup of ancient code. From this point there is no way back. * Implemented the last bits of DistributeExecutor * 'Implemented' execute by fallback to old API in RemoteExecutor * Fixed an EdgeCase with Scatter Executor, where the data was modified in place right after scatter. This has side-effects on peer executors. So we for now do a full copy of every block here. * Removed invalid assertions, also fixed RegisterInput to DistributeExecutor * Removed obsolete TODOs * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * Made the branch compile again * Update arangod/Aql/IdExecutor.cpp Co-Authored-By: Lars Maier <lars@arangodb.com> * Intermediate commit. Empty might not compile * Added some external helper functions to compare ItemBlocks in different variants * Added a brief description for the bunch of input parameters in HashedCollect * Allow unordered output tests in TestHelper class. * Added first HashedCollectExecutor test using the helper * Allow to use never wait in getSome / skipSome variant of WaitingExecutionBlockMock. * Added more HashedCollect tests (multi group, count) and started with framework for AggregateMethods * Fixed simple Aggregate Test * Improved output for unordered Row in Output test. * Added test around collecting of Arrays and Objects * Also allow a fullCount call in the old Execute Wrapper * Almost Completed register and call behaviour test suite * Added a todo for a missing test * Improved setup of AqlTestCase framework, not only start the server once. This speeds up AQL test by at least a factor of 10. * implemented newStyle HashedCollect executor. * Cleanup of old code. * Added test cases where the upstream producer lies to HashedCollect. * Let HashedCollect and Impl life with the fact that upstream lies. * Improved gtest with multiple calls. * Simplify input consume logic of HashedCollect Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Lars Maier <lars@arangodb.com> * Feature/aql subquery operations stack enumerate list (#10988) * produce inputRange * produce inputRange test * produce inputRange test 2 * apply new api * enable new api * enumeratelistexecutor changes * added hackis shouldSkip method * add new execution style for enumerate lsit * added skip for enumerate list * edit of enumerate list executor * fixed enumlist executor * optimized skip loop * more tests, using new framework * allow empty output checks * added new style tests, removed old tests * removed old produceRows method * fixed tests * added one missing exit case * added another exit in SHADOWROWS mode, as we need to take a look at fullCount there * Revert "added another exit in SHADOWROWS mode, as we need to take a look at fullCount there" This reverts commit e74b7cede78f81b8fece034fc921a969e91dda12. * do not take care of fullCount in executor itself, impl will do * removed obsolete comment * rm _skipped because unused, return upstreamstate * added invalid value test, added border edge tests for softLimit, hardLimit and fullCount * api change * added test for inner states inside the executor * added new tests * fixed assert * removed two not needed ifs * asserted a case in our test helper * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * undo this commit before merge * undo this commit before merge * Merged Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation inject stack into fetcher (#11124) * Added api to pass through the stack to the dependencyProxy. THis is just temporary and doomed to be removed as soon as API is completed. * Let depdendency Proxy forward the Stack. This may not work in all cases. * Default implement Assignment constructor. * Removed Query jenkins debug output again * Fixed forwarding of stack in skipSome * Tweak ExecutorTestHelper to test pipelines (#11079) * Tweak ExecutorTestHelper to test pipelines * Modifications * Fixup tests * Address some comments * Fix. * AQL execute in LimitExecutor (#10886) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Fixed merge * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Reverted accidental change * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Assert no input rows in skip * Began writing a test framework for LimitExecutor * Removed unused code * Continued test framework * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip * Make AqlItemBlockInputRange hold a skipped count * Added == for LimitStats, moved code to .cpp file * Added Stats return value to skipRowsRange * WIP: Implement LimitExecutor test framework * Fixed namespace * Added missing advanceRow call * Added parametrized test cases * Dump * Made test output more readable * Fixed merge conflicts * Set engine in createFakeQuery() * [WIP] Dump: not working * Fixed merge conflicts * Switch from boost::variant to std::variant * Fix comparison of differently sized blocks * Fixed usage of temporary * Fixed skip handling in ExecutionBlockImpl * Add upstream fullCount in LimitExecutor * Fix assertion that was too narrow * Again, fixed skip handling in ExecutionBlockImpl * Extended ExecutorTestHelper, mainly a loop to handle passthrough Executors * Fixed expected skipped count * Fixed bugs in LimitExecutor * Another bugfix in LimitExecutor * Fixed expected output calculation * Removed some comments * Improve test speed by not recreating MockAqlServer for every case * Allow additional variants in ExecutorTestHelper * Tried to fix a bug in LimitExecutor, not quite working * A little cleanup * Toggle lieAboutHasMore * Reverted broken fix, fixed WaitingExecutionBlockMock * Fixed merge conflicts * Add LimitStats to expected checks in tests * Fixed another leftover merge conflict * Fixed fullCount stats * Removed old code from LimitExecutor * Updated ExecutionBlockImpl to allow removal of old executor methods * Removed old tests * Simplified test code * Added comments and constructors to AqlCall * Instantiate test cases with multiple different calls * Fixed a bug in run(), made looping optional * Fixed a bug in DependencyProxy * Fixed a bug and an assertion in LimitExecutor * Added assertions and a fix in WaitingExecutionBlockMock * Enabled stats tests again * Made the executor compile under Mac * Let AqlHelper compare profiled stats as well * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * Modified skip logic in LimitExecutor to simplify what has to be reported to downStream and what is handled by internal skip. Also allow to do skip and fullCount in one go, if limit is 0 * Adapted the Expected skipped output in Limit test * Forward fullCount from client through LIMIT. Simplyfied produceLogic. * Reduced the amount of Tests in LimitExecutor. Need to decide on a suitable subset * Removed lieAboutHasMore by simply adding an empty block to the end that is blocking return in the WaitingExecutor * Fixed SingleRowFetcher to allow a Range with only skip information. * Intermed version of ExecutionBlockImpl. Needs to be continued * Allow hardLimit to pass getSome check * Intermed commit * Unified FastForward and FullCount code path * Produce and Skip go to FastForward if the Executor reports done and the output row is full. This way we can handle more FUllCount cases straight away. * Fixed the expected value for FullCount stats in test. This is rather awkward to automatically compute... * Fixed lying of the ExecutionBlockMock. Did not lie correctly if the limit was reached by softLimit. * Make the rules for supporting the OldAPI more relaxed. * Added a missmatch of outputColumns and internal output columns in TestHelper * Fixed AqlItemBlockRangeTests * Implemented unreachable code, otherwise our Jenkins does not compile it ;( * Fixed SortedCollect, it might count the last group twice on fullCount, if produce ends exactly in this group * Use AqlTestCase in ReturnExecutorTest * Return Executor needs to report skipped from upstream properly * Fixed fullCount call behaviour * Fixed expected numberSkipped in LimitExecutor test * Added some DEBUG level log messages in FastForward * Fixed reporting of skip in skipSome case and removed dead code * Fixed Impl test, we cannot let the WaitingBlockMock do counting on ShadowRows, it is not implemented there. * Removed unused function * Pointeless comment * Fixed skipSome of Filter * Avoid endless Loop in indexExecutor fullCount, where it tried to skipped 0 documents forever * Fixed return state of IndexExecutor * Fake DONE on hardLimit for oldAPI, apply only on toplevel. Improve shortcutting behviour to jump into FastForward everytime HARDLimit is reached, even if output is full. * Added an assertion that multiple UPSTREAM call rounds do not mess up the callStack by leaving the _upstreamCall on top. Now the executor itself pops off the call, this is subject to change if we manage the Stack differently * Was erlaube skip? Adjsuted tests to honor that we can do skip and limit now in one go, for all executors that are updated already * Fixed ConstFetcher, to not modify the CallStack, it only has a reference, where the other Fetchers copy it to the Executors, and modify it there * Do not overfetch on passthrough if the block is full. * We cannot create DONE on oldStyle if we need fullcount... * Unfortunately we teporarily need more calls in fullCount case, as long as one of the Executors above is not yet on execute * Mhm gcc and clang do not aggree on which types are equal... * Fixed compile issue. Co-authored-by: Michael Hackstein <michael@arangodb.com> Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> * Feature/aql subquery operations stack calculation exec enumerate exec (#11098) * removed unused import * re-ordered asserts to see which register was wrongly used * added implementation of produceRows with inputRange * added first test to test new produceRows including inputRange * rm skip from calc executor * rm not correct assert, added second test * added produceRows inputrange hasmore test * added new produceRows + test * added skipRowsRange enumerate collection executor * added skipRowsRange enumerate collection executor test * added assert * adde fake query helper, one without auto trx but with prepare plan, the other one without prepare but with trx - also removed fakeit mocks from enumeratecollectionexecutor tests * fixed skipRowsRange in enumerateCollectionExecutor, added test as well * removed todo * added skipAll * added EnumerateCollection as new style executor * added produce and skip rows, not ready yet * added new tests, still not ready, need unsorted compare operators in helper, waiting * rm logs * intermidiate commit - may not work * removed unused import * added implementation of produceRows with inputRange * added first test to test new produceRows including inputRange * rm skip from calc executor * rm not correct assert, added second test * added produceRows inputrange hasmore test * added new produceRows + test * added skipRowsRange enumerate collection executor * added skipRowsRange enumerate collection executor test * added assert * adde fake query helper, one without auto trx but with prepare plan, the other one without prepare but with trx - also removed fakeit mocks from enumeratecollectionexecutor tests * fixed skipRowsRange in enumerateCollectionExecutor, added test as well * removed todo * added skipAll * added EnumerateCollection as new style executor * added produce and skip rows, not ready yet * added new tests, still not ready, need unsorted compare operators in helper, waiting * rm logs * intermidiate commit - may not work * make branch compile again * mv assert * protected server * make enum collection tests work again * fixed ExecBlockImpl * todo optimization * fix gtest calc executor * header file enumcollexec, helper method * fix shrink method in aqlitemblock * impl * nodiscard * calc executor * rm empty line * fix calc exec * tmp re-enable old method * tmp make tests green again, needs cleanup later * use uint64_t instead of u_int64_t * removed unused code * Update arangod/Aql/CalculationExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * Update arangod/Aql/CalculationExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * remove implementation code of old api * Update arangod/Aql/EnumerateCollectionExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * buildable tests again * rm unused code * added assert * make tests compile again ... * disable few more tests, need to be optimized after subquery end start merges Co-authored-by: Michael Hackstein <michael@arangodb.com> * Quickfix for compilation error due to merge * Feature/aql subquery execution block impl execute implementation k shortest path executor (#10940) * Add new produceRow API code for KShortestPathsExecutor * Add KShortestPathsExecutor test to build * Add KShortestPathsExecutor test * Test and bugfix for KShortestPathExecutor * Update k-shortest-paths executor for new interface * Update tests for k-shortest-paths executor * Make ShortestPathExecutor and KShortestPathExecutor more similar * Fix merge messup * Refactor KShortestPathsExecutorInfos * Move KShortestPathsExecutor to new executor interface * Fix tests * Resolve naming conflict * Address some review comments * Fix skip * Fix for new signatures * Fix KShortestPathsExecutor * Fix KShortestPaths regression Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation traversal executor (#11023) * Move to new executor interface * Fix tests and some bugs * new skip signature * new skip signature * Fix Traversal Executor Skipping * Fix compiler complaint Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move SubqueryStartExecutor to new interface (#11025) * Move SubqueryStartExecutor to new interface * Diddle * Tests pass for SubqueryStartExecutor * Delete most of the SubqueryEndExecutor tests They are irrelevant now as they only tested shadowrow behaviour. Shadow Rows are now handled by ExecutionBlockImpl. * Skeleton tests * Junk * Cleanup and make work * Tweak ExecutorTestHelper to test pipelines * SubqueryStart/SubqueryEnd * Blafummel. * Modifications * Fixup tests * Address some comments * Fix. * Fix tests * Various fixes. * Made the branch compile on Mac * Intermediate commit, does not compile, required for merge * Fixed one of the SplicedSubqueryTests * Seperated ShadowRow handling for SubqueryStart/End executors to get a better overview and understanding on how it should behave. * Allow to use a fullCallStack in ExecutorTestHelper. * Added asserts on SubqueryEnd that is not allowed to be called with Skipped from upstream * Use AqlTestCase in SubqueryStartExecutorTest. Transformed first test to TestHelper pipeline * Let ExecutorTestHelper also test for shadowRows * Adapted SingleInput=>Data+Shadow test to testing Helepr and make it pass * Transformed existing SubqueryStart tests to Helper, fixed a test where there was no place for the ShadowRow to be written * Fixed Skip in SubqueryStartExecutor * Allow the Pipeline to add Consumer and dependency without fiddling with the Deque from outside * Added Skip adn ShadowRowForwarding tests to SubqueryStartExecutor * Fixed fullCount of SubqueryStart to be called properly. * Renamed isModificationSubquery -> isModificatioNode again * Consume more than one subquery en-block * Fixed debugReporting in SplicedSubqueryTest. Disabled the skip in subquery tests for now * Removed AQL debug log level from testing config * First version of Subquery Skip bypassing. Including 1 passing test. * Added disabled tests for skip, and proposed to do it later * Fixed reporting of FILTERED in fullCount of EnumerateCollectionExecutor * Enable calculationExecutor execute interface * Fixed compile issues * To Velocypack the modification Subquery value * Fixed illegal assert and reactivated failure tests * Fixed compile issue * Deactivated Calculation again * Added Compatibility Mode to AqlCallStack * Removed IdExecutor<void> the only feature it has can be moved into IDexecutor in general. This still has to be done * When falling back in BlocksWithClients use compatibility stack * ConstFetcher now correctly reports if it skipped all input * Fixed a bug in ExecutionBlockImpl, which would go into FetchMore state, even if the Executor still has data and is happy to produce more, but the output is full. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute remote node api flag (#11159) * Clarified comment * Added api property to RemoteNode * Added missing include * Feature/aql subquery operations stack i research view executor (#11140) * added arangosearch view executor to new style * added new produceRows and skipRowsRange * ires * test jenkins * modify aql call * cleanup branch * Update arangod/Aql/IResearchViewExecutor.cpp Co-Authored-By: Dronplane <andrei.lobov@arangodb.com> * applied suggested changes * also enable IResearchViewMergeExecutor * Fixed an issue in skipRange, where the input was not moved forward in all cases. * Removed unused local variables Co-authored-by: Dronplane <andrei.lobov@arangodb.com> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move SingleRemoteModificationExecutor to new interface (#11166) * Move SingleRemoteModificationExecutor to new interface * Activate SingleRemoteModificationExecutor * Inspect the result of remote modification Co-authored-by: Michael Hackstein <michael@arangodb.com> * New style DistinctCollect (#11096) * First implementation of new style executor for DistinctCollect. * Tests for distinct collect. * Remove debug output. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation noresults (#11180) * Use ExecutorHelper in tests * Implemented no Results fetcher * Moved Constrained sort to execute API (#11173) * Moved Constrained sort to execute API * The LIMIT in ConstrainedSort can actually fetch TOO much, if there is too small input from upstream * Improved profiler tests, we need less calls now \o/ * Fixed unlimited call to limit executor * Feature/aql subquery operations stack aql item block input matrix (#11167) * added new file to cmakelist * added new method skipAllRemainingDataRows * added first implementation of execute - AllRowsFetcher * impl is now using skipAllRemainingDataRows method, also execute methods now using Fetcher::DataRange instead of inputRange * impl is now using skipAllRemainingDataRows method, also execute methods now using Fetcher::DataRange instead of inputRange - header file * AllRowsFetcher now using AqlItemBlockInputMatrix as DataRange, first implementation of execute * first implementation of AqlItemBlockInputMatrix * more cleanup * optimized execut elogic * optimized logic in inputmatrix, still a few todos left * added SortExecutor as newStyle * added produceRows and skipRows * logs (to be removed), skipAllRemainingRows * logs * inputMatrix * changed return type of skipAllRemainingDataRows * impl logs * logs, added clear method * sort test * typo, log * latest changes * cleanup, assert * rm logs * added more todos * todo * new method names, compile again * Reset ExecutorReturnedDone for SubqueryStart and SubqueryEnd. * InputMatrix needs to return state DONE on every input request. It guarantees that we have all input * Removed invalid Assertion, and removed early return if output is full. * Removed DebugLogging from CalculationTest * Fixed skipping in SortExecutor * Fixed upstreamState return of AqlItemMatrix. It is only finalState iff there is no data * Let the InputMatrix drop all rows after FullCount * Allow bypassing with hardLimit 0 in InputMatrix * Disabled the old-code assertion. This way the SortExecutor tests we have will be red, but not crash, allowing the jenkins to produce meaningfull results * Fix assert in WAITING case * Add rudimentary sort executor GTests * Fixed assertion on InputMatrix that starts right away with a shadowRow * Removed declared but not implemented function * changed from: peekDataRowAndState -> to: peekDataRow Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute in restaqlhandler (#11175) * Updated comment * Added two static strings * Avoid unintentional streaming operator of string_view as a container * Added parser for execute rest call * Fixed ambiguous overload in ResultT * Added execute and executeForClient to ExecutionEngine * Implemented executeForClient in RestAqlHandler * Add default value * Updated comment * Changed error attribute * Fixed duplicate log id * Try to make MSVC work * Moved AqlExecuteResult to a separate file * Tried to fix MSVC compile error * Revert "Tried to fix MSVC compile error" This reverts commit f6f43b2c7e4bfe17dafe65f48bcd5c0b6e5c69a9. * Next try to fix MSVC * Revert "Next try to fix MSVC" This reverts commit 528c4c795d10ee0aa6686d143be1f6faa6e9b553. * Revert "Try to make MSVC work" This reverts commit ba7d9c072fccb969da1cdf045eadf49c297da8e9. * Work around MSVC shortcoming * Support executors with multiple dependencies (#11181) * Use DataRange in some places * Add MultiAqlItemBlockInputRange * First implementation of life, universe, and everything * implement skip for UnsortedGather * Some more wiring * Attempt at hacking up execute for DependencyProxy * Compilation fixes! * Fixed AqlCall compilation and unified execute and executeForDependency * Fixed WAITIGN assertion * Sonderlocken. * Foo * Make compile * Make tests pass. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move ModificationExecutors to new interface (#11165) * Move ModificationExecutors to new interface * added inputRange method to inputMatrix class, added switch between DataRange Input <-> Matrix for ModificationExecutors * forgot to enable allrows modification executor in fastForwardType * removed obsolete comment * removed obsolete comment Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery operations stack materialize executor (#11192) * removed obsolete comment, added produceRows and skipRowsRange, enabled MaterializeExecutor as newStyleExecutor * forgot to use advanceRow, fetch input during skipRowsRange * removed old implementations * removed comment, fixed a logical loop error * skip could be simplified as well * temp workaround Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute remote (#11197) * Clarified comment * Updated comment * Added two static strings * Avoid unintentional streaming operator of string_view as a container * Added parser for execute rest call * Fixed ambiguous overload in ResultT * Added execute and executeForClient to ExecutionEngine * Implemented executeForClient in RestAqlHandler * Add default value * Updated comment * Changed error attribute * Added API toggle to RemoteExecutor * Fixed compile error * Fixed duplicate log id * Try to make MSVC work * Moved AqlExecuteResult to a separate file * Moved AqlExecuteResult to a separate file * Implemtented execute in Remote; except for (de)serialization, which is still missing * Tried to fix MSVC compile error * Revert "Tried to fix MSVC compile error" This reverts commit f6f43b2c7e4bfe17dafe65f48bcd5c0b6e5c69a9. * Next try to fix MSVC * Implemented (de)serialization for RemoteExecutor::execute * Fixes * Added minimal AqlCall (de)serialization tests * Bugfix * Added minimal (de)serialization tests for AqlCallStack and AqlExecuteResult * Fixed test * Changed Serialization format a little * Bugfix * Initialize SingletonBlock immediately with an empty row * Revert "Next try to fix MSVC" This reverts commit 528c4c795d10ee0aa6686d143be1f6faa6e9b553. * Revert "Try to make MSVC work" This reverts commit ba7d9c072fccb969da1cdf045eadf49c297da8e9. * Work around MSVC shortcoming * Work around MSVC shortcoming * Attempt to fix windows compile issue Co-authored-by: Michael Hackstein <michael@arangodb.com> * Added test for distinct collect with random order. (#11184) * Feature/aql subquery execution block impl execute implementation subquery executor (#11196) * Intermediate commit, does not compile. * New style for SubqueryExecutor. * Removed bad assert. * Reset subsqueryInitialised in skip. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation count collect (#11172) * Move SubqueryStartExecutor to new interface * Diddle * Tests pass for SubqueryStartExecutor * Delete most of the SubqueryEndExecutor tests They are irrelevant now as they only tested shadowrow behaviour. Shadow Rows are now handled by ExecutionBlockImpl. * Skeleton tests * Junk * Cleanup and make work * Tweak ExecutorTestHelper to test pipelines * SubqueryStart/SubqueryEnd * Blafummel. * Modifications * Fixup tests * Address some comments * Fix. * Fix tests * Various fixes. * Made the branch compile on Mac * Intermediate commit, does not compile, required for merge * Fixed one of the SplicedSubqueryTests * Seperated ShadowRow handling for SubqueryStart/End executors to get a better overview and understanding on how it should behave. * Allow to use a fullCallStack in ExecutorTestHelper. * Added asserts on SubqueryEnd that is not allowed to be called with Skipped from upstream * Use AqlTestCase in SubqueryStartExecutorTest. Transformed first test to TestHelper pipeline * Let ExecutorTestHelper also test for shadowRows * Adapted SingleInput=>Data+Shadow test to testing Helepr and make it pass * Transformed existing SubqueryStart tests to Helper, fixed a test where there was no place for the ShadowRow to be written * Fixed Skip in SubqueryStartExecutor * Allow the Pipeline to add Consumer and dependency without fiddling with the Deque from outside * Added Skip adn ShadowRowForwarding tests to SubqueryStartExecutor * Fixed fullCount of SubqueryStart to be called properly. * Renamed isModificationSubquery -> isModificatioNode again * Consume more than one subquery en-block * Fixed debugReporting in SplicedSubqueryTest. Disabled the skip in subquery tests for now * Removed AQL debug log level from testing config * First version of Subquery Skip bypassing. Including 1 passing test. * Added disabled tests for skip, and proposed to do it later * Fixed reporting of FILTERED in fullCount of EnumerateCollectionExecutor * Enable calculationExecutor execute interface * Fixed compile issues * To Velocypack the modification Subquery value * Fixed illegal assert and reactivated failure tests * Fixed compile issue * Deactivated Calculation again * Added a testSuite for countCollectExecutor * Fixed Subquery Test for CountCollect * Added CountCollectImplementation * Added Compatibility Mode to AqlCallStack * Removed IdExecutor<void> the only feature it has can be moved into IDexecutor in general. This still has to be done * When falling back in BlocksWithClients use compatibility stack * Fixed EnumerateCollectionExecutor * Fixed fullCountBehaviour of EnumerateList. Also removed a unlimited,fullCount combination in IndexExecutor * Fixed endless loop while skipAl in OperationCursor * Fixed View Executor, which sends a no-limit no-offset upstream. * Fixed forwarding in NoResultsExecutor Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> * Activate CalculationExecutors (#11177) * Fixed compile bug-due to hidden merge conflict * Feature/aql subquery execution block impl execute implementation base bugfixes (#11201) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Fix memory leak in traversal (#11210) * AqlSubqueryExecutionBlockImplExecuteImplementation use moveValueInto instead of cloneValueInto (#11213) * Fix memory leak in traversal * Use move instead of clone for AqlValues * Feature/aql subquery execution block impl execute implementation gather executors (#11202) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Feature/aql subquery execute parallel gather (#11215) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute parallel gather 2 (#11221) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather * [WIP] Changed Api for MultiDepExecutors in ExecBlockImpl (not yet in the actual executors) * Moved AqlCallSet into a separate file * Changed SortingGather to use the new API * Changed ParallelUnsortedGather to use the new API * Changed UnsortedGather to use the new API * Moved AqlCall operator<< into .cpp file * Implement operator<< for AqlCallSet * Fix boolean mix-up * Fixed state machine: go to UPSTREAM when the AqlCallSet is not empty * Fixed assertion * Bugfix * SortingGather bugfixes * Added init() method to fix an assertion and cleanup * Removed unused variable * Fixed constrained sort * Fixed constrained sort #2 * Fix boolean mix-up * Remove old interface * Use call parameter for upstream request in produceRows * Remove more old interface code * Add skip methods to MultiAqlItemBlockInputRange * Skip in UnsortedGather * skip for UnsortedGather * Fix skip and upstream calls for UnsortedGather * skipRowsRange change * Remove useless comments * Moved multi-dep related code from ExeBlockImpl to MultiFetcher * Cleanup in SortingGather, implemented parallel fullCount * Try to fix a windows compile error * Simplify and extend skipRowsRange for UnsortedGatherExecutor * Made ParallelUnsortedGather actually parallel * Removed erroneous assertion * Undid erroneous change * Fixed MacOs compile. Also disabled tests for non-relevant AqlCallStacks. They will be removed * Fixed initialize Cursor for multi dependency blocks * Fixed fullCount case in parallel unsorted gather * Fixed fullCount upstream call of ParallelUnsortedGatherExecutor * Fixed fullCount in SortingGather * Windows \o/ if you cannot work properly with constexpr and static asserts, we do not let you do it! * Do not advance in Unsorted gather if there are still rows to skip * Add more comparison operators for AqlCall limits * Send clientCall limits to upstream in SortingGather * Improved fullCount in SortingGatherExectur * Disabled a cluster profile test. We now ask the RemoteNode more often if it already has data. It is a bit unclear to me if this is now better performance wise (<< i think so) or triggers undesired side effects * Helpless attempt to work around issues in stonage Visual Studio Compiler we are using. * Clearly adding an operator on a well defined type causes ambigousness on random basic types using the operator Co-authored-by: Michael Hackstein <michael@arangodb.com> Co-authored-by: Markus Pfeiffer <markus@arangodb.com> * Feature/aql subquery execution block impl execute implementation bypass skip (#11203) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Added an implementation for a SkipResult, in order to simplify exchange of it. * Moved production API -> SkipResult * Made tests compile with new SkipResult * Added a test using skip and limit on subqueries * Prepared to use subqueries in SkipResult * Let subqueries modify the SkipResult subquery stack * Fixed return state of GatherNode * Activate all spliced subquery tests \o/ * Let SubqueryEnd honor the client request * Added a Maintainer only test for the stack, if it is 36 compatible * Added first part of side-effect executors. They now send upstream a fetch-all stack. Need to fix downstream reporting, and call the executor with hardLimit 0 * Add a fake FASTFORWARD call into a subquery-skipped ModificationExecutor. * Added helper shadow row function for SideEffect executors * Let the Modification Executor also produce data, even if no FullCount is used. * Revert "Let the Modification Executor also produce data, even if no FullCount is used." This reverts commit b1c6af5674de445f9e3d6343feb0e3cad422588c. * Re…
mchacki
added a commit
that referenced
this issue
Mar 25, 2020
…h sub queries (#11318) * Added first draft of AqlCallObject * Added AqlItemBlockInputRange * Added Header file for AqlCallStack with comments. No implementation yet. * Added AqlCallStack first draft implementation. Made everything compile * Added empty dummy stubs for execute() in ExecutionBlock * Implementation of new API in FilterExecutor. Rough implementation in ExecutionBlockImpl (not complete, only skip path) everything compiles, but is not tested * Fixed off by one error in InputRange. Fixed FilterExecutor * Added some more tests if executor cannot fulfill atMost * Added heplper functions to the AqlCall struct to avoid mangeling around with variants everywhere. * Add a member DataRange to ExecutionBlockImpl. Also now all Fetcher expose a DataRange, this might not be the correct one every where * Added preliminary minimal implementation of execute to DependencyProxy and SingleRowFetcher. Both untested and yet incomplete, they will only work for MVP. * added first implementation of count collect datarange produceRows function + test * Revert "added first implementation of count collect datarange produceRows function + test" This reverts commit 236825280bb5fd03dfe01febfc4dcf1a2d2e2c2f. * First draft of execute implementation, not yet handling ShadowRows though. This state is broken in the sense that a server does not boot with it and catch tests are broken * First working draft of execute() call in ExecutionBlockImpl. Tests are locally green, however there is at least one query still red (on ShadowRows) * Removed non finished implementation from this Branch. It moved to seperate branch * working - not done yet * Revert "working - not done yet" This reverts commit aa275b48ffc2c8fc9a3c364e2241321ec9c9293f. * first implementation of a ShadowRow fetching interface on AqlItemBlockInputRange * single row fetcher, execute + tests * Fixed SingleRowFetcherTest. Also an AqlItemBlockInputRange can now handle ShadowRows properly * Another fix on handling of shadow rows in InputRanges * Improved human readable output of SortLimit test, while validating the result * Removed debug profile execution of a query * Use simpleVPack printing for Trace of Execute, better to read by humans * Added tests for AqlItemBlockInputRange and adjusted implementation accordingly. * Fixed issue where ExecutionBlockImpl returns DONE, although there are still rows in the DataRange * Final modification of ShadowRows in FilterExecutor. All but profiling tests pass. This needs some discussion about overfetching of data, and propagation of hardLimit * Removed debug log includes * Moved responsibility for the client call into the output row. This can precisely count the elements it has written, and forms a single source of truth. The output row now works as the requested block by client that needs to be filled. Now the FilterExecutorTests are red. * Fixed FilterExecutorTest * Expose required features to AqlCall controlled by the output aql item row. Adapted executionBlockImpl to report there * Fixed bug with AqlCallStack pop operation that actually caused invalid memory access * outputRow now honors limits whan called for isFull. * Fixed isFull() in OutputAqlItemRow * Use the inbound call in skipRows for executors, in order to do a more precise upstream call * Make linker happy with operator< on CallLimit * Implement operator+ on AqlCall::Limit, also use std::visit for readability. * FilterExecutor getSome now propagates offset + min(limits) to upstream. This is pessimistic and avoids overfetching in all cases. NOTE: FilterExecutor should be moved to skipvariant using getSome * Fixed profiler tests for filter. These cannot be guaranteed anymore and it was an undesired, yet uncritical side effect that the tests have been green before * Fixed JSLint * Remove special casing for FilterExecutor * use if constexpr to distinguish between ported executors and non-ported executors enabling step-by-step porting and testing * Add skeleton skip code * Replace C&P accident * Simplify and implement skip for new executor interface * Fixup skipping for filter executor * Rename some methods for clarity and consistency * AqlItemBlockInputRange - hasMore is now hasDataRow - next is now nextDataRow - peek is now peekDataRow - state is now upstreamState - additional method upstreamHasMore * Introduce function to allocate an output block * Fix skipSome simulation bug * Some small cleanups * Replaced assertion on atMost on the output size. Otherwise we got into issues with atMost changes between waiting calls (can occur in our intermediate state now) * Attempt at fixing execute * Fix merge conflict * Feature/aql subquery execution block impl execute implementation exec block impl tests (#10838) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation shortest path executor (#10780) * Hook up new ShortestPathExecutor * Small fixes * Removed now unused code. * Moved ShortestPathExecutor tests to use Execute only. * Added a test for shortestPathExecutor where the output is smaller than the amount of entries within the path. * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Fixed Windows compile issue * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Windows compile fix * Attempt at fixing ShortestPathExecutorTest for Windows * Windows compile fix * Added a skip test and fixed the producing executor to recreate the list once for every input line. * Frankenstein my tests into the branch * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Hack * Removed designated initializers. Thanks for not supporting it MSVC! * Update ShortestPathExecutorTest * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Umbau 2.0 * Reshape output and skip a bit * Some test fixes * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Square 1 * Input validation * ShortestPathExecutor with fullCount and tests Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation exec block impl tests shadow rows (#10970) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Added tests for ExecutionBlockImpl, when there is a WAITING passed through. There is no test yet for in between waitings. * Allow to configure WAITING variant in WAITING block mock. * Use the WaitingBlockMock to be independent of Singleton in test * Added WAITING + Passthrough tests * Added a test to path through shadowRows without data. * Waiting ExecutionBlock Mock can now return/skip partial blocks. However it does NOT CARE for ShadowRows * Added tests and fixed an issue in ShadowRow forwarding * Added tests for shadowRow forwarding * Added the intenral state of Execute as member of ExecutionBlockImpl. We need to retain parts of this state accross returns * Fixed all Waiting and multiple call tests * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Fixed invalid memory access in test code. Do never return a lambda that references a stack variable... * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip * Allways -> Always. * Added tests for pass-by of non-relevant Subqueries. Fixed a bug found on the way in WaitingBlockMock. Now some other tests fail \o/ * Fixed failed tests. Now tests correspond to correct behaviour of WaitingExecutionBlockMock * Intermediate version, doe snot work * Fixed the empty subquery tests Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation sorted collect (#11020) * SortedCollect executor and some tests. * More tests. * More const and skip implemented. * Fixed some bugs found by tests. * Refactored skip. * Broken state with test framework. * Extracted testing framework into separate file and adjusted it to use WaitingBlockMock. However it is not yet fully implemented * Added another SortedExecutor test without fullcount. This passes. Also assert on skip and State * Removed obsolete assertion in skip, added another nonFullcount test. * More tests. Fixed some bugs in the test framework. * More cleanup. * Remove version control marker. * Removed wrong assertion. * Apply suggestions from code review Co-Authored-By: Michael Hackstein <michael@arangodb.com> * Remove dead code. * Removed fetcher. Co-authored-by: Michael Hackstein <michael@arangodb.com> * ReturnExecutor New style (#10831) * ReturnExecutor implemented. Tests misssing. * Apply suggestions from code review Co-Authored-By: Tobias Gödderz <tobias@arangodb.com> * Implemented skipRows in Return. Also fixed ExecutionBlock references in static asserts * Implemented first new style test for Return. * Moved the traceExecute calls into baseclass. * Added tracing around execute in WaitingExecutionBlockMock. Fixed a test bug, if the AqlCall covers all of the first block, but only a part of the Second block. You would only get the part of the second block. * Added more tests for standard behaviour of ReturnExecutor * Added an assertion. * Simplified call to ReturnExecutor * Prepare tests for doCount. Also added test for different input register. * Added accessor for _stats in Engine. * Included count tests in ReturnExecutorTest. * Removed old obsolete test. * Removed obsolete code-path * Fixed unrelated bug in FilterExecutor. In some cases it did lie on skipRows that it has more, even if it does not have and the upstream does not have as well. Thank you jenkins for hitting into this race :+1: Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Tweak ExecutorTestHelper to test pipelines * Revert "Tweak ExecutorTestHelper to test pipelines" This reverts commit 6359f0c926cad30b4efd7ce263fb3b64563621b8. * Added Stats return value to skipRowsRange (#11081) * Added Stats return value to skipRowsRange * Fixed catch test. Somehow i merged the incorrect part Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Additional Assertion in ExecutionBlockImpl (#11077) * Added a assertion arround skipRowsInRange that ensures that the Call and the skippedLocal do not get out-of-sync * Improved the assertion * Feature/aql subquery operations stack id executor (#10986) * added inputRange skip and produce * added inputRange skip and produce * Added shouldSkip helper to AqlCall * Added implementation if execute to Constfetcher. Not specifically tested yet. * Moved IDExecutor to newStyle executor * Added some documentation to SingleRowFetcher * Adapted IdExecutor tests. * Added an assertion in ConstFetcher * Fixed initialize cursor in ExecutionBlockImpl * Added tests for Initilaize cursor in IdExecutor. There is a special implementation for this... * Fixed linker issues, because of variables in cpp files named identical and not beeing static * Started refactoring the Scatter/Gather Executor logic. Added an intermediate class that does not use a template and can be casted to. This state compiles but will not work. * Made BlocksWithClientsImpl a template, based on an executor * Added a dummy implementation to fetch a new block. Distribute not yet implemented. * Added superoptimistic first implementation of ScatterExecutor return. Only works if there are no shadowRows. * Implemented logic to slice Blocks in Scatter according to the given call. Also added handfull of test for it, not complete though * Fixed a Scatter Executor handling of softLimit. Added another test to ansure diverse calls of clients to Scatter * Added random ordering test framework on ScatterExecutor * Added some tests for ScatterExecutor including subqueries * Removed todo, as it has been solved by now * Implemented Scatter by using IDExecutor. Tests not green yet * Implementation of ConstFetcher that allows to inject fullBlocks of AqlItemRows, and not only single rows. Also handles ShadowRows now in a correct way * Do not jump over relevant shadow rows in ConstFetcher. * Fixed tests. * Moved information of DistributeExecutor into the Infos. This resulted in adapting the Constructor API. I used this situation to do a big cleanup of ancient code. From this point there is no way back. * Implemented the last bits of DistributeExecutor * 'Implemented' execute by fallback to old API in RemoteExecutor * Fixed an EdgeCase with Scatter Executor, where the data was modified in place right after scatter. This has side-effects on peer executors. So we for now do a full copy of every block here. * Removed invalid assertions, also fixed RegisterInput to DistributeExecutor * Removed obsolete TODOs * Update arangod/Aql/IdExecutor.cpp Co-Authored-By: Lars Maier <lars@arangodb.com> Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Lars Maier <lars@arangodb.com> * Replaced the operator== and operator!= with `isSameBlockAndIndex`. (#11088) * New style IndexExecutor. (#11029) * New style IndexExecutor. * Small changes. * Fixing some logic for skipping. * Small changes. * TRI_ASSERT(FALSE); * Removed _fetcher member. (again) * Feature/aql subquery execution block impl execute implementation debug logging (#11112) * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * Feature/aql subquery operations stack hashed collect executor (#11103) * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * added inputRange skip and produce * added inputRange skip and produce * Added shouldSkip helper to AqlCall * Added implementation if execute to Constfetcher. Not specifically tested yet. * Moved IDExecutor to newStyle executor * Added some documentation to SingleRowFetcher * Adapted IdExecutor tests. * Added an assertion in ConstFetcher * Fixed initialize cursor in ExecutionBlockImpl * Added tests for Initilaize cursor in IdExecutor. There is a special implementation for this... * Fixed linker issues, because of variables in cpp files named identical and not beeing static * Started refactoring the Scatter/Gather Executor logic. Added an intermediate class that does not use a template and can be casted to. This state compiles but will not work. * Made BlocksWithClientsImpl a template, based on an executor * Added a dummy implementation to fetch a new block. Distribute not yet implemented. * Added superoptimistic first implementation of ScatterExecutor return. Only works if there are no shadowRows. * Implemented logic to slice Blocks in Scatter according to the given call. Also added handfull of test for it, not complete though * Fixed a Scatter Executor handling of softLimit. Added another test to ansure diverse calls of clients to Scatter * Added random ordering test framework on ScatterExecutor * Added some tests for ScatterExecutor including subqueries * Removed todo, as it has been solved by now * Implemented Scatter by using IDExecutor. Tests not green yet * Implementation of ConstFetcher that allows to inject fullBlocks of AqlItemRows, and not only single rows. Also handles ShadowRows now in a correct way * Do not jump over relevant shadow rows in ConstFetcher. * Fixed tests. * Moved information of DistributeExecutor into the Infos. This resulted in adapting the Constructor API. I used this situation to do a big cleanup of ancient code. From this point there is no way back. * Implemented the last bits of DistributeExecutor * 'Implemented' execute by fallback to old API in RemoteExecutor * Fixed an EdgeCase with Scatter Executor, where the data was modified in place right after scatter. This has side-effects on peer executors. So we for now do a full copy of every block here. * Removed invalid assertions, also fixed RegisterInput to DistributeExecutor * Removed obsolete TODOs * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * Made the branch compile again * Update arangod/Aql/IdExecutor.cpp Co-Authored-By: Lars Maier <lars@arangodb.com> * Intermediate commit. Empty might not compile * Added some external helper functions to compare ItemBlocks in different variants * Added a brief description for the bunch of input parameters in HashedCollect * Allow unordered output tests in TestHelper class. * Added first HashedCollectExecutor test using the helper * Allow to use never wait in getSome / skipSome variant of WaitingExecutionBlockMock. * Added more HashedCollect tests (multi group, count) and started with framework for AggregateMethods * Fixed simple Aggregate Test * Improved output for unordered Row in Output test. * Added test around collecting of Arrays and Objects * Also allow a fullCount call in the old Execute Wrapper * Almost Completed register and call behaviour test suite * Added a todo for a missing test * Improved setup of AqlTestCase framework, not only start the server once. This speeds up AQL test by at least a factor of 10. * implemented newStyle HashedCollect executor. * Cleanup of old code. * Added test cases where the upstream producer lies to HashedCollect. * Let HashedCollect and Impl life with the fact that upstream lies. * Improved gtest with multiple calls. * Simplify input consume logic of HashedCollect Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Lars Maier <lars@arangodb.com> * Feature/aql subquery operations stack enumerate list (#10988) * produce inputRange * produce inputRange test * produce inputRange test 2 * apply new api * enable new api * enumeratelistexecutor changes * added hackis shouldSkip method * add new execution style for enumerate lsit * added skip for enumerate list * edit of enumerate list executor * fixed enumlist executor * optimized skip loop * more tests, using new framework * allow empty output checks * added new style tests, removed old tests * removed old produceRows method * fixed tests * added one missing exit case * added another exit in SHADOWROWS mode, as we need to take a look at fullCount there * Revert "added another exit in SHADOWROWS mode, as we need to take a look at fullCount there" This reverts commit e74b7cede78f81b8fece034fc921a969e91dda12. * do not take care of fullCount in executor itself, impl will do * removed obsolete comment * rm _skipped because unused, return upstreamstate * added invalid value test, added border edge tests for softLimit, hardLimit and fullCount * api change * added test for inner states inside the executor * added new tests * fixed assert * removed two not needed ifs * asserted a case in our test helper * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * undo this commit before merge * undo this commit before merge * Merged Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation inject stack into fetcher (#11124) * Added api to pass through the stack to the dependencyProxy. THis is just temporary and doomed to be removed as soon as API is completed. * Let depdendency Proxy forward the Stack. This may not work in all cases. * Default implement Assignment constructor. * Removed Query jenkins debug output again * Fixed forwarding of stack in skipSome * Tweak ExecutorTestHelper to test pipelines (#11079) * Tweak ExecutorTestHelper to test pipelines * Modifications * Fixup tests * Address some comments * Fix. * AQL execute in LimitExecutor (#10886) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Fixed merge * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Reverted accidental change * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Assert no input rows in skip * Began writing a test framework for LimitExecutor * Removed unused code * Continued test framework * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip * Make AqlItemBlockInputRange hold a skipped count * Added == for LimitStats, moved code to .cpp file * Added Stats return value to skipRowsRange * WIP: Implement LimitExecutor test framework * Fixed namespace * Added missing advanceRow call * Added parametrized test cases * Dump * Made test output more readable * Fixed merge conflicts * Set engine in createFakeQuery() * [WIP] Dump: not working * Fixed merge conflicts * Switch from boost::variant to std::variant * Fix comparison of differently sized blocks * Fixed usage of temporary * Fixed skip handling in ExecutionBlockImpl * Add upstream fullCount in LimitExecutor * Fix assertion that was too narrow * Again, fixed skip handling in ExecutionBlockImpl * Extended ExecutorTestHelper, mainly a loop to handle passthrough Executors * Fixed expected skipped count * Fixed bugs in LimitExecutor * Another bugfix in LimitExecutor * Fixed expected output calculation * Removed some comments * Improve test speed by not recreating MockAqlServer for every case * Allow additional variants in ExecutorTestHelper * Tried to fix a bug in LimitExecutor, not quite working * A little cleanup * Toggle lieAboutHasMore * Reverted broken fix, fixed WaitingExecutionBlockMock * Fixed merge conflicts * Add LimitStats to expected checks in tests * Fixed another leftover merge conflict * Fixed fullCount stats * Removed old code from LimitExecutor * Updated ExecutionBlockImpl to allow removal of old executor methods * Removed old tests * Simplified test code * Added comments and constructors to AqlCall * Instantiate test cases with multiple different calls * Fixed a bug in run(), made looping optional * Fixed a bug in DependencyProxy * Fixed a bug and an assertion in LimitExecutor * Added assertions and a fix in WaitingExecutionBlockMock * Enabled stats tests again * Made the executor compile under Mac * Let AqlHelper compare profiled stats as well * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * Modified skip logic in LimitExecutor to simplify what has to be reported to downStream and what is handled by internal skip. Also allow to do skip and fullCount in one go, if limit is 0 * Adapted the Expected skipped output in Limit test * Forward fullCount from client through LIMIT. Simplyfied produceLogic. * Reduced the amount of Tests in LimitExecutor. Need to decide on a suitable subset * Removed lieAboutHasMore by simply adding an empty block to the end that is blocking return in the WaitingExecutor * Fixed SingleRowFetcher to allow a Range with only skip information. * Intermed version of ExecutionBlockImpl. Needs to be continued * Allow hardLimit to pass getSome check * Intermed commit * Unified FastForward and FullCount code path * Produce and Skip go to FastForward if the Executor reports done and the output row is full. This way we can handle more FUllCount cases straight away. * Fixed the expected value for FullCount stats in test. This is rather awkward to automatically compute... * Fixed lying of the ExecutionBlockMock. Did not lie correctly if the limit was reached by softLimit. * Make the rules for supporting the OldAPI more relaxed. * Added a missmatch of outputColumns and internal output columns in TestHelper * Fixed AqlItemBlockRangeTests * Implemented unreachable code, otherwise our Jenkins does not compile it ;( * Fixed SortedCollect, it might count the last group twice on fullCount, if produce ends exactly in this group * Use AqlTestCase in ReturnExecutorTest * Return Executor needs to report skipped from upstream properly * Fixed fullCount call behaviour * Fixed expected numberSkipped in LimitExecutor test * Added some DEBUG level log messages in FastForward * Fixed reporting of skip in skipSome case and removed dead code * Fixed Impl test, we cannot let the WaitingBlockMock do counting on ShadowRows, it is not implemented there. * Removed unused function * Pointeless comment * Fixed skipSome of Filter * Avoid endless Loop in indexExecutor fullCount, where it tried to skipped 0 documents forever * Fixed return state of IndexExecutor * Fake DONE on hardLimit for oldAPI, apply only on toplevel. Improve shortcutting behviour to jump into FastForward everytime HARDLimit is reached, even if output is full. * Added an assertion that multiple UPSTREAM call rounds do not mess up the callStack by leaving the _upstreamCall on top. Now the executor itself pops off the call, this is subject to change if we manage the Stack differently * Was erlaube skip? Adjsuted tests to honor that we can do skip and limit now in one go, for all executors that are updated already * Fixed ConstFetcher, to not modify the CallStack, it only has a reference, where the other Fetchers copy it to the Executors, and modify it there * Do not overfetch on passthrough if the block is full. * We cannot create DONE on oldStyle if we need fullcount... * Unfortunately we teporarily need more calls in fullCount case, as long as one of the Executors above is not yet on execute * Mhm gcc and clang do not aggree on which types are equal... * Fixed compile issue. Co-authored-by: Michael Hackstein <michael@arangodb.com> Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> * Feature/aql subquery operations stack calculation exec enumerate exec (#11098) * removed unused import * re-ordered asserts to see which register was wrongly used * added implementation of produceRows with inputRange * added first test to test new produceRows including inputRange * rm skip from calc executor * rm not correct assert, added second test * added produceRows inputrange hasmore test * added new produceRows + test * added skipRowsRange enumerate collection executor * added skipRowsRange enumerate collection executor test * added assert * adde fake query helper, one without auto trx but with prepare plan, the other one without prepare but with trx - also removed fakeit mocks from enumeratecollectionexecutor tests * fixed skipRowsRange in enumerateCollectionExecutor, added test as well * removed todo * added skipAll * added EnumerateCollection as new style executor * added produce and skip rows, not ready yet * added new tests, still not ready, need unsorted compare operators in helper, waiting * rm logs * intermidiate commit - may not work * removed unused import * added implementation of produceRows with inputRange * added first test to test new produceRows including inputRange * rm skip from calc executor * rm not correct assert, added second test * added produceRows inputrange hasmore test * added new produceRows + test * added skipRowsRange enumerate collection executor * added skipRowsRange enumerate collection executor test * added assert * adde fake query helper, one without auto trx but with prepare plan, the other one without prepare but with trx - also removed fakeit mocks from enumeratecollectionexecutor tests * fixed skipRowsRange in enumerateCollectionExecutor, added test as well * removed todo * added skipAll * added EnumerateCollection as new style executor * added produce and skip rows, not ready yet * added new tests, still not ready, need unsorted compare operators in helper, waiting * rm logs * intermidiate commit - may not work * make branch compile again * mv assert * protected server * make enum collection tests work again * fixed ExecBlockImpl * todo optimization * fix gtest calc executor * header file enumcollexec, helper method * fix shrink method in aqlitemblock * impl * nodiscard * calc executor * rm empty line * fix calc exec * tmp re-enable old method * tmp make tests green again, needs cleanup later * use uint64_t instead of u_int64_t * removed unused code * Update arangod/Aql/CalculationExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * Update arangod/Aql/CalculationExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * remove implementation code of old api * Update arangod/Aql/EnumerateCollectionExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * buildable tests again * rm unused code * added assert * make tests compile again ... * disable few more tests, need to be optimized after subquery end start merges Co-authored-by: Michael Hackstein <michael@arangodb.com> * Quickfix for compilation error due to merge * Feature/aql subquery execution block impl execute implementation k shortest path executor (#10940) * Add new produceRow API code for KShortestPathsExecutor * Add KShortestPathsExecutor test to build * Add KShortestPathsExecutor test * Test and bugfix for KShortestPathExecutor * Update k-shortest-paths executor for new interface * Update tests for k-shortest-paths executor * Make ShortestPathExecutor and KShortestPathExecutor more similar * Fix merge messup * Refactor KShortestPathsExecutorInfos * Move KShortestPathsExecutor to new executor interface * Fix tests * Resolve naming conflict * Address some review comments * Fix skip * Fix for new signatures * Fix KShortestPathsExecutor * Fix KShortestPaths regression Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation traversal executor (#11023) * Move to new executor interface * Fix tests and some bugs * new skip signature * new skip signature * Fix Traversal Executor Skipping * Fix compiler complaint Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move SubqueryStartExecutor to new interface (#11025) * Move SubqueryStartExecutor to new interface * Diddle * Tests pass for SubqueryStartExecutor * Delete most of the SubqueryEndExecutor tests They are irrelevant now as they only tested shadowrow behaviour. Shadow Rows are now handled by ExecutionBlockImpl. * Skeleton tests * Junk * Cleanup and make work * Tweak ExecutorTestHelper to test pipelines * SubqueryStart/SubqueryEnd * Blafummel. * Modifications * Fixup tests * Address some comments * Fix. * Fix tests * Various fixes. * Made the branch compile on Mac * Intermediate commit, does not compile, required for merge * Fixed one of the SplicedSubqueryTests * Seperated ShadowRow handling for SubqueryStart/End executors to get a better overview and understanding on how it should behave. * Allow to use a fullCallStack in ExecutorTestHelper. * Added asserts on SubqueryEnd that is not allowed to be called with Skipped from upstream * Use AqlTestCase in SubqueryStartExecutorTest. Transformed first test to TestHelper pipeline * Let ExecutorTestHelper also test for shadowRows * Adapted SingleInput=>Data+Shadow test to testing Helepr and make it pass * Transformed existing SubqueryStart tests to Helper, fixed a test where there was no place for the ShadowRow to be written * Fixed Skip in SubqueryStartExecutor * Allow the Pipeline to add Consumer and dependency without fiddling with the Deque from outside * Added Skip adn ShadowRowForwarding tests to SubqueryStartExecutor * Fixed fullCount of SubqueryStart to be called properly. * Renamed isModificationSubquery -> isModificatioNode again * Consume more than one subquery en-block * Fixed debugReporting in SplicedSubqueryTest. Disabled the skip in subquery tests for now * Removed AQL debug log level from testing config * First version of Subquery Skip bypassing. Including 1 passing test. * Added disabled tests for skip, and proposed to do it later * Fixed reporting of FILTERED in fullCount of EnumerateCollectionExecutor * Enable calculationExecutor execute interface * Fixed compile issues * To Velocypack the modification Subquery value * Fixed illegal assert and reactivated failure tests * Fixed compile issue * Deactivated Calculation again * Added Compatibility Mode to AqlCallStack * Removed IdExecutor<void> the only feature it has can be moved into IDexecutor in general. This still has to be done * When falling back in BlocksWithClients use compatibility stack * ConstFetcher now correctly reports if it skipped all input * Fixed a bug in ExecutionBlockImpl, which would go into FetchMore state, even if the Executor still has data and is happy to produce more, but the output is full. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute remote node api flag (#11159) * Clarified comment * Added api property to RemoteNode * Added missing include * Feature/aql subquery operations stack i research view executor (#11140) * added arangosearch view executor to new style * added new produceRows and skipRowsRange * ires * test jenkins * modify aql call * cleanup branch * Update arangod/Aql/IResearchViewExecutor.cpp Co-Authored-By: Dronplane <andrei.lobov@arangodb.com> * applied suggested changes * also enable IResearchViewMergeExecutor * Fixed an issue in skipRange, where the input was not moved forward in all cases. * Removed unused local variables Co-authored-by: Dronplane <andrei.lobov@arangodb.com> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move SingleRemoteModificationExecutor to new interface (#11166) * Move SingleRemoteModificationExecutor to new interface * Activate SingleRemoteModificationExecutor * Inspect the result of remote modification Co-authored-by: Michael Hackstein <michael@arangodb.com> * New style DistinctCollect (#11096) * First implementation of new style executor for DistinctCollect. * Tests for distinct collect. * Remove debug output. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation noresults (#11180) * Use ExecutorHelper in tests * Implemented no Results fetcher * Moved Constrained sort to execute API (#11173) * Moved Constrained sort to execute API * The LIMIT in ConstrainedSort can actually fetch TOO much, if there is too small input from upstream * Improved profiler tests, we need less calls now \o/ * Fixed unlimited call to limit executor * Feature/aql subquery operations stack aql item block input matrix (#11167) * added new file to cmakelist * added new method skipAllRemainingDataRows * added first implementation of execute - AllRowsFetcher * impl is now using skipAllRemainingDataRows method, also execute methods now using Fetcher::DataRange instead of inputRange * impl is now using skipAllRemainingDataRows method, also execute methods now using Fetcher::DataRange instead of inputRange - header file * AllRowsFetcher now using AqlItemBlockInputMatrix as DataRange, first implementation of execute * first implementation of AqlItemBlockInputMatrix * more cleanup * optimized execut elogic * optimized logic in inputmatrix, still a few todos left * added SortExecutor as newStyle * added produceRows and skipRows * logs (to be removed), skipAllRemainingRows * logs * inputMatrix * changed return type of skipAllRemainingDataRows * impl logs * logs, added clear method * sort test * typo, log * latest changes * cleanup, assert * rm logs * added more todos * todo * new method names, compile again * Reset ExecutorReturnedDone for SubqueryStart and SubqueryEnd. * InputMatrix needs to return state DONE on every input request. It guarantees that we have all input * Removed invalid Assertion, and removed early return if output is full. * Removed DebugLogging from CalculationTest * Fixed skipping in SortExecutor * Fixed upstreamState return of AqlItemMatrix. It is only finalState iff there is no data * Let the InputMatrix drop all rows after FullCount * Allow bypassing with hardLimit 0 in InputMatrix * Disabled the old-code assertion. This way the SortExecutor tests we have will be red, but not crash, allowing the jenkins to produce meaningfull results * Fix assert in WAITING case * Add rudimentary sort executor GTests * Fixed assertion on InputMatrix that starts right away with a shadowRow * Removed declared but not implemented function * changed from: peekDataRowAndState -> to: peekDataRow Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute in restaqlhandler (#11175) * Updated comment * Added two static strings * Avoid unintentional streaming operator of string_view as a container * Added parser for execute rest call * Fixed ambiguous overload in ResultT * Added execute and executeForClient to ExecutionEngine * Implemented executeForClient in RestAqlHandler * Add default value * Updated comment * Changed error attribute * Fixed duplicate log id * Try to make MSVC work * Moved AqlExecuteResult to a separate file * Tried to fix MSVC compile error * Revert "Tried to fix MSVC compile error" This reverts commit f6f43b2c7e4bfe17dafe65f48bcd5c0b6e5c69a9. * Next try to fix MSVC * Revert "Next try to fix MSVC" This reverts commit 528c4c795d10ee0aa6686d143be1f6faa6e9b553. * Revert "Try to make MSVC work" This reverts commit ba7d9c072fccb969da1cdf045eadf49c297da8e9. * Work around MSVC shortcoming * Support executors with multiple dependencies (#11181) * Use DataRange in some places * Add MultiAqlItemBlockInputRange * First implementation of life, universe, and everything * implement skip for UnsortedGather * Some more wiring * Attempt at hacking up execute for DependencyProxy * Compilation fixes! * Fixed AqlCall compilation and unified execute and executeForDependency * Fixed WAITIGN assertion * Sonderlocken. * Foo * Make compile * Make tests pass. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move ModificationExecutors to new interface (#11165) * Move ModificationExecutors to new interface * added inputRange method to inputMatrix class, added switch between DataRange Input <-> Matrix for ModificationExecutors * forgot to enable allrows modification executor in fastForwardType * removed obsolete comment * removed obsolete comment Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery operations stack materialize executor (#11192) * removed obsolete comment, added produceRows and skipRowsRange, enabled MaterializeExecutor as newStyleExecutor * forgot to use advanceRow, fetch input during skipRowsRange * removed old implementations * removed comment, fixed a logical loop error * skip could be simplified as well * temp workaround Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute remote (#11197) * Clarified comment * Updated comment * Added two static strings * Avoid unintentional streaming operator of string_view as a container * Added parser for execute rest call * Fixed ambiguous overload in ResultT * Added execute and executeForClient to ExecutionEngine * Implemented executeForClient in RestAqlHandler * Add default value * Updated comment * Changed error attribute * Added API toggle to RemoteExecutor * Fixed compile error * Fixed duplicate log id * Try to make MSVC work * Moved AqlExecuteResult to a separate file * Moved AqlExecuteResult to a separate file * Implemtented execute in Remote; except for (de)serialization, which is still missing * Tried to fix MSVC compile error * Revert "Tried to fix MSVC compile error" This reverts commit f6f43b2c7e4bfe17dafe65f48bcd5c0b6e5c69a9. * Next try to fix MSVC * Implemented (de)serialization for RemoteExecutor::execute * Fixes * Added minimal AqlCall (de)serialization tests * Bugfix * Added minimal (de)serialization tests for AqlCallStack and AqlExecuteResult * Fixed test * Changed Serialization format a little * Bugfix * Initialize SingletonBlock immediately with an empty row * Revert "Next try to fix MSVC" This reverts commit 528c4c795d10ee0aa6686d143be1f6faa6e9b553. * Revert "Try to make MSVC work" This reverts commit ba7d9c072fccb969da1cdf045eadf49c297da8e9. * Work around MSVC shortcoming * Work around MSVC shortcoming * Attempt to fix windows compile issue Co-authored-by: Michael Hackstein <michael@arangodb.com> * Added test for distinct collect with random order. (#11184) * Feature/aql subquery execution block impl execute implementation subquery executor (#11196) * Intermediate commit, does not compile. * New style for SubqueryExecutor. * Removed bad assert. * Reset subsqueryInitialised in skip. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation count collect (#11172) * Move SubqueryStartExecutor to new interface * Diddle * Tests pass for SubqueryStartExecutor * Delete most of the SubqueryEndExecutor tests They are irrelevant now as they only tested shadowrow behaviour. Shadow Rows are now handled by ExecutionBlockImpl. * Skeleton tests * Junk * Cleanup and make work * Tweak ExecutorTestHelper to test pipelines * SubqueryStart/SubqueryEnd * Blafummel. * Modifications * Fixup tests * Address some comments * Fix. * Fix tests * Various fixes. * Made the branch compile on Mac * Intermediate commit, does not compile, required for merge * Fixed one of the SplicedSubqueryTests * Seperated ShadowRow handling for SubqueryStart/End executors to get a better overview and understanding on how it should behave. * Allow to use a fullCallStack in ExecutorTestHelper. * Added asserts on SubqueryEnd that is not allowed to be called with Skipped from upstream * Use AqlTestCase in SubqueryStartExecutorTest. Transformed first test to TestHelper pipeline * Let ExecutorTestHelper also test for shadowRows * Adapted SingleInput=>Data+Shadow test to testing Helepr and make it pass * Transformed existing SubqueryStart tests to Helper, fixed a test where there was no place for the ShadowRow to be written * Fixed Skip in SubqueryStartExecutor * Allow the Pipeline to add Consumer and dependency without fiddling with the Deque from outside * Added Skip adn ShadowRowForwarding tests to SubqueryStartExecutor * Fixed fullCount of SubqueryStart to be called properly. * Renamed isModificationSubquery -> isModificatioNode again * Consume more than one subquery en-block * Fixed debugReporting in SplicedSubqueryTest. Disabled the skip in subquery tests for now * Removed AQL debug log level from testing config * First version of Subquery Skip bypassing. Including 1 passing test. * Added disabled tests for skip, and proposed to do it later * Fixed reporting of FILTERED in fullCount of EnumerateCollectionExecutor * Enable calculationExecutor execute interface * Fixed compile issues * To Velocypack the modification Subquery value * Fixed illegal assert and reactivated failure tests * Fixed compile issue * Deactivated Calculation again * Added a testSuite for countCollectExecutor * Fixed Subquery Test for CountCollect * Added CountCollectImplementation * Added Compatibility Mode to AqlCallStack * Removed IdExecutor<void> the only feature it has can be moved into IDexecutor in general. This still has to be done * When falling back in BlocksWithClients use compatibility stack * Fixed EnumerateCollectionExecutor * Fixed fullCountBehaviour of EnumerateList. Also removed a unlimited,fullCount combination in IndexExecutor * Fixed endless loop while skipAl in OperationCursor * Fixed View Executor, which sends a no-limit no-offset upstream. * Fixed forwarding in NoResultsExecutor Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> * Activate CalculationExecutors (#11177) * Fixed compile bug-due to hidden merge conflict * Feature/aql subquery execution block impl execute implementation base bugfixes (#11201) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Fix memory leak in traversal (#11210) * AqlSubqueryExecutionBlockImplExecuteImplementation use moveValueInto instead of cloneValueInto (#11213) * Fix memory leak in traversal * Use move instead of clone for AqlValues * Feature/aql subquery execution block impl execute implementation gather executors (#11202) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Feature/aql subquery execute parallel gather (#11215) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute parallel gather 2 (#11221) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather * [WIP] Changed Api for MultiDepExecutors in ExecBlockImpl (not yet in the actual executors) * Moved AqlCallSet into a separate file * Changed SortingGather to use the new API * Changed ParallelUnsortedGather to use the new API * Changed UnsortedGather to use the new API * Moved AqlCall operator<< into .cpp file * Implement operator<< for AqlCallSet * Fix boolean mix-up * Fixed state machine: go to UPSTREAM when the AqlCallSet is not empty * Fixed assertion * Bugfix * SortingGather bugfixes * Added init() method to fix an assertion and cleanup * Removed unused variable * Fixed constrained sort * Fixed constrained sort #2 * Fix boolean mix-up * Remove old interface * Use call parameter for upstream request in produceRows * Remove more old interface code * Add skip methods to MultiAqlItemBlockInputRange * Skip in UnsortedGather * skip for UnsortedGather * Fix skip and upstream calls for UnsortedGather * skipRowsRange change * Remove useless comments * Moved multi-dep related code from ExeBlockImpl to MultiFetcher * Cleanup in SortingGather, implemented parallel fullCount * Try to fix a windows compile error * Simplify and extend skipRowsRange for UnsortedGatherExecutor * Made ParallelUnsortedGather actually parallel * Removed erroneous assertion * Undid erroneous change * Fixed MacOs compile. Also disabled tests for non-relevant AqlCallStacks. They will be removed * Fixed initialize Cursor for multi dependency blocks * Fixed fullCount case in parallel unsorted gather * Fixed fullCount upstream call of ParallelUnsortedGatherExecutor * Fixed fullCount in SortingGather * Windows \o/ if you cannot work properly with constexpr and static asserts, we do not let you do it! * Do not advance in Unsorted gather if there are still rows to skip * Add more comparison operators for AqlCall limits * Send clientCall limits to upstream in SortingGather * Improved fullCount in SortingGatherExectur * Disabled a cluster profile test. We now ask the RemoteNode more often if it already has data. It is a bit unclear to me if this is now better performance wise (<< i think so) or triggers undesired side effects * Helpless attempt to work around issues in stonage Visual Studio Compiler we are using. * Clearly adding an operator on a well defined type causes ambigousness on random basic types using the operator Co-authored-by: Michael Hackstein <michael@arangodb.com> Co-authored-by: Markus Pfeiffer <markus@arangodb.com> * Feature/aql subquery execution block impl execute implementation bypass skip (#11203) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Added an implementation for a SkipResult, in order to simplify exchange of it. * Moved production API -> SkipResult * Made tests compile with new SkipResult * Added a test using skip and limit on subqueries * Prepared to use subqueries in SkipResult * Let subqueries modify the SkipResult subquery stack * Fixed return state of GatherNode * Activate all spliced subquery tests \o/ * Let SubqueryEnd honor the client request * Added a Maintainer only test for the stack, if it is 36 compatible * Added first part of side-effect executors. They now send upstream a fetch-all stack. Need to fix downstream reporting, and call the executor with hardLimit 0 * Add a fake FASTFORWARD call into a subquery-skipped ModificationExecutor. * Added helper shadow row function for SideEffect executors * Let the Modification Executor also produce data, even if no FullCount is used. * Revert "Let the Modification Executor also produce data, even if no FullCount is used." This reverts commit b1c6af5674de445f9e3d6343feb0e3cad422588c. * Revert "…
mchacki
added a commit
that referenced
this issue
Apr 2, 2020
…esting (#11349) * Added first draft of AqlCallObject * Added AqlItemBlockInputRange * Added Header file for AqlCallStack with comments. No implementation yet. * Added AqlCallStack first draft implementation. Made everything compile * Added empty dummy stubs for execute() in ExecutionBlock * Implementation of new API in FilterExecutor. Rough implementation in ExecutionBlockImpl (not complete, only skip path) everything compiles, but is not tested * Fixed off by one error in InputRange. Fixed FilterExecutor * Added some more tests if executor cannot fulfill atMost * Added heplper functions to the AqlCall struct to avoid mangeling around with variants everywhere. * Add a member DataRange to ExecutionBlockImpl. Also now all Fetcher expose a DataRange, this might not be the correct one every where * Added preliminary minimal implementation of execute to DependencyProxy and SingleRowFetcher. Both untested and yet incomplete, they will only work for MVP. * added first implementation of count collect datarange produceRows function + test * Revert "added first implementation of count collect datarange produceRows function + test" This reverts commit 236825280bb5fd03dfe01febfc4dcf1a2d2e2c2f. * First draft of execute implementation, not yet handling ShadowRows though. This state is broken in the sense that a server does not boot with it and catch tests are broken * First working draft of execute() call in ExecutionBlockImpl. Tests are locally green, however there is at least one query still red (on ShadowRows) * Removed non finished implementation from this Branch. It moved to seperate branch * working - not done yet * Revert "working - not done yet" This reverts commit aa275b48ffc2c8fc9a3c364e2241321ec9c9293f. * first implementation of a ShadowRow fetching interface on AqlItemBlockInputRange * single row fetcher, execute + tests * Fixed SingleRowFetcherTest. Also an AqlItemBlockInputRange can now handle ShadowRows properly * Another fix on handling of shadow rows in InputRanges * Improved human readable output of SortLimit test, while validating the result * Removed debug profile execution of a query * Use simpleVPack printing for Trace of Execute, better to read by humans * Added tests for AqlItemBlockInputRange and adjusted implementation accordingly. * Fixed issue where ExecutionBlockImpl returns DONE, although there are still rows in the DataRange * Final modification of ShadowRows in FilterExecutor. All but profiling tests pass. This needs some discussion about overfetching of data, and propagation of hardLimit * Removed debug log includes * Moved responsibility for the client call into the output row. This can precisely count the elements it has written, and forms a single source of truth. The output row now works as the requested block by client that needs to be filled. Now the FilterExecutorTests are red. * Fixed FilterExecutorTest * Expose required features to AqlCall controlled by the output aql item row. Adapted executionBlockImpl to report there * Fixed bug with AqlCallStack pop operation that actually caused invalid memory access * outputRow now honors limits whan called for isFull. * Fixed isFull() in OutputAqlItemRow * Use the inbound call in skipRows for executors, in order to do a more precise upstream call * Make linker happy with operator< on CallLimit * Implement operator+ on AqlCall::Limit, also use std::visit for readability. * FilterExecutor getSome now propagates offset + min(limits) to upstream. This is pessimistic and avoids overfetching in all cases. NOTE: FilterExecutor should be moved to skipvariant using getSome * Fixed profiler tests for filter. These cannot be guaranteed anymore and it was an undesired, yet uncritical side effect that the tests have been green before * Fixed JSLint * Remove special casing for FilterExecutor * use if constexpr to distinguish between ported executors and non-ported executors enabling step-by-step porting and testing * Add skeleton skip code * Replace C&P accident * Simplify and implement skip for new executor interface * Fixup skipping for filter executor * Rename some methods for clarity and consistency * AqlItemBlockInputRange - hasMore is now hasDataRow - next is now nextDataRow - peek is now peekDataRow - state is now upstreamState - additional method upstreamHasMore * Introduce function to allocate an output block * Fix skipSome simulation bug * Some small cleanups * Replaced assertion on atMost on the output size. Otherwise we got into issues with atMost changes between waiting calls (can occur in our intermediate state now) * Attempt at fixing execute * Fix merge conflict * Feature/aql subquery execution block impl execute implementation exec block impl tests (#10838) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation shortest path executor (#10780) * Hook up new ShortestPathExecutor * Small fixes * Removed now unused code. * Moved ShortestPathExecutor tests to use Execute only. * Added a test for shortestPathExecutor where the output is smaller than the amount of entries within the path. * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Fixed Windows compile issue * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Windows compile fix * Attempt at fixing ShortestPathExecutorTest for Windows * Windows compile fix * Added a skip test and fixed the producing executor to recreate the list once for every input line. * Frankenstein my tests into the branch * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Hack * Removed designated initializers. Thanks for not supporting it MSVC! * Update ShortestPathExecutorTest * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Umbau 2.0 * Reshape output and skip a bit * Some test fixes * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Square 1 * Input validation * ShortestPathExecutor with fullCount and tests Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation exec block impl tests shadow rows (#10970) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Added tests for ExecutionBlockImpl, when there is a WAITING passed through. There is no test yet for in between waitings. * Allow to configure WAITING variant in WAITING block mock. * Use the WaitingBlockMock to be independent of Singleton in test * Added WAITING + Passthrough tests * Added a test to path through shadowRows without data. * Waiting ExecutionBlock Mock can now return/skip partial blocks. However it does NOT CARE for ShadowRows * Added tests and fixed an issue in ShadowRow forwarding * Added tests for shadowRow forwarding * Added the intenral state of Execute as member of ExecutionBlockImpl. We need to retain parts of this state accross returns * Fixed all Waiting and multiple call tests * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Fixed invalid memory access in test code. Do never return a lambda that references a stack variable... * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip * Allways -> Always. * Added tests for pass-by of non-relevant Subqueries. Fixed a bug found on the way in WaitingBlockMock. Now some other tests fail \o/ * Fixed failed tests. Now tests correspond to correct behaviour of WaitingExecutionBlockMock * Intermediate version, doe snot work * Fixed the empty subquery tests Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation sorted collect (#11020) * SortedCollect executor and some tests. * More tests. * More const and skip implemented. * Fixed some bugs found by tests. * Refactored skip. * Broken state with test framework. * Extracted testing framework into separate file and adjusted it to use WaitingBlockMock. However it is not yet fully implemented * Added another SortedExecutor test without fullcount. This passes. Also assert on skip and State * Removed obsolete assertion in skip, added another nonFullcount test. * More tests. Fixed some bugs in the test framework. * More cleanup. * Remove version control marker. * Removed wrong assertion. * Apply suggestions from code review Co-Authored-By: Michael Hackstein <michael@arangodb.com> * Remove dead code. * Removed fetcher. Co-authored-by: Michael Hackstein <michael@arangodb.com> * ReturnExecutor New style (#10831) * ReturnExecutor implemented. Tests misssing. * Apply suggestions from code review Co-Authored-By: Tobias Gödderz <tobias@arangodb.com> * Implemented skipRows in Return. Also fixed ExecutionBlock references in static asserts * Implemented first new style test for Return. * Moved the traceExecute calls into baseclass. * Added tracing around execute in WaitingExecutionBlockMock. Fixed a test bug, if the AqlCall covers all of the first block, but only a part of the Second block. You would only get the part of the second block. * Added more tests for standard behaviour of ReturnExecutor * Added an assertion. * Simplified call to ReturnExecutor * Prepare tests for doCount. Also added test for different input register. * Added accessor for _stats in Engine. * Included count tests in ReturnExecutorTest. * Removed old obsolete test. * Removed obsolete code-path * Fixed unrelated bug in FilterExecutor. In some cases it did lie on skipRows that it has more, even if it does not have and the upstream does not have as well. Thank you jenkins for hitting into this race :+1: Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Tweak ExecutorTestHelper to test pipelines * Revert "Tweak ExecutorTestHelper to test pipelines" This reverts commit 6359f0c926cad30b4efd7ce263fb3b64563621b8. * Added Stats return value to skipRowsRange (#11081) * Added Stats return value to skipRowsRange * Fixed catch test. Somehow i merged the incorrect part Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Additional Assertion in ExecutionBlockImpl (#11077) * Added a assertion arround skipRowsInRange that ensures that the Call and the skippedLocal do not get out-of-sync * Improved the assertion * Feature/aql subquery operations stack id executor (#10986) * added inputRange skip and produce * added inputRange skip and produce * Added shouldSkip helper to AqlCall * Added implementation if execute to Constfetcher. Not specifically tested yet. * Moved IDExecutor to newStyle executor * Added some documentation to SingleRowFetcher * Adapted IdExecutor tests. * Added an assertion in ConstFetcher * Fixed initialize cursor in ExecutionBlockImpl * Added tests for Initilaize cursor in IdExecutor. There is a special implementation for this... * Fixed linker issues, because of variables in cpp files named identical and not beeing static * Started refactoring the Scatter/Gather Executor logic. Added an intermediate class that does not use a template and can be casted to. This state compiles but will not work. * Made BlocksWithClientsImpl a template, based on an executor * Added a dummy implementation to fetch a new block. Distribute not yet implemented. * Added superoptimistic first implementation of ScatterExecutor return. Only works if there are no shadowRows. * Implemented logic to slice Blocks in Scatter according to the given call. Also added handfull of test for it, not complete though * Fixed a Scatter Executor handling of softLimit. Added another test to ansure diverse calls of clients to Scatter * Added random ordering test framework on ScatterExecutor * Added some tests for ScatterExecutor including subqueries * Removed todo, as it has been solved by now * Implemented Scatter by using IDExecutor. Tests not green yet * Implementation of ConstFetcher that allows to inject fullBlocks of AqlItemRows, and not only single rows. Also handles ShadowRows now in a correct way * Do not jump over relevant shadow rows in ConstFetcher. * Fixed tests. * Moved information of DistributeExecutor into the Infos. This resulted in adapting the Constructor API. I used this situation to do a big cleanup of ancient code. From this point there is no way back. * Implemented the last bits of DistributeExecutor * 'Implemented' execute by fallback to old API in RemoteExecutor * Fixed an EdgeCase with Scatter Executor, where the data was modified in place right after scatter. This has side-effects on peer executors. So we for now do a full copy of every block here. * Removed invalid assertions, also fixed RegisterInput to DistributeExecutor * Removed obsolete TODOs * Update arangod/Aql/IdExecutor.cpp Co-Authored-By: Lars Maier <lars@arangodb.com> Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Lars Maier <lars@arangodb.com> * Replaced the operator== and operator!= with `isSameBlockAndIndex`. (#11088) * New style IndexExecutor. (#11029) * New style IndexExecutor. * Small changes. * Fixing some logic for skipping. * Small changes. * TRI_ASSERT(FALSE); * Removed _fetcher member. (again) * Feature/aql subquery execution block impl execute implementation debug logging (#11112) * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * Feature/aql subquery operations stack hashed collect executor (#11103) * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * added inputRange skip and produce * added inputRange skip and produce * Added shouldSkip helper to AqlCall * Added implementation if execute to Constfetcher. Not specifically tested yet. * Moved IDExecutor to newStyle executor * Added some documentation to SingleRowFetcher * Adapted IdExecutor tests. * Added an assertion in ConstFetcher * Fixed initialize cursor in ExecutionBlockImpl * Added tests for Initilaize cursor in IdExecutor. There is a special implementation for this... * Fixed linker issues, because of variables in cpp files named identical and not beeing static * Started refactoring the Scatter/Gather Executor logic. Added an intermediate class that does not use a template and can be casted to. This state compiles but will not work. * Made BlocksWithClientsImpl a template, based on an executor * Added a dummy implementation to fetch a new block. Distribute not yet implemented. * Added superoptimistic first implementation of ScatterExecutor return. Only works if there are no shadowRows. * Implemented logic to slice Blocks in Scatter according to the given call. Also added handfull of test for it, not complete though * Fixed a Scatter Executor handling of softLimit. Added another test to ansure diverse calls of clients to Scatter * Added random ordering test framework on ScatterExecutor * Added some tests for ScatterExecutor including subqueries * Removed todo, as it has been solved by now * Implemented Scatter by using IDExecutor. Tests not green yet * Implementation of ConstFetcher that allows to inject fullBlocks of AqlItemRows, and not only single rows. Also handles ShadowRows now in a correct way * Do not jump over relevant shadow rows in ConstFetcher. * Fixed tests. * Moved information of DistributeExecutor into the Infos. This resulted in adapting the Constructor API. I used this situation to do a big cleanup of ancient code. From this point there is no way back. * Implemented the last bits of DistributeExecutor * 'Implemented' execute by fallback to old API in RemoteExecutor * Fixed an EdgeCase with Scatter Executor, where the data was modified in place right after scatter. This has side-effects on peer executors. So we for now do a full copy of every block here. * Removed invalid assertions, also fixed RegisterInput to DistributeExecutor * Removed obsolete TODOs * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * Made the branch compile again * Update arangod/Aql/IdExecutor.cpp Co-Authored-By: Lars Maier <lars@arangodb.com> * Intermediate commit. Empty might not compile * Added some external helper functions to compare ItemBlocks in different variants * Added a brief description for the bunch of input parameters in HashedCollect * Allow unordered output tests in TestHelper class. * Added first HashedCollectExecutor test using the helper * Allow to use never wait in getSome / skipSome variant of WaitingExecutionBlockMock. * Added more HashedCollect tests (multi group, count) and started with framework for AggregateMethods * Fixed simple Aggregate Test * Improved output for unordered Row in Output test. * Added test around collecting of Arrays and Objects * Also allow a fullCount call in the old Execute Wrapper * Almost Completed register and call behaviour test suite * Added a todo for a missing test * Improved setup of AqlTestCase framework, not only start the server once. This speeds up AQL test by at least a factor of 10. * implemented newStyle HashedCollect executor. * Cleanup of old code. * Added test cases where the upstream producer lies to HashedCollect. * Let HashedCollect and Impl life with the fact that upstream lies. * Improved gtest with multiple calls. * Simplify input consume logic of HashedCollect Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Lars Maier <lars@arangodb.com> * Feature/aql subquery operations stack enumerate list (#10988) * produce inputRange * produce inputRange test * produce inputRange test 2 * apply new api * enable new api * enumeratelistexecutor changes * added hackis shouldSkip method * add new execution style for enumerate lsit * added skip for enumerate list * edit of enumerate list executor * fixed enumlist executor * optimized skip loop * more tests, using new framework * allow empty output checks * added new style tests, removed old tests * removed old produceRows method * fixed tests * added one missing exit case * added another exit in SHADOWROWS mode, as we need to take a look at fullCount there * Revert "added another exit in SHADOWROWS mode, as we need to take a look at fullCount there" This reverts commit e74b7cede78f81b8fece034fc921a969e91dda12. * do not take care of fullCount in executor itself, impl will do * removed obsolete comment * rm _skipped because unused, return upstreamstate * added invalid value test, added border edge tests for softLimit, hardLimit and fullCount * api change * added test for inner states inside the executor * added new tests * fixed assert * removed two not needed ifs * asserted a case in our test helper * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * undo this commit before merge * undo this commit before merge * Merged Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation inject stack into fetcher (#11124) * Added api to pass through the stack to the dependencyProxy. THis is just temporary and doomed to be removed as soon as API is completed. * Let depdendency Proxy forward the Stack. This may not work in all cases. * Default implement Assignment constructor. * Removed Query jenkins debug output again * Fixed forwarding of stack in skipSome * Tweak ExecutorTestHelper to test pipelines (#11079) * Tweak ExecutorTestHelper to test pipelines * Modifications * Fixup tests * Address some comments * Fix. * AQL execute in LimitExecutor (#10886) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Fixed merge * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Reverted accidental change * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Assert no input rows in skip * Began writing a test framework for LimitExecutor * Removed unused code * Continued test framework * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip * Make AqlItemBlockInputRange hold a skipped count * Added == for LimitStats, moved code to .cpp file * Added Stats return value to skipRowsRange * WIP: Implement LimitExecutor test framework * Fixed namespace * Added missing advanceRow call * Added parametrized test cases * Dump * Made test output more readable * Fixed merge conflicts * Set engine in createFakeQuery() * [WIP] Dump: not working * Fixed merge conflicts * Switch from boost::variant to std::variant * Fix comparison of differently sized blocks * Fixed usage of temporary * Fixed skip handling in ExecutionBlockImpl * Add upstream fullCount in LimitExecutor * Fix assertion that was too narrow * Again, fixed skip handling in ExecutionBlockImpl * Extended ExecutorTestHelper, mainly a loop to handle passthrough Executors * Fixed expected skipped count * Fixed bugs in LimitExecutor * Another bugfix in LimitExecutor * Fixed expected output calculation * Removed some comments * Improve test speed by not recreating MockAqlServer for every case * Allow additional variants in ExecutorTestHelper * Tried to fix a bug in LimitExecutor, not quite working * A little cleanup * Toggle lieAboutHasMore * Reverted broken fix, fixed WaitingExecutionBlockMock * Fixed merge conflicts * Add LimitStats to expected checks in tests * Fixed another leftover merge conflict * Fixed fullCount stats * Removed old code from LimitExecutor * Updated ExecutionBlockImpl to allow removal of old executor methods * Removed old tests * Simplified test code * Added comments and constructors to AqlCall * Instantiate test cases with multiple different calls * Fixed a bug in run(), made looping optional * Fixed a bug in DependencyProxy * Fixed a bug and an assertion in LimitExecutor * Added assertions and a fix in WaitingExecutionBlockMock * Enabled stats tests again * Made the executor compile under Mac * Let AqlHelper compare profiled stats as well * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * Modified skip logic in LimitExecutor to simplify what has to be reported to downStream and what is handled by internal skip. Also allow to do skip and fullCount in one go, if limit is 0 * Adapted the Expected skipped output in Limit test * Forward fullCount from client through LIMIT. Simplyfied produceLogic. * Reduced the amount of Tests in LimitExecutor. Need to decide on a suitable subset * Removed lieAboutHasMore by simply adding an empty block to the end that is blocking return in the WaitingExecutor * Fixed SingleRowFetcher to allow a Range with only skip information. * Intermed version of ExecutionBlockImpl. Needs to be continued * Allow hardLimit to pass getSome check * Intermed commit * Unified FastForward and FullCount code path * Produce and Skip go to FastForward if the Executor reports done and the output row is full. This way we can handle more FUllCount cases straight away. * Fixed the expected value for FullCount stats in test. This is rather awkward to automatically compute... * Fixed lying of the ExecutionBlockMock. Did not lie correctly if the limit was reached by softLimit. * Make the rules for supporting the OldAPI more relaxed. * Added a missmatch of outputColumns and internal output columns in TestHelper * Fixed AqlItemBlockRangeTests * Implemented unreachable code, otherwise our Jenkins does not compile it ;( * Fixed SortedCollect, it might count the last group twice on fullCount, if produce ends exactly in this group * Use AqlTestCase in ReturnExecutorTest * Return Executor needs to report skipped from upstream properly * Fixed fullCount call behaviour * Fixed expected numberSkipped in LimitExecutor test * Added some DEBUG level log messages in FastForward * Fixed reporting of skip in skipSome case and removed dead code * Fixed Impl test, we cannot let the WaitingBlockMock do counting on ShadowRows, it is not implemented there. * Removed unused function * Pointeless comment * Fixed skipSome of Filter * Avoid endless Loop in indexExecutor fullCount, where it tried to skipped 0 documents forever * Fixed return state of IndexExecutor * Fake DONE on hardLimit for oldAPI, apply only on toplevel. Improve shortcutting behviour to jump into FastForward everytime HARDLimit is reached, even if output is full. * Added an assertion that multiple UPSTREAM call rounds do not mess up the callStack by leaving the _upstreamCall on top. Now the executor itself pops off the call, this is subject to change if we manage the Stack differently * Was erlaube skip? Adjsuted tests to honor that we can do skip and limit now in one go, for all executors that are updated already * Fixed ConstFetcher, to not modify the CallStack, it only has a reference, where the other Fetchers copy it to the Executors, and modify it there * Do not overfetch on passthrough if the block is full. * We cannot create DONE on oldStyle if we need fullcount... * Unfortunately we teporarily need more calls in fullCount case, as long as one of the Executors above is not yet on execute * Mhm gcc and clang do not aggree on which types are equal... * Fixed compile issue. Co-authored-by: Michael Hackstein <michael@arangodb.com> Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> * Feature/aql subquery operations stack calculation exec enumerate exec (#11098) * removed unused import * re-ordered asserts to see which register was wrongly used * added implementation of produceRows with inputRange * added first test to test new produceRows including inputRange * rm skip from calc executor * rm not correct assert, added second test * added produceRows inputrange hasmore test * added new produceRows + test * added skipRowsRange enumerate collection executor * added skipRowsRange enumerate collection executor test * added assert * adde fake query helper, one without auto trx but with prepare plan, the other one without prepare but with trx - also removed fakeit mocks from enumeratecollectionexecutor tests * fixed skipRowsRange in enumerateCollectionExecutor, added test as well * removed todo * added skipAll * added EnumerateCollection as new style executor * added produce and skip rows, not ready yet * added new tests, still not ready, need unsorted compare operators in helper, waiting * rm logs * intermidiate commit - may not work * removed unused import * added implementation of produceRows with inputRange * added first test to test new produceRows including inputRange * rm skip from calc executor * rm not correct assert, added second test * added produceRows inputrange hasmore test * added new produceRows + test * added skipRowsRange enumerate collection executor * added skipRowsRange enumerate collection executor test * added assert * adde fake query helper, one without auto trx but with prepare plan, the other one without prepare but with trx - also removed fakeit mocks from enumeratecollectionexecutor tests * fixed skipRowsRange in enumerateCollectionExecutor, added test as well * removed todo * added skipAll * added EnumerateCollection as new style executor * added produce and skip rows, not ready yet * added new tests, still not ready, need unsorted compare operators in helper, waiting * rm logs * intermidiate commit - may not work * make branch compile again * mv assert * protected server * make enum collection tests work again * fixed ExecBlockImpl * todo optimization * fix gtest calc executor * header file enumcollexec, helper method * fix shrink method in aqlitemblock * impl * nodiscard * calc executor * rm empty line * fix calc exec * tmp re-enable old method * tmp make tests green again, needs cleanup later * use uint64_t instead of u_int64_t * removed unused code * Update arangod/Aql/CalculationExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * Update arangod/Aql/CalculationExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * remove implementation code of old api * Update arangod/Aql/EnumerateCollectionExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * buildable tests again * rm unused code * added assert * make tests compile again ... * disable few more tests, need to be optimized after subquery end start merges Co-authored-by: Michael Hackstein <michael@arangodb.com> * Quickfix for compilation error due to merge * Feature/aql subquery execution block impl execute implementation k shortest path executor (#10940) * Add new produceRow API code for KShortestPathsExecutor * Add KShortestPathsExecutor test to build * Add KShortestPathsExecutor test * Test and bugfix for KShortestPathExecutor * Update k-shortest-paths executor for new interface * Update tests for k-shortest-paths executor * Make ShortestPathExecutor and KShortestPathExecutor more similar * Fix merge messup * Refactor KShortestPathsExecutorInfos * Move KShortestPathsExecutor to new executor interface * Fix tests * Resolve naming conflict * Address some review comments * Fix skip * Fix for new signatures * Fix KShortestPathsExecutor * Fix KShortestPaths regression Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation traversal executor (#11023) * Move to new executor interface * Fix tests and some bugs * new skip signature * new skip signature * Fix Traversal Executor Skipping * Fix compiler complaint Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move SubqueryStartExecutor to new interface (#11025) * Move SubqueryStartExecutor to new interface * Diddle * Tests pass for SubqueryStartExecutor * Delete most of the SubqueryEndExecutor tests They are irrelevant now as they only tested shadowrow behaviour. Shadow Rows are now handled by ExecutionBlockImpl. * Skeleton tests * Junk * Cleanup and make work * Tweak ExecutorTestHelper to test pipelines * SubqueryStart/SubqueryEnd * Blafummel. * Modifications * Fixup tests * Address some comments * Fix. * Fix tests * Various fixes. * Made the branch compile on Mac * Intermediate commit, does not compile, required for merge * Fixed one of the SplicedSubqueryTests * Seperated ShadowRow handling for SubqueryStart/End executors to get a better overview and understanding on how it should behave. * Allow to use a fullCallStack in ExecutorTestHelper. * Added asserts on SubqueryEnd that is not allowed to be called with Skipped from upstream * Use AqlTestCase in SubqueryStartExecutorTest. Transformed first test to TestHelper pipeline * Let ExecutorTestHelper also test for shadowRows * Adapted SingleInput=>Data+Shadow test to testing Helepr and make it pass * Transformed existing SubqueryStart tests to Helper, fixed a test where there was no place for the ShadowRow to be written * Fixed Skip in SubqueryStartExecutor * Allow the Pipeline to add Consumer and dependency without fiddling with the Deque from outside * Added Skip adn ShadowRowForwarding tests to SubqueryStartExecutor * Fixed fullCount of SubqueryStart to be called properly. * Renamed isModificationSubquery -> isModificatioNode again * Consume more than one subquery en-block * Fixed debugReporting in SplicedSubqueryTest. Disabled the skip in subquery tests for now * Removed AQL debug log level from testing config * First version of Subquery Skip bypassing. Including 1 passing test. * Added disabled tests for skip, and proposed to do it later * Fixed reporting of FILTERED in fullCount of EnumerateCollectionExecutor * Enable calculationExecutor execute interface * Fixed compile issues * To Velocypack the modification Subquery value * Fixed illegal assert and reactivated failure tests * Fixed compile issue * Deactivated Calculation again * Added Compatibility Mode to AqlCallStack * Removed IdExecutor<void> the only feature it has can be moved into IDexecutor in general. This still has to be done * When falling back in BlocksWithClients use compatibility stack * ConstFetcher now correctly reports if it skipped all input * Fixed a bug in ExecutionBlockImpl, which would go into FetchMore state, even if the Executor still has data and is happy to produce more, but the output is full. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute remote node api flag (#11159) * Clarified comment * Added api property to RemoteNode * Added missing include * Feature/aql subquery operations stack i research view executor (#11140) * added arangosearch view executor to new style * added new produceRows and skipRowsRange * ires * test jenkins * modify aql call * cleanup branch * Update arangod/Aql/IResearchViewExecutor.cpp Co-Authored-By: Dronplane <andrei.lobov@arangodb.com> * applied suggested changes * also enable IResearchViewMergeExecutor * Fixed an issue in skipRange, where the input was not moved forward in all cases. * Removed unused local variables Co-authored-by: Dronplane <andrei.lobov@arangodb.com> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move SingleRemoteModificationExecutor to new interface (#11166) * Move SingleRemoteModificationExecutor to new interface * Activate SingleRemoteModificationExecutor * Inspect the result of remote modification Co-authored-by: Michael Hackstein <michael@arangodb.com> * New style DistinctCollect (#11096) * First implementation of new style executor for DistinctCollect. * Tests for distinct collect. * Remove debug output. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation noresults (#11180) * Use ExecutorHelper in tests * Implemented no Results fetcher * Moved Constrained sort to execute API (#11173) * Moved Constrained sort to execute API * The LIMIT in ConstrainedSort can actually fetch TOO much, if there is too small input from upstream * Improved profiler tests, we need less calls now \o/ * Fixed unlimited call to limit executor * Feature/aql subquery operations stack aql item block input matrix (#11167) * added new file to cmakelist * added new method skipAllRemainingDataRows * added first implementation of execute - AllRowsFetcher * impl is now using skipAllRemainingDataRows method, also execute methods now using Fetcher::DataRange instead of inputRange * impl is now using skipAllRemainingDataRows method, also execute methods now using Fetcher::DataRange instead of inputRange - header file * AllRowsFetcher now using AqlItemBlockInputMatrix as DataRange, first implementation of execute * first implementation of AqlItemBlockInputMatrix * more cleanup * optimized execut elogic * optimized logic in inputmatrix, still a few todos left * added SortExecutor as newStyle * added produceRows and skipRows * logs (to be removed), skipAllRemainingRows * logs * inputMatrix * changed return type of skipAllRemainingDataRows * impl logs * logs, added clear method * sort test * typo, log * latest changes * cleanup, assert * rm logs * added more todos * todo * new method names, compile again * Reset ExecutorReturnedDone for SubqueryStart and SubqueryEnd. * InputMatrix needs to return state DONE on every input request. It guarantees that we have all input * Removed invalid Assertion, and removed early return if output is full. * Removed DebugLogging from CalculationTest * Fixed skipping in SortExecutor * Fixed upstreamState return of AqlItemMatrix. It is only finalState iff there is no data * Let the InputMatrix drop all rows after FullCount * Allow bypassing with hardLimit 0 in InputMatrix * Disabled the old-code assertion. This way the SortExecutor tests we have will be red, but not crash, allowing the jenkins to produce meaningfull results * Fix assert in WAITING case * Add rudimentary sort executor GTests * Fixed assertion on InputMatrix that starts right away with a shadowRow * Removed declared but not implemented function * changed from: peekDataRowAndState -> to: peekDataRow Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute in restaqlhandler (#11175) * Updated comment * Added two static strings * Avoid unintentional streaming operator of string_view as a container * Added parser for execute rest call * Fixed ambiguous overload in ResultT * Added execute and executeForClient to ExecutionEngine * Implemented executeForClient in RestAqlHandler * Add default value * Updated comment * Changed error attribute * Fixed duplicate log id * Try to make MSVC work * Moved AqlExecuteResult to a separate file * Tried to fix MSVC compile error * Revert "Tried to fix MSVC compile error" This reverts commit f6f43b2c7e4bfe17dafe65f48bcd5c0b6e5c69a9. * Next try to fix MSVC * Revert "Next try to fix MSVC" This reverts commit 528c4c795d10ee0aa6686d143be1f6faa6e9b553. * Revert "Try to make MSVC work" This reverts commit ba7d9c072fccb969da1cdf045eadf49c297da8e9. * Work around MSVC shortcoming * Support executors with multiple dependencies (#11181) * Use DataRange in some places * Add MultiAqlItemBlockInputRange * First implementation of life, universe, and everything * implement skip for UnsortedGather * Some more wiring * Attempt at hacking up execute for DependencyProxy * Compilation fixes! * Fixed AqlCall compilation and unified execute and executeForDependency * Fixed WAITIGN assertion * Sonderlocken. * Foo * Make compile * Make tests pass. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move ModificationExecutors to new interface (#11165) * Move ModificationExecutors to new interface * added inputRange method to inputMatrix class, added switch between DataRange Input <-> Matrix for ModificationExecutors * forgot to enable allrows modification executor in fastForwardType * removed obsolete comment * removed obsolete comment Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery operations stack materialize executor (#11192) * removed obsolete comment, added produceRows and skipRowsRange, enabled MaterializeExecutor as newStyleExecutor * forgot to use advanceRow, fetch input during skipRowsRange * removed old implementations * removed comment, fixed a logical loop error * skip could be simplified as well * temp workaround Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute remote (#11197) * Clarified comment * Updated comment * Added two static strings * Avoid unintentional streaming operator of string_view as a container * Added parser for execute rest call * Fixed ambiguous overload in ResultT * Added execute and executeForClient to ExecutionEngine * Implemented executeForClient in RestAqlHandler * Add default value * Updated comment * Changed error attribute * Added API toggle to RemoteExecutor * Fixed compile error * Fixed duplicate log id * Try to make MSVC work * Moved AqlExecuteResult to a separate file * Moved AqlExecuteResult to a separate file * Implemtented execute in Remote; except for (de)serialization, which is still missing * Tried to fix MSVC compile error * Revert "Tried to fix MSVC compile error" This reverts commit f6f43b2c7e4bfe17dafe65f48bcd5c0b6e5c69a9. * Next try to fix MSVC * Implemented (de)serialization for RemoteExecutor::execute * Fixes * Added minimal AqlCall (de)serialization tests * Bugfix * Added minimal (de)serialization tests for AqlCallStack and AqlExecuteResult * Fixed test * Changed Serialization format a little * Bugfix * Initialize SingletonBlock immediately with an empty row * Revert "Next try to fix MSVC" This reverts commit 528c4c795d10ee0aa6686d143be1f6faa6e9b553. * Revert "Try to make MSVC work" This reverts commit ba7d9c072fccb969da1cdf045eadf49c297da8e9. * Work around MSVC shortcoming * Work around MSVC shortcoming * Attempt to fix windows compile issue Co-authored-by: Michael Hackstein <michael@arangodb.com> * Added test for distinct collect with random order. (#11184) * Feature/aql subquery execution block impl execute implementation subquery executor (#11196) * Intermediate commit, does not compile. * New style for SubqueryExecutor. * Removed bad assert. * Reset subsqueryInitialised in skip. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation count collect (#11172) * Move SubqueryStartExecutor to new interface * Diddle * Tests pass for SubqueryStartExecutor * Delete most of the SubqueryEndExecutor tests They are irrelevant now as they only tested shadowrow behaviour. Shadow Rows are now handled by ExecutionBlockImpl. * Skeleton tests * Junk * Cleanup and make work * Tweak ExecutorTestHelper to test pipelines * SubqueryStart/SubqueryEnd * Blafummel. * Modifications * Fixup tests * Address some comments * Fix. * Fix tests * Various fixes. * Made the branch compile on Mac * Intermediate commit, does not compile, required for merge * Fixed one of the SplicedSubqueryTests * Seperated ShadowRow handling for SubqueryStart/End executors to get a better overview and understanding on how it should behave. * Allow to use a fullCallStack in ExecutorTestHelper. * Added asserts on SubqueryEnd that is not allowed to be called with Skipped from upstream * Use AqlTestCase in SubqueryStartExecutorTest. Transformed first test to TestHelper pipeline * Let ExecutorTestHelper also test for shadowRows * Adapted SingleInput=>Data+Shadow test to testing Helepr and make it pass * Transformed existing SubqueryStart tests to Helper, fixed a test where there was no place for the ShadowRow to be written * Fixed Skip in SubqueryStartExecutor * Allow the Pipeline to add Consumer and dependency without fiddling with the Deque from outside * Added Skip adn ShadowRowForwarding tests to SubqueryStartExecutor * Fixed fullCount of SubqueryStart to be called properly. * Renamed isModificationSubquery -> isModificatioNode again * Consume more than one subquery en-block * Fixed debugReporting in SplicedSubqueryTest. Disabled the skip in subquery tests for now * Removed AQL debug log level from testing config * First version of Subquery Skip bypassing. Including 1 passing test. * Added disabled tests for skip, and proposed to do it later * Fixed reporting of FILTERED in fullCount of EnumerateCollectionExecutor * Enable calculationExecutor execute interface * Fixed compile issues * To Velocypack the modification Subquery value * Fixed illegal assert and reactivated failure tests * Fixed compile issue * Deactivated Calculation again * Added a testSuite for countCollectExecutor * Fixed Subquery Test for CountCollect * Added CountCollectImplementation * Added Compatibility Mode to AqlCallStack * Removed IdExecutor<void> the only feature it has can be moved into IDexecutor in general. This still has to be done * When falling back in BlocksWithClients use compatibility stack * Fixed EnumerateCollectionExecutor * Fixed fullCountBehaviour of EnumerateList. Also removed a unlimited,fullCount combination in IndexExecutor * Fixed endless loop while skipAl in OperationCursor * Fixed View Executor, which sends a no-limit no-offset upstream. * Fixed forwarding in NoResultsExecutor Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> * Activate CalculationExecutors (#11177) * Fixed compile bug-due to hidden merge conflict * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Started removing old getSome/skipSome API. This version does not compile yet * Harvested more, however the old tests still exist and do not compile now. * Feature/aql subquery execution block impl execute implementation base bugfixes (#11201) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Fix memory leak in traversal (#11210) * AqlSubqueryExecutionBlockImplExecuteImplementation use moveValueInto instead of cloneValueInto (#11213) * Fix memory leak in traversal * Use move instead of clone for AqlValues * Make compile * Feature/aql subquery execution block impl execute implementation gather executors (#11202) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Feature/aql subquery execute parallel gather (#11215) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute parallel gather 2 (#11221) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather * [WIP] Changed Api for MultiDepExecutors in ExecBlockImpl (not yet in the actual executors) * Moved AqlCallSet into a separate file * Changed SortingGather to use the new API * Changed ParallelUnsortedGather to use the new API * Changed UnsortedGather to use the new API * Moved AqlCall operator<< into .cpp file * Implement operator<< for AqlCallSet * Fix boolean mix-up * Fixed state machine: go to UPSTREAM when the AqlCallSet is not empty * Fixed assertion * Bugfix * SortingGather bugfixes * Added init() method to fix an assertion and cleanup * Removed unused variable * Fixed constrained sort * Fixed constrained sort #2 * Fix boolean mix-up * Remove old interface * Use call parameter for upstream request in produceRows * Remove more old interface code * Add skip methods to MultiAqlItemBlockInputRange * Skip in UnsortedGather * skip for UnsortedGather * Fix skip and upstream calls for UnsortedGather * skipRowsRange change * Remove useless comments * Moved multi-dep related code from ExeBlockImpl to MultiFetcher * Cleanup in SortingGather, implemented parallel fullCount * Try to fix a windows compile error * Simplify and extend skipRowsRange for UnsortedGatherExecutor * Made ParallelUnsortedGather actually parallel * Removed erroneous assertion * Undid erroneous change * Fixed MacOs compile. Also disabled tests for non-relevant AqlCallStacks. They will be removed * Fixed initialize Cursor for multi dependency blocks * Fixed fullCount case in parallel unsorted gather * Fixed fullCount upstream call of ParallelUnsortedGatherExecutor * Fixed fullCount in SortingGather * Windows \o/ if you cannot work properly with constexpr and static asserts, we do not let you do it! * Do not advance in Unsorted gather if there are still rows to skip * Add more comparison operators for AqlCall limits * Send clientCall limits to upstream in SortingGather * Improved fullCount in SortingGatherExectur * Disabled a cluster profile test. We now ask the RemoteNode more often if it already has data. It is a bit unclear to me if this is now better performance wise (<< i think so) or triggers undesired side effects * Helpless attempt to work around issues in stonage Visual Studio Compiler we are using. * Clearly adding an operator on a well defined type causes ambigousness on random basic types using the operator Co-authored-by: Michael Hackstein <michael@arangodb.com> Co-authored-by: Markus Pfeiffer <markus@arangodb.com> * Feature/aql subquery execution block impl execute implementation bypass skip (#11203) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of P…
mchacki
added a commit
that referenced
this issue
Apr 8, 2020
…esting (#11349) * Added first draft of AqlCallObject * Added AqlItemBlockInputRange * Added Header file for AqlCallStack with comments. No implementation yet. * Added AqlCallStack first draft implementation. Made everything compile * Added empty dummy stubs for execute() in ExecutionBlock * Implementation of new API in FilterExecutor. Rough implementation in ExecutionBlockImpl (not complete, only skip path) everything compiles, but is not tested * Fixed off by one error in InputRange. Fixed FilterExecutor * Added some more tests if executor cannot fulfill atMost * Added heplper functions to the AqlCall struct to avoid mangeling around with variants everywhere. * Add a member DataRange to ExecutionBlockImpl. Also now all Fetcher expose a DataRange, this might not be the correct one every where * Added preliminary minimal implementation of execute to DependencyProxy and SingleRowFetcher. Both untested and yet incomplete, they will only work for MVP. * added first implementation of count collect datarange produceRows function + test * Revert "added first implementation of count collect datarange produceRows function + test" This reverts commit 236825280bb5fd03dfe01febfc4dcf1a2d2e2c2f. * First draft of execute implementation, not yet handling ShadowRows though. This state is broken in the sense that a server does not boot with it and catch tests are broken * First working draft of execute() call in ExecutionBlockImpl. Tests are locally green, however there is at least one query still red (on ShadowRows) * Removed non finished implementation from this Branch. It moved to seperate branch * working - not done yet * Revert "working - not done yet" This reverts commit aa275b48ffc2c8fc9a3c364e2241321ec9c9293f. * first implementation of a ShadowRow fetching interface on AqlItemBlockInputRange * single row fetcher, execute + tests * Fixed SingleRowFetcherTest. Also an AqlItemBlockInputRange can now handle ShadowRows properly * Another fix on handling of shadow rows in InputRanges * Improved human readable output of SortLimit test, while validating the result * Removed debug profile execution of a query * Use simpleVPack printing for Trace of Execute, better to read by humans * Added tests for AqlItemBlockInputRange and adjusted implementation accordingly. * Fixed issue where ExecutionBlockImpl returns DONE, although there are still rows in the DataRange * Final modification of ShadowRows in FilterExecutor. All but profiling tests pass. This needs some discussion about overfetching of data, and propagation of hardLimit * Removed debug log includes * Moved responsibility for the client call into the output row. This can precisely count the elements it has written, and forms a single source of truth. The output row now works as the requested block by client that needs to be filled. Now the FilterExecutorTests are red. * Fixed FilterExecutorTest * Expose required features to AqlCall controlled by the output aql item row. Adapted executionBlockImpl to report there * Fixed bug with AqlCallStack pop operation that actually caused invalid memory access * outputRow now honors limits whan called for isFull. * Fixed isFull() in OutputAqlItemRow * Use the inbound call in skipRows for executors, in order to do a more precise upstream call * Make linker happy with operator< on CallLimit * Implement operator+ on AqlCall::Limit, also use std::visit for readability. * FilterExecutor getSome now propagates offset + min(limits) to upstream. This is pessimistic and avoids overfetching in all cases. NOTE: FilterExecutor should be moved to skipvariant using getSome * Fixed profiler tests for filter. These cannot be guaranteed anymore and it was an undesired, yet uncritical side effect that the tests have been green before * Fixed JSLint * Remove special casing for FilterExecutor * use if constexpr to distinguish between ported executors and non-ported executors enabling step-by-step porting and testing * Add skeleton skip code * Replace C&P accident * Simplify and implement skip for new executor interface * Fixup skipping for filter executor * Rename some methods for clarity and consistency * AqlItemBlockInputRange - hasMore is now hasDataRow - next is now nextDataRow - peek is now peekDataRow - state is now upstreamState - additional method upstreamHasMore * Introduce function to allocate an output block * Fix skipSome simulation bug * Some small cleanups * Replaced assertion on atMost on the output size. Otherwise we got into issues with atMost changes between waiting calls (can occur in our intermediate state now) * Attempt at fixing execute * Fix merge conflict * Feature/aql subquery execution block impl execute implementation exec block impl tests (#10838) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation shortest path executor (#10780) * Hook up new ShortestPathExecutor * Small fixes * Removed now unused code. * Moved ShortestPathExecutor tests to use Execute only. * Added a test for shortestPathExecutor where the output is smaller than the amount of entries within the path. * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Fixed Windows compile issue * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Windows compile fix * Attempt at fixing ShortestPathExecutorTest for Windows * Windows compile fix * Added a skip test and fixed the producing executor to recreate the list once for every input line. * Frankenstein my tests into the branch * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Hack * Removed designated initializers. Thanks for not supporting it MSVC! * Update ShortestPathExecutorTest * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Umbau 2.0 * Reshape output and skip a bit * Some test fixes * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Square 1 * Input validation * ShortestPathExecutor with fullCount and tests Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation exec block impl tests shadow rows (#10970) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Added tests for ExecutionBlockImpl, when there is a WAITING passed through. There is no test yet for in between waitings. * Allow to configure WAITING variant in WAITING block mock. * Use the WaitingBlockMock to be independent of Singleton in test * Added WAITING + Passthrough tests * Added a test to path through shadowRows without data. * Waiting ExecutionBlock Mock can now return/skip partial blocks. However it does NOT CARE for ShadowRows * Added tests and fixed an issue in ShadowRow forwarding * Added tests for shadowRow forwarding * Added the intenral state of Execute as member of ExecutionBlockImpl. We need to retain parts of this state accross returns * Fixed all Waiting and multiple call tests * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Fixed invalid memory access in test code. Do never return a lambda that references a stack variable... * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip * Allways -> Always. * Added tests for pass-by of non-relevant Subqueries. Fixed a bug found on the way in WaitingBlockMock. Now some other tests fail \o/ * Fixed failed tests. Now tests correspond to correct behaviour of WaitingExecutionBlockMock * Intermediate version, doe snot work * Fixed the empty subquery tests Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation sorted collect (#11020) * SortedCollect executor and some tests. * More tests. * More const and skip implemented. * Fixed some bugs found by tests. * Refactored skip. * Broken state with test framework. * Extracted testing framework into separate file and adjusted it to use WaitingBlockMock. However it is not yet fully implemented * Added another SortedExecutor test without fullcount. This passes. Also assert on skip and State * Removed obsolete assertion in skip, added another nonFullcount test. * More tests. Fixed some bugs in the test framework. * More cleanup. * Remove version control marker. * Removed wrong assertion. * Apply suggestions from code review Co-Authored-By: Michael Hackstein <michael@arangodb.com> * Remove dead code. * Removed fetcher. Co-authored-by: Michael Hackstein <michael@arangodb.com> * ReturnExecutor New style (#10831) * ReturnExecutor implemented. Tests misssing. * Apply suggestions from code review Co-Authored-By: Tobias Gödderz <tobias@arangodb.com> * Implemented skipRows in Return. Also fixed ExecutionBlock references in static asserts * Implemented first new style test for Return. * Moved the traceExecute calls into baseclass. * Added tracing around execute in WaitingExecutionBlockMock. Fixed a test bug, if the AqlCall covers all of the first block, but only a part of the Second block. You would only get the part of the second block. * Added more tests for standard behaviour of ReturnExecutor * Added an assertion. * Simplified call to ReturnExecutor * Prepare tests for doCount. Also added test for different input register. * Added accessor for _stats in Engine. * Included count tests in ReturnExecutorTest. * Removed old obsolete test. * Removed obsolete code-path * Fixed unrelated bug in FilterExecutor. In some cases it did lie on skipRows that it has more, even if it does not have and the upstream does not have as well. Thank you jenkins for hitting into this race :+1: Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Tweak ExecutorTestHelper to test pipelines * Revert "Tweak ExecutorTestHelper to test pipelines" This reverts commit 6359f0c926cad30b4efd7ce263fb3b64563621b8. * Added Stats return value to skipRowsRange (#11081) * Added Stats return value to skipRowsRange * Fixed catch test. Somehow i merged the incorrect part Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Additional Assertion in ExecutionBlockImpl (#11077) * Added a assertion arround skipRowsInRange that ensures that the Call and the skippedLocal do not get out-of-sync * Improved the assertion * Feature/aql subquery operations stack id executor (#10986) * added inputRange skip and produce * added inputRange skip and produce * Added shouldSkip helper to AqlCall * Added implementation if execute to Constfetcher. Not specifically tested yet. * Moved IDExecutor to newStyle executor * Added some documentation to SingleRowFetcher * Adapted IdExecutor tests. * Added an assertion in ConstFetcher * Fixed initialize cursor in ExecutionBlockImpl * Added tests for Initilaize cursor in IdExecutor. There is a special implementation for this... * Fixed linker issues, because of variables in cpp files named identical and not beeing static * Started refactoring the Scatter/Gather Executor logic. Added an intermediate class that does not use a template and can be casted to. This state compiles but will not work. * Made BlocksWithClientsImpl a template, based on an executor * Added a dummy implementation to fetch a new block. Distribute not yet implemented. * Added superoptimistic first implementation of ScatterExecutor return. Only works if there are no shadowRows. * Implemented logic to slice Blocks in Scatter according to the given call. Also added handfull of test for it, not complete though * Fixed a Scatter Executor handling of softLimit. Added another test to ansure diverse calls of clients to Scatter * Added random ordering test framework on ScatterExecutor * Added some tests for ScatterExecutor including subqueries * Removed todo, as it has been solved by now * Implemented Scatter by using IDExecutor. Tests not green yet * Implementation of ConstFetcher that allows to inject fullBlocks of AqlItemRows, and not only single rows. Also handles ShadowRows now in a correct way * Do not jump over relevant shadow rows in ConstFetcher. * Fixed tests. * Moved information of DistributeExecutor into the Infos. This resulted in adapting the Constructor API. I used this situation to do a big cleanup of ancient code. From this point there is no way back. * Implemented the last bits of DistributeExecutor * 'Implemented' execute by fallback to old API in RemoteExecutor * Fixed an EdgeCase with Scatter Executor, where the data was modified in place right after scatter. This has side-effects on peer executors. So we for now do a full copy of every block here. * Removed invalid assertions, also fixed RegisterInput to DistributeExecutor * Removed obsolete TODOs * Update arangod/Aql/IdExecutor.cpp Co-Authored-By: Lars Maier <lars@arangodb.com> Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Lars Maier <lars@arangodb.com> * Replaced the operator== and operator!= with `isSameBlockAndIndex`. (#11088) * New style IndexExecutor. (#11029) * New style IndexExecutor. * Small changes. * Fixing some logic for skipping. * Small changes. * TRI_ASSERT(FALSE); * Removed _fetcher member. (again) * Feature/aql subquery execution block impl execute implementation debug logging (#11112) * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * Feature/aql subquery operations stack hashed collect executor (#11103) * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * added inputRange skip and produce * added inputRange skip and produce * Added shouldSkip helper to AqlCall * Added implementation if execute to Constfetcher. Not specifically tested yet. * Moved IDExecutor to newStyle executor * Added some documentation to SingleRowFetcher * Adapted IdExecutor tests. * Added an assertion in ConstFetcher * Fixed initialize cursor in ExecutionBlockImpl * Added tests for Initilaize cursor in IdExecutor. There is a special implementation for this... * Fixed linker issues, because of variables in cpp files named identical and not beeing static * Started refactoring the Scatter/Gather Executor logic. Added an intermediate class that does not use a template and can be casted to. This state compiles but will not work. * Made BlocksWithClientsImpl a template, based on an executor * Added a dummy implementation to fetch a new block. Distribute not yet implemented. * Added superoptimistic first implementation of ScatterExecutor return. Only works if there are no shadowRows. * Implemented logic to slice Blocks in Scatter according to the given call. Also added handfull of test for it, not complete though * Fixed a Scatter Executor handling of softLimit. Added another test to ansure diverse calls of clients to Scatter * Added random ordering test framework on ScatterExecutor * Added some tests for ScatterExecutor including subqueries * Removed todo, as it has been solved by now * Implemented Scatter by using IDExecutor. Tests not green yet * Implementation of ConstFetcher that allows to inject fullBlocks of AqlItemRows, and not only single rows. Also handles ShadowRows now in a correct way * Do not jump over relevant shadow rows in ConstFetcher. * Fixed tests. * Moved information of DistributeExecutor into the Infos. This resulted in adapting the Constructor API. I used this situation to do a big cleanup of ancient code. From this point there is no way back. * Implemented the last bits of DistributeExecutor * 'Implemented' execute by fallback to old API in RemoteExecutor * Fixed an EdgeCase with Scatter Executor, where the data was modified in place right after scatter. This has side-effects on peer executors. So we for now do a full copy of every block here. * Removed invalid assertions, also fixed RegisterInput to DistributeExecutor * Removed obsolete TODOs * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * Made the branch compile again * Update arangod/Aql/IdExecutor.cpp Co-Authored-By: Lars Maier <lars@arangodb.com> * Intermediate commit. Empty might not compile * Added some external helper functions to compare ItemBlocks in different variants * Added a brief description for the bunch of input parameters in HashedCollect * Allow unordered output tests in TestHelper class. * Added first HashedCollectExecutor test using the helper * Allow to use never wait in getSome / skipSome variant of WaitingExecutionBlockMock. * Added more HashedCollect tests (multi group, count) and started with framework for AggregateMethods * Fixed simple Aggregate Test * Improved output for unordered Row in Output test. * Added test around collecting of Arrays and Objects * Also allow a fullCount call in the old Execute Wrapper * Almost Completed register and call behaviour test suite * Added a todo for a missing test * Improved setup of AqlTestCase framework, not only start the server once. This speeds up AQL test by at least a factor of 10. * implemented newStyle HashedCollect executor. * Cleanup of old code. * Added test cases where the upstream producer lies to HashedCollect. * Let HashedCollect and Impl life with the fact that upstream lies. * Improved gtest with multiple calls. * Simplify input consume logic of HashedCollect Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Lars Maier <lars@arangodb.com> * Feature/aql subquery operations stack enumerate list (#10988) * produce inputRange * produce inputRange test * produce inputRange test 2 * apply new api * enable new api * enumeratelistexecutor changes * added hackis shouldSkip method * add new execution style for enumerate lsit * added skip for enumerate list * edit of enumerate list executor * fixed enumlist executor * optimized skip loop * more tests, using new framework * allow empty output checks * added new style tests, removed old tests * removed old produceRows method * fixed tests * added one missing exit case * added another exit in SHADOWROWS mode, as we need to take a look at fullCount there * Revert "added another exit in SHADOWROWS mode, as we need to take a look at fullCount there" This reverts commit e74b7cede78f81b8fece034fc921a969e91dda12. * do not take care of fullCount in executor itself, impl will do * removed obsolete comment * rm _skipped because unused, return upstreamstate * added invalid value test, added border edge tests for softLimit, hardLimit and fullCount * api change * added test for inner states inside the executor * added new tests * fixed assert * removed two not needed ifs * asserted a case in our test helper * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * undo this commit before merge * undo this commit before merge * Merged Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation inject stack into fetcher (#11124) * Added api to pass through the stack to the dependencyProxy. THis is just temporary and doomed to be removed as soon as API is completed. * Let depdendency Proxy forward the Stack. This may not work in all cases. * Default implement Assignment constructor. * Removed Query jenkins debug output again * Fixed forwarding of stack in skipSome * Tweak ExecutorTestHelper to test pipelines (#11079) * Tweak ExecutorTestHelper to test pipelines * Modifications * Fixup tests * Address some comments * Fix. * AQL execute in LimitExecutor (#10886) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Fixed merge * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Reverted accidental change * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Assert no input rows in skip * Began writing a test framework for LimitExecutor * Removed unused code * Continued test framework * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip * Make AqlItemBlockInputRange hold a skipped count * Added == for LimitStats, moved code to .cpp file * Added Stats return value to skipRowsRange * WIP: Implement LimitExecutor test framework * Fixed namespace * Added missing advanceRow call * Added parametrized test cases * Dump * Made test output more readable * Fixed merge conflicts * Set engine in createFakeQuery() * [WIP] Dump: not working * Fixed merge conflicts * Switch from boost::variant to std::variant * Fix comparison of differently sized blocks * Fixed usage of temporary * Fixed skip handling in ExecutionBlockImpl * Add upstream fullCount in LimitExecutor * Fix assertion that was too narrow * Again, fixed skip handling in ExecutionBlockImpl * Extended ExecutorTestHelper, mainly a loop to handle passthrough Executors * Fixed expected skipped count * Fixed bugs in LimitExecutor * Another bugfix in LimitExecutor * Fixed expected output calculation * Removed some comments * Improve test speed by not recreating MockAqlServer for every case * Allow additional variants in ExecutorTestHelper * Tried to fix a bug in LimitExecutor, not quite working * A little cleanup * Toggle lieAboutHasMore * Reverted broken fix, fixed WaitingExecutionBlockMock * Fixed merge conflicts * Add LimitStats to expected checks in tests * Fixed another leftover merge conflict * Fixed fullCount stats * Removed old code from LimitExecutor * Updated ExecutionBlockImpl to allow removal of old executor methods * Removed old tests * Simplified test code * Added comments and constructors to AqlCall * Instantiate test cases with multiple different calls * Fixed a bug in run(), made looping optional * Fixed a bug in DependencyProxy * Fixed a bug and an assertion in LimitExecutor * Added assertions and a fix in WaitingExecutionBlockMock * Enabled stats tests again * Made the executor compile under Mac * Let AqlHelper compare profiled stats as well * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * Modified skip logic in LimitExecutor to simplify what has to be reported to downStream and what is handled by internal skip. Also allow to do skip and fullCount in one go, if limit is 0 * Adapted the Expected skipped output in Limit test * Forward fullCount from client through LIMIT. Simplyfied produceLogic. * Reduced the amount of Tests in LimitExecutor. Need to decide on a suitable subset * Removed lieAboutHasMore by simply adding an empty block to the end that is blocking return in the WaitingExecutor * Fixed SingleRowFetcher to allow a Range with only skip information. * Intermed version of ExecutionBlockImpl. Needs to be continued * Allow hardLimit to pass getSome check * Intermed commit * Unified FastForward and FullCount code path * Produce and Skip go to FastForward if the Executor reports done and the output row is full. This way we can handle more FUllCount cases straight away. * Fixed the expected value for FullCount stats in test. This is rather awkward to automatically compute... * Fixed lying of the ExecutionBlockMock. Did not lie correctly if the limit was reached by softLimit. * Make the rules for supporting the OldAPI more relaxed. * Added a missmatch of outputColumns and internal output columns in TestHelper * Fixed AqlItemBlockRangeTests * Implemented unreachable code, otherwise our Jenkins does not compile it ;( * Fixed SortedCollect, it might count the last group twice on fullCount, if produce ends exactly in this group * Use AqlTestCase in ReturnExecutorTest * Return Executor needs to report skipped from upstream properly * Fixed fullCount call behaviour * Fixed expected numberSkipped in LimitExecutor test * Added some DEBUG level log messages in FastForward * Fixed reporting of skip in skipSome case and removed dead code * Fixed Impl test, we cannot let the WaitingBlockMock do counting on ShadowRows, it is not implemented there. * Removed unused function * Pointeless comment * Fixed skipSome of Filter * Avoid endless Loop in indexExecutor fullCount, where it tried to skipped 0 documents forever * Fixed return state of IndexExecutor * Fake DONE on hardLimit for oldAPI, apply only on toplevel. Improve shortcutting behviour to jump into FastForward everytime HARDLimit is reached, even if output is full. * Added an assertion that multiple UPSTREAM call rounds do not mess up the callStack by leaving the _upstreamCall on top. Now the executor itself pops off the call, this is subject to change if we manage the Stack differently * Was erlaube skip? Adjsuted tests to honor that we can do skip and limit now in one go, for all executors that are updated already * Fixed ConstFetcher, to not modify the CallStack, it only has a reference, where the other Fetchers copy it to the Executors, and modify it there * Do not overfetch on passthrough if the block is full. * We cannot create DONE on oldStyle if we need fullcount... * Unfortunately we teporarily need more calls in fullCount case, as long as one of the Executors above is not yet on execute * Mhm gcc and clang do not aggree on which types are equal... * Fixed compile issue. Co-authored-by: Michael Hackstein <michael@arangodb.com> Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> * Feature/aql subquery operations stack calculation exec enumerate exec (#11098) * removed unused import * re-ordered asserts to see which register was wrongly used * added implementation of produceRows with inputRange * added first test to test new produceRows including inputRange * rm skip from calc executor * rm not correct assert, added second test * added produceRows inputrange hasmore test * added new produceRows + test * added skipRowsRange enumerate collection executor * added skipRowsRange enumerate collection executor test * added assert * adde fake query helper, one without auto trx but with prepare plan, the other one without prepare but with trx - also removed fakeit mocks from enumeratecollectionexecutor tests * fixed skipRowsRange in enumerateCollectionExecutor, added test as well * removed todo * added skipAll * added EnumerateCollection as new style executor * added produce and skip rows, not ready yet * added new tests, still not ready, need unsorted compare operators in helper, waiting * rm logs * intermidiate commit - may not work * removed unused import * added implementation of produceRows with inputRange * added first test to test new produceRows including inputRange * rm skip from calc executor * rm not correct assert, added second test * added produceRows inputrange hasmore test * added new produceRows + test * added skipRowsRange enumerate collection executor * added skipRowsRange enumerate collection executor test * added assert * adde fake query helper, one without auto trx but with prepare plan, the other one without prepare but with trx - also removed fakeit mocks from enumeratecollectionexecutor tests * fixed skipRowsRange in enumerateCollectionExecutor, added test as well * removed todo * added skipAll * added EnumerateCollection as new style executor * added produce and skip rows, not ready yet * added new tests, still not ready, need unsorted compare operators in helper, waiting * rm logs * intermidiate commit - may not work * make branch compile again * mv assert * protected server * make enum collection tests work again * fixed ExecBlockImpl * todo optimization * fix gtest calc executor * header file enumcollexec, helper method * fix shrink method in aqlitemblock * impl * nodiscard * calc executor * rm empty line * fix calc exec * tmp re-enable old method * tmp make tests green again, needs cleanup later * use uint64_t instead of u_int64_t * removed unused code * Update arangod/Aql/CalculationExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * Update arangod/Aql/CalculationExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * remove implementation code of old api * Update arangod/Aql/EnumerateCollectionExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * buildable tests again * rm unused code * added assert * make tests compile again ... * disable few more tests, need to be optimized after subquery end start merges Co-authored-by: Michael Hackstein <michael@arangodb.com> * Quickfix for compilation error due to merge * Feature/aql subquery execution block impl execute implementation k shortest path executor (#10940) * Add new produceRow API code for KShortestPathsExecutor * Add KShortestPathsExecutor test to build * Add KShortestPathsExecutor test * Test and bugfix for KShortestPathExecutor * Update k-shortest-paths executor for new interface * Update tests for k-shortest-paths executor * Make ShortestPathExecutor and KShortestPathExecutor more similar * Fix merge messup * Refactor KShortestPathsExecutorInfos * Move KShortestPathsExecutor to new executor interface * Fix tests * Resolve naming conflict * Address some review comments * Fix skip * Fix for new signatures * Fix KShortestPathsExecutor * Fix KShortestPaths regression Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation traversal executor (#11023) * Move to new executor interface * Fix tests and some bugs * new skip signature * new skip signature * Fix Traversal Executor Skipping * Fix compiler complaint Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move SubqueryStartExecutor to new interface (#11025) * Move SubqueryStartExecutor to new interface * Diddle * Tests pass for SubqueryStartExecutor * Delete most of the SubqueryEndExecutor tests They are irrelevant now as they only tested shadowrow behaviour. Shadow Rows are now handled by ExecutionBlockImpl. * Skeleton tests * Junk * Cleanup and make work * Tweak ExecutorTestHelper to test pipelines * SubqueryStart/SubqueryEnd * Blafummel. * Modifications * Fixup tests * Address some comments * Fix. * Fix tests * Various fixes. * Made the branch compile on Mac * Intermediate commit, does not compile, required for merge * Fixed one of the SplicedSubqueryTests * Seperated ShadowRow handling for SubqueryStart/End executors to get a better overview and understanding on how it should behave. * Allow to use a fullCallStack in ExecutorTestHelper. * Added asserts on SubqueryEnd that is not allowed to be called with Skipped from upstream * Use AqlTestCase in SubqueryStartExecutorTest. Transformed first test to TestHelper pipeline * Let ExecutorTestHelper also test for shadowRows * Adapted SingleInput=>Data+Shadow test to testing Helepr and make it pass * Transformed existing SubqueryStart tests to Helper, fixed a test where there was no place for the ShadowRow to be written * Fixed Skip in SubqueryStartExecutor * Allow the Pipeline to add Consumer and dependency without fiddling with the Deque from outside * Added Skip adn ShadowRowForwarding tests to SubqueryStartExecutor * Fixed fullCount of SubqueryStart to be called properly. * Renamed isModificationSubquery -> isModificatioNode again * Consume more than one subquery en-block * Fixed debugReporting in SplicedSubqueryTest. Disabled the skip in subquery tests for now * Removed AQL debug log level from testing config * First version of Subquery Skip bypassing. Including 1 passing test. * Added disabled tests for skip, and proposed to do it later * Fixed reporting of FILTERED in fullCount of EnumerateCollectionExecutor * Enable calculationExecutor execute interface * Fixed compile issues * To Velocypack the modification Subquery value * Fixed illegal assert and reactivated failure tests * Fixed compile issue * Deactivated Calculation again * Added Compatibility Mode to AqlCallStack * Removed IdExecutor<void> the only feature it has can be moved into IDexecutor in general. This still has to be done * When falling back in BlocksWithClients use compatibility stack * ConstFetcher now correctly reports if it skipped all input * Fixed a bug in ExecutionBlockImpl, which would go into FetchMore state, even if the Executor still has data and is happy to produce more, but the output is full. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute remote node api flag (#11159) * Clarified comment * Added api property to RemoteNode * Added missing include * Feature/aql subquery operations stack i research view executor (#11140) * added arangosearch view executor to new style * added new produceRows and skipRowsRange * ires * test jenkins * modify aql call * cleanup branch * Update arangod/Aql/IResearchViewExecutor.cpp Co-Authored-By: Dronplane <andrei.lobov@arangodb.com> * applied suggested changes * also enable IResearchViewMergeExecutor * Fixed an issue in skipRange, where the input was not moved forward in all cases. * Removed unused local variables Co-authored-by: Dronplane <andrei.lobov@arangodb.com> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move SingleRemoteModificationExecutor to new interface (#11166) * Move SingleRemoteModificationExecutor to new interface * Activate SingleRemoteModificationExecutor * Inspect the result of remote modification Co-authored-by: Michael Hackstein <michael@arangodb.com> * New style DistinctCollect (#11096) * First implementation of new style executor for DistinctCollect. * Tests for distinct collect. * Remove debug output. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation noresults (#11180) * Use ExecutorHelper in tests * Implemented no Results fetcher * Moved Constrained sort to execute API (#11173) * Moved Constrained sort to execute API * The LIMIT in ConstrainedSort can actually fetch TOO much, if there is too small input from upstream * Improved profiler tests, we need less calls now \o/ * Fixed unlimited call to limit executor * Feature/aql subquery operations stack aql item block input matrix (#11167) * added new file to cmakelist * added new method skipAllRemainingDataRows * added first implementation of execute - AllRowsFetcher * impl is now using skipAllRemainingDataRows method, also execute methods now using Fetcher::DataRange instead of inputRange * impl is now using skipAllRemainingDataRows method, also execute methods now using Fetcher::DataRange instead of inputRange - header file * AllRowsFetcher now using AqlItemBlockInputMatrix as DataRange, first implementation of execute * first implementation of AqlItemBlockInputMatrix * more cleanup * optimized execut elogic * optimized logic in inputmatrix, still a few todos left * added SortExecutor as newStyle * added produceRows and skipRows * logs (to be removed), skipAllRemainingRows * logs * inputMatrix * changed return type of skipAllRemainingDataRows * impl logs * logs, added clear method * sort test * typo, log * latest changes * cleanup, assert * rm logs * added more todos * todo * new method names, compile again * Reset ExecutorReturnedDone for SubqueryStart and SubqueryEnd. * InputMatrix needs to return state DONE on every input request. It guarantees that we have all input * Removed invalid Assertion, and removed early return if output is full. * Removed DebugLogging from CalculationTest * Fixed skipping in SortExecutor * Fixed upstreamState return of AqlItemMatrix. It is only finalState iff there is no data * Let the InputMatrix drop all rows after FullCount * Allow bypassing with hardLimit 0 in InputMatrix * Disabled the old-code assertion. This way the SortExecutor tests we have will be red, but not crash, allowing the jenkins to produce meaningfull results * Fix assert in WAITING case * Add rudimentary sort executor GTests * Fixed assertion on InputMatrix that starts right away with a shadowRow * Removed declared but not implemented function * changed from: peekDataRowAndState -> to: peekDataRow Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute in restaqlhandler (#11175) * Updated comment * Added two static strings * Avoid unintentional streaming operator of string_view as a container * Added parser for execute rest call * Fixed ambiguous overload in ResultT * Added execute and executeForClient to ExecutionEngine * Implemented executeForClient in RestAqlHandler * Add default value * Updated comment * Changed error attribute * Fixed duplicate log id * Try to make MSVC work * Moved AqlExecuteResult to a separate file * Tried to fix MSVC compile error * Revert "Tried to fix MSVC compile error" This reverts commit f6f43b2c7e4bfe17dafe65f48bcd5c0b6e5c69a9. * Next try to fix MSVC * Revert "Next try to fix MSVC" This reverts commit 528c4c795d10ee0aa6686d143be1f6faa6e9b553. * Revert "Try to make MSVC work" This reverts commit ba7d9c072fccb969da1cdf045eadf49c297da8e9. * Work around MSVC shortcoming * Support executors with multiple dependencies (#11181) * Use DataRange in some places * Add MultiAqlItemBlockInputRange * First implementation of life, universe, and everything * implement skip for UnsortedGather * Some more wiring * Attempt at hacking up execute for DependencyProxy * Compilation fixes! * Fixed AqlCall compilation and unified execute and executeForDependency * Fixed WAITIGN assertion * Sonderlocken. * Foo * Make compile * Make tests pass. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move ModificationExecutors to new interface (#11165) * Move ModificationExecutors to new interface * added inputRange method to inputMatrix class, added switch between DataRange Input <-> Matrix for ModificationExecutors * forgot to enable allrows modification executor in fastForwardType * removed obsolete comment * removed obsolete comment Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery operations stack materialize executor (#11192) * removed obsolete comment, added produceRows and skipRowsRange, enabled MaterializeExecutor as newStyleExecutor * forgot to use advanceRow, fetch input during skipRowsRange * removed old implementations * removed comment, fixed a logical loop error * skip could be simplified as well * temp workaround Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute remote (#11197) * Clarified comment * Updated comment * Added two static strings * Avoid unintentional streaming operator of string_view as a container * Added parser for execute rest call * Fixed ambiguous overload in ResultT * Added execute and executeForClient to ExecutionEngine * Implemented executeForClient in RestAqlHandler * Add default value * Updated comment * Changed error attribute * Added API toggle to RemoteExecutor * Fixed compile error * Fixed duplicate log id * Try to make MSVC work * Moved AqlExecuteResult to a separate file * Moved AqlExecuteResult to a separate file * Implemtented execute in Remote; except for (de)serialization, which is still missing * Tried to fix MSVC compile error * Revert "Tried to fix MSVC compile error" This reverts commit f6f43b2c7e4bfe17dafe65f48bcd5c0b6e5c69a9. * Next try to fix MSVC * Implemented (de)serialization for RemoteExecutor::execute * Fixes * Added minimal AqlCall (de)serialization tests * Bugfix * Added minimal (de)serialization tests for AqlCallStack and AqlExecuteResult * Fixed test * Changed Serialization format a little * Bugfix * Initialize SingletonBlock immediately with an empty row * Revert "Next try to fix MSVC" This reverts commit 528c4c795d10ee0aa6686d143be1f6faa6e9b553. * Revert "Try to make MSVC work" This reverts commit ba7d9c072fccb969da1cdf045eadf49c297da8e9. * Work around MSVC shortcoming * Work around MSVC shortcoming * Attempt to fix windows compile issue Co-authored-by: Michael Hackstein <michael@arangodb.com> * Added test for distinct collect with random order. (#11184) * Feature/aql subquery execution block impl execute implementation subquery executor (#11196) * Intermediate commit, does not compile. * New style for SubqueryExecutor. * Removed bad assert. * Reset subsqueryInitialised in skip. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation count collect (#11172) * Move SubqueryStartExecutor to new interface * Diddle * Tests pass for SubqueryStartExecutor * Delete most of the SubqueryEndExecutor tests They are irrelevant now as they only tested shadowrow behaviour. Shadow Rows are now handled by ExecutionBlockImpl. * Skeleton tests * Junk * Cleanup and make work * Tweak ExecutorTestHelper to test pipelines * SubqueryStart/SubqueryEnd * Blafummel. * Modifications * Fixup tests * Address some comments * Fix. * Fix tests * Various fixes. * Made the branch compile on Mac * Intermediate commit, does not compile, required for merge * Fixed one of the SplicedSubqueryTests * Seperated ShadowRow handling for SubqueryStart/End executors to get a better overview and understanding on how it should behave. * Allow to use a fullCallStack in ExecutorTestHelper. * Added asserts on SubqueryEnd that is not allowed to be called with Skipped from upstream * Use AqlTestCase in SubqueryStartExecutorTest. Transformed first test to TestHelper pipeline * Let ExecutorTestHelper also test for shadowRows * Adapted SingleInput=>Data+Shadow test to testing Helepr and make it pass * Transformed existing SubqueryStart tests to Helper, fixed a test where there was no place for the ShadowRow to be written * Fixed Skip in SubqueryStartExecutor * Allow the Pipeline to add Consumer and dependency without fiddling with the Deque from outside * Added Skip adn ShadowRowForwarding tests to SubqueryStartExecutor * Fixed fullCount of SubqueryStart to be called properly. * Renamed isModificationSubquery -> isModificatioNode again * Consume more than one subquery en-block * Fixed debugReporting in SplicedSubqueryTest. Disabled the skip in subquery tests for now * Removed AQL debug log level from testing config * First version of Subquery Skip bypassing. Including 1 passing test. * Added disabled tests for skip, and proposed to do it later * Fixed reporting of FILTERED in fullCount of EnumerateCollectionExecutor * Enable calculationExecutor execute interface * Fixed compile issues * To Velocypack the modification Subquery value * Fixed illegal assert and reactivated failure tests * Fixed compile issue * Deactivated Calculation again * Added a testSuite for countCollectExecutor * Fixed Subquery Test for CountCollect * Added CountCollectImplementation * Added Compatibility Mode to AqlCallStack * Removed IdExecutor<void> the only feature it has can be moved into IDexecutor in general. This still has to be done * When falling back in BlocksWithClients use compatibility stack * Fixed EnumerateCollectionExecutor * Fixed fullCountBehaviour of EnumerateList. Also removed a unlimited,fullCount combination in IndexExecutor * Fixed endless loop while skipAl in OperationCursor * Fixed View Executor, which sends a no-limit no-offset upstream. * Fixed forwarding in NoResultsExecutor Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> * Activate CalculationExecutors (#11177) * Fixed compile bug-due to hidden merge conflict * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Started removing old getSome/skipSome API. This version does not compile yet * Harvested more, however the old tests still exist and do not compile now. * Feature/aql subquery execution block impl execute implementation base bugfixes (#11201) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Fix memory leak in traversal (#11210) * AqlSubqueryExecutionBlockImplExecuteImplementation use moveValueInto instead of cloneValueInto (#11213) * Fix memory leak in traversal * Use move instead of clone for AqlValues * Make compile * Feature/aql subquery execution block impl execute implementation gather executors (#11202) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Feature/aql subquery execute parallel gather (#11215) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute parallel gather 2 (#11221) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather * [WIP] Changed Api for MultiDepExecutors in ExecBlockImpl (not yet in the actual executors) * Moved AqlCallSet into a separate file * Changed SortingGather to use the new API * Changed ParallelUnsortedGather to use the new API * Changed UnsortedGather to use the new API * Moved AqlCall operator<< into .cpp file * Implement operator<< for AqlCallSet * Fix boolean mix-up * Fixed state machine: go to UPSTREAM when the AqlCallSet is not empty * Fixed assertion * Bugfix * SortingGather bugfixes * Added init() method to fix an assertion and cleanup * Removed unused variable * Fixed constrained sort * Fixed constrained sort #2 * Fix boolean mix-up * Remove old interface * Use call parameter for upstream request in produceRows * Remove more old interface code * Add skip methods to MultiAqlItemBlockInputRange * Skip in UnsortedGather * skip for UnsortedGather * Fix skip and upstream calls for UnsortedGather * skipRowsRange change * Remove useless comments * Moved multi-dep related code from ExeBlockImpl to MultiFetcher * Cleanup in SortingGather, implemented parallel fullCount * Try to fix a windows compile error * Simplify and extend skipRowsRange for UnsortedGatherExecutor * Made ParallelUnsortedGather actually parallel * Removed erroneous assertion * Undid erroneous change * Fixed MacOs compile. Also disabled tests for non-relevant AqlCallStacks. They will be removed * Fixed initialize Cursor for multi dependency blocks * Fixed fullCount case in parallel unsorted gather * Fixed fullCount upstream call of ParallelUnsortedGatherExecutor * Fixed fullCount in SortingGather * Windows \o/ if you cannot work properly with constexpr and static asserts, we do not let you do it! * Do not advance in Unsorted gather if there are still rows to skip * Add more comparison operators for AqlCall limits * Send clientCall limits to upstream in SortingGather * Improved fullCount in SortingGatherExectur * Disabled a cluster profile test. We now ask the RemoteNode more often if it already has data. It is a bit unclear to me if this is now better performance wise (<< i think so) or triggers undesired side effects * Helpless attempt to work around issues in stonage Visual Studio Compiler we are using. * Clearly adding an operator on a well defined type causes ambigousness on random basic types using the operator Co-authored-by: Michael Hackstein <michael@arangodb.com> Co-authored-by: Markus Pfeiffer <markus@arangodb.com> * Feature/aql subquery execution block impl execute implementation bypass skip (#11203) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of P…
KVS85
added a commit
that referenced
this issue
Apr 8, 2020
* arangodb release 3.7.0 * Feature/aql subquery execution block impl execute implementation harvesting (#11349) * Added first draft of AqlCallObject * Added AqlItemBlockInputRange * Added Header file for AqlCallStack with comments. No implementation yet. * Added AqlCallStack first draft implementation. Made everything compile * Added empty dummy stubs for execute() in ExecutionBlock * Implementation of new API in FilterExecutor. Rough implementation in ExecutionBlockImpl (not complete, only skip path) everything compiles, but is not tested * Fixed off by one error in InputRange. Fixed FilterExecutor * Added some more tests if executor cannot fulfill atMost * Added heplper functions to the AqlCall struct to avoid mangeling around with variants everywhere. * Add a member DataRange to ExecutionBlockImpl. Also now all Fetcher expose a DataRange, this might not be the correct one every where * Added preliminary minimal implementation of execute to DependencyProxy and SingleRowFetcher. Both untested and yet incomplete, they will only work for MVP. * added first implementation of count collect datarange produceRows function + test * Revert "added first implementation of count collect datarange produceRows function + test" This reverts commit 236825280bb5fd03dfe01febfc4dcf1a2d2e2c2f. * First draft of execute implementation, not yet handling ShadowRows though. This state is broken in the sense that a server does not boot with it and catch tests are broken * First working draft of execute() call in ExecutionBlockImpl. Tests are locally green, however there is at least one query still red (on ShadowRows) * Removed non finished implementation from this Branch. It moved to seperate branch * working - not done yet * Revert "working - not done yet" This reverts commit aa275b48ffc2c8fc9a3c364e2241321ec9c9293f. * first implementation of a ShadowRow fetching interface on AqlItemBlockInputRange * single row fetcher, execute + tests * Fixed SingleRowFetcherTest. Also an AqlItemBlockInputRange can now handle ShadowRows properly * Another fix on handling of shadow rows in InputRanges * Improved human readable output of SortLimit test, while validating the result * Removed debug profile execution of a query * Use simpleVPack printing for Trace of Execute, better to read by humans * Added tests for AqlItemBlockInputRange and adjusted implementation accordingly. * Fixed issue where ExecutionBlockImpl returns DONE, although there are still rows in the DataRange * Final modification of ShadowRows in FilterExecutor. All but profiling tests pass. This needs some discussion about overfetching of data, and propagation of hardLimit * Removed debug log includes * Moved responsibility for the client call into the output row. This can precisely count the elements it has written, and forms a single source of truth. The output row now works as the requested block by client that needs to be filled. Now the FilterExecutorTests are red. * Fixed FilterExecutorTest * Expose required features to AqlCall controlled by the output aql item row. Adapted executionBlockImpl to report there * Fixed bug with AqlCallStack pop operation that actually caused invalid memory access * outputRow now honors limits whan called for isFull. * Fixed isFull() in OutputAqlItemRow * Use the inbound call in skipRows for executors, in order to do a more precise upstream call * Make linker happy with operator< on CallLimit * Implement operator+ on AqlCall::Limit, also use std::visit for readability. * FilterExecutor getSome now propagates offset + min(limits) to upstream. This is pessimistic and avoids overfetching in all cases. NOTE: FilterExecutor should be moved to skipvariant using getSome * Fixed profiler tests for filter. These cannot be guaranteed anymore and it was an undesired, yet uncritical side effect that the tests have been green before * Fixed JSLint * Remove special casing for FilterExecutor * use if constexpr to distinguish between ported executors and non-ported executors enabling step-by-step porting and testing * Add skeleton skip code * Replace C&P accident * Simplify and implement skip for new executor interface * Fixup skipping for filter executor * Rename some methods for clarity and consistency * AqlItemBlockInputRange - hasMore is now hasDataRow - next is now nextDataRow - peek is now peekDataRow - state is now upstreamState - additional method upstreamHasMore * Introduce function to allocate an output block * Fix skipSome simulation bug * Some small cleanups * Replaced assertion on atMost on the output size. Otherwise we got into issues with atMost changes between waiting calls (can occur in our intermediate state now) * Attempt at fixing execute * Fix merge conflict * Feature/aql subquery execution block impl execute implementation exec block impl tests (#10838) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation shortest path executor (#10780) * Hook up new ShortestPathExecutor * Small fixes * Removed now unused code. * Moved ShortestPathExecutor tests to use Execute only. * Added a test for shortestPathExecutor where the output is smaller than the amount of entries within the path. * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Fixed Windows compile issue * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Windows compile fix * Attempt at fixing ShortestPathExecutorTest for Windows * Windows compile fix * Added a skip test and fixed the producing executor to recreate the list once for every input line. * Frankenstein my tests into the branch * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Hack * Removed designated initializers. Thanks for not supporting it MSVC! * Update ShortestPathExecutorTest * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Umbau 2.0 * Reshape output and skip a bit * Some test fixes * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Square 1 * Input validation * ShortestPathExecutor with fullCount and tests Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation exec block impl tests shadow rows (#10970) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Added tests for ExecutionBlockImpl, when there is a WAITING passed through. There is no test yet for in between waitings. * Allow to configure WAITING variant in WAITING block mock. * Use the WaitingBlockMock to be independent of Singleton in test * Added WAITING + Passthrough tests * Added a test to path through shadowRows without data. * Waiting ExecutionBlock Mock can now return/skip partial blocks. However it does NOT CARE for ShadowRows * Added tests and fixed an issue in ShadowRow forwarding * Added tests for shadowRow forwarding * Added the intenral state of Execute as member of ExecutionBlockImpl. We need to retain parts of this state accross returns * Fixed all Waiting and multiple call tests * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Fixed invalid memory access in test code. Do never return a lambda that references a stack variable... * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip * Allways -> Always. * Added tests for pass-by of non-relevant Subqueries. Fixed a bug found on the way in WaitingBlockMock. Now some other tests fail \o/ * Fixed failed tests. Now tests correspond to correct behaviour of WaitingExecutionBlockMock * Intermediate version, doe snot work * Fixed the empty subquery tests Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Feature/aql subquery execution block impl execute implementation sorted collect (#11020) * SortedCollect executor and some tests. * More tests. * More const and skip implemented. * Fixed some bugs found by tests. * Refactored skip. * Broken state with test framework. * Extracted testing framework into separate file and adjusted it to use WaitingBlockMock. However it is not yet fully implemented * Added another SortedExecutor test without fullcount. This passes. Also assert on skip and State * Removed obsolete assertion in skip, added another nonFullcount test. * More tests. Fixed some bugs in the test framework. * More cleanup. * Remove version control marker. * Removed wrong assertion. * Apply suggestions from code review Co-Authored-By: Michael Hackstein <michael@arangodb.com> * Remove dead code. * Removed fetcher. Co-authored-by: Michael Hackstein <michael@arangodb.com> * ReturnExecutor New style (#10831) * ReturnExecutor implemented. Tests misssing. * Apply suggestions from code review Co-Authored-By: Tobias Gödderz <tobias@arangodb.com> * Implemented skipRows in Return. Also fixed ExecutionBlock references in static asserts * Implemented first new style test for Return. * Moved the traceExecute calls into baseclass. * Added tracing around execute in WaitingExecutionBlockMock. Fixed a test bug, if the AqlCall covers all of the first block, but only a part of the Second block. You would only get the part of the second block. * Added more tests for standard behaviour of ReturnExecutor * Added an assertion. * Simplified call to ReturnExecutor * Prepare tests for doCount. Also added test for different input register. * Added accessor for _stats in Engine. * Included count tests in ReturnExecutorTest. * Removed old obsolete test. * Removed obsolete code-path * Fixed unrelated bug in FilterExecutor. In some cases it did lie on skipRows that it has more, even if it does not have and the upstream does not have as well. Thank you jenkins for hitting into this race :+1: Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Tweak ExecutorTestHelper to test pipelines * Revert "Tweak ExecutorTestHelper to test pipelines" This reverts commit 6359f0c926cad30b4efd7ce263fb3b64563621b8. * Added Stats return value to skipRowsRange (#11081) * Added Stats return value to skipRowsRange * Fixed catch test. Somehow i merged the incorrect part Co-authored-by: Tobias Gödderz <github@tobias.goedderz.info> * Additional Assertion in ExecutionBlockImpl (#11077) * Added a assertion arround skipRowsInRange that ensures that the Call and the skippedLocal do not get out-of-sync * Improved the assertion * Feature/aql subquery operations stack id executor (#10986) * added inputRange skip and produce * added inputRange skip and produce * Added shouldSkip helper to AqlCall * Added implementation if execute to Constfetcher. Not specifically tested yet. * Moved IDExecutor to newStyle executor * Added some documentation to SingleRowFetcher * Adapted IdExecutor tests. * Added an assertion in ConstFetcher * Fixed initialize cursor in ExecutionBlockImpl * Added tests for Initilaize cursor in IdExecutor. There is a special implementation for this... * Fixed linker issues, because of variables in cpp files named identical and not beeing static * Started refactoring the Scatter/Gather Executor logic. Added an intermediate class that does not use a template and can be casted to. This state compiles but will not work. * Made BlocksWithClientsImpl a template, based on an executor * Added a dummy implementation to fetch a new block. Distribute not yet implemented. * Added superoptimistic first implementation of ScatterExecutor return. Only works if there are no shadowRows. * Implemented logic to slice Blocks in Scatter according to the given call. Also added handfull of test for it, not complete though * Fixed a Scatter Executor handling of softLimit. Added another test to ansure diverse calls of clients to Scatter * Added random ordering test framework on ScatterExecutor * Added some tests for ScatterExecutor including subqueries * Removed todo, as it has been solved by now * Implemented Scatter by using IDExecutor. Tests not green yet * Implementation of ConstFetcher that allows to inject fullBlocks of AqlItemRows, and not only single rows. Also handles ShadowRows now in a correct way * Do not jump over relevant shadow rows in ConstFetcher. * Fixed tests. * Moved information of DistributeExecutor into the Infos. This resulted in adapting the Constructor API. I used this situation to do a big cleanup of ancient code. From this point there is no way back. * Implemented the last bits of DistributeExecutor * 'Implemented' execute by fallback to old API in RemoteExecutor * Fixed an EdgeCase with Scatter Executor, where the data was modified in place right after scatter. This has side-effects on peer executors. So we for now do a full copy of every block here. * Removed invalid assertions, also fixed RegisterInput to DistributeExecutor * Removed obsolete TODOs * Update arangod/Aql/IdExecutor.cpp Co-Authored-By: Lars Maier <lars@arangodb.com> Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Lars Maier <lars@arangodb.com> * Replaced the operator== and operator!= with `isSameBlockAndIndex`. (#11088) * New style IndexExecutor. (#11029) * New style IndexExecutor. * Small changes. * Fixing some logic for skipping. * Small changes. * TRI_ASSERT(FALSE); * Removed _fetcher member. (again) * Feature/aql subquery execution block impl execute implementation debug logging (#11112) * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * Feature/aql subquery operations stack hashed collect executor (#11103) * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * added inputRange skip and produce * added inputRange skip and produce * Added shouldSkip helper to AqlCall * Added implementation if execute to Constfetcher. Not specifically tested yet. * Moved IDExecutor to newStyle executor * Added some documentation to SingleRowFetcher * Adapted IdExecutor tests. * Added an assertion in ConstFetcher * Fixed initialize cursor in ExecutionBlockImpl * Added tests for Initilaize cursor in IdExecutor. There is a special implementation for this... * Fixed linker issues, because of variables in cpp files named identical and not beeing static * Started refactoring the Scatter/Gather Executor logic. Added an intermediate class that does not use a template and can be casted to. This state compiles but will not work. * Made BlocksWithClientsImpl a template, based on an executor * Added a dummy implementation to fetch a new block. Distribute not yet implemented. * Added superoptimistic first implementation of ScatterExecutor return. Only works if there are no shadowRows. * Implemented logic to slice Blocks in Scatter according to the given call. Also added handfull of test for it, not complete though * Fixed a Scatter Executor handling of softLimit. Added another test to ansure diverse calls of clients to Scatter * Added random ordering test framework on ScatterExecutor * Added some tests for ScatterExecutor including subqueries * Removed todo, as it has been solved by now * Implemented Scatter by using IDExecutor. Tests not green yet * Implementation of ConstFetcher that allows to inject fullBlocks of AqlItemRows, and not only single rows. Also handles ShadowRows now in a correct way * Do not jump over relevant shadow rows in ConstFetcher. * Fixed tests. * Moved information of DistributeExecutor into the Infos. This resulted in adapting the Constructor API. I used this situation to do a big cleanup of ancient code. From this point there is no way back. * Implemented the last bits of DistributeExecutor * 'Implemented' execute by fallback to old API in RemoteExecutor * Fixed an EdgeCase with Scatter Executor, where the data was modified in place right after scatter. This has side-effects on peer executors. So we for now do a full copy of every block here. * Removed invalid assertions, also fixed RegisterInput to DistributeExecutor * Removed obsolete TODOs * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * produce inputRange * produce inputRange test 1 * produce inputRange test 2 * produce inputRange test 3 * finalized tests * Made the branch compile again * Update arangod/Aql/IdExecutor.cpp Co-Authored-By: Lars Maier <lars@arangodb.com> * Intermediate commit. Empty might not compile * Added some external helper functions to compare ItemBlocks in different variants * Added a brief description for the bunch of input parameters in HashedCollect * Allow unordered output tests in TestHelper class. * Added first HashedCollectExecutor test using the helper * Allow to use never wait in getSome / skipSome variant of WaitingExecutionBlockMock. * Added more HashedCollect tests (multi group, count) and started with framework for AggregateMethods * Fixed simple Aggregate Test * Improved output for unordered Row in Output test. * Added test around collecting of Arrays and Objects * Also allow a fullCount call in the old Execute Wrapper * Almost Completed register and call behaviour test suite * Added a todo for a missing test * Improved setup of AqlTestCase framework, not only start the server once. This speeds up AQL test by at least a factor of 10. * implemented newStyle HashedCollect executor. * Cleanup of old code. * Added test cases where the upstream producer lies to HashedCollect. * Let HashedCollect and Impl life with the fact that upstream lies. * Improved gtest with multiple calls. * Simplify input consume logic of HashedCollect Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Lars Maier <lars@arangodb.com> * Feature/aql subquery operations stack enumerate list (#10988) * produce inputRange * produce inputRange test * produce inputRange test 2 * apply new api * enable new api * enumeratelistexecutor changes * added hackis shouldSkip method * add new execution style for enumerate lsit * added skip for enumerate list * edit of enumerate list executor * fixed enumlist executor * optimized skip loop * more tests, using new framework * allow empty output checks * added new style tests, removed old tests * removed old produceRows method * fixed tests * added one missing exit case * added another exit in SHADOWROWS mode, as we need to take a look at fullCount there * Revert "added another exit in SHADOWROWS mode, as we need to take a look at fullCount there" This reverts commit e74b7cede78f81b8fece034fc921a969e91dda12. * do not take care of fullCount in executor itself, impl will do * removed obsolete comment * rm _skipped because unused, return upstreamstate * added invalid value test, added border edge tests for softLimit, hardLimit and fullCount * api change * added test for inner states inside the executor * added new tests * fixed assert * removed two not needed ifs * asserted a case in our test helper * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * undo this commit before merge * undo this commit before merge * Merged Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation inject stack into fetcher (#11124) * Added api to pass through the stack to the dependencyProxy. THis is just temporary and doomed to be removed as soon as API is completed. * Let depdendency Proxy forward the Stack. This may not work in all cases. * Default implement Assignment constructor. * Removed Query jenkins debug output again * Fixed forwarding of stack in skipSome * Tweak ExecutorTestHelper to test pipelines (#11079) * Tweak ExecutorTestHelper to test pipelines * Modifications * Fixup tests * Address some comments * Fix. * AQL execute in LimitExecutor (#10886) * Added a generic Lambda Executor, this can be used in ExecutionBlockImplTests to have finegrained control over the action happening within the Executor * Added first test using the LambdaExecutor. * Added some tests around Execute. GetSome seems to be okayish. Skip not. Namely skipped numbers are not reported correctly. * Made the first ExecutionBlock Execute integration test pass. Still other tests are failing * Simplified the IsSkipSome test, this allows the C++ tests for Mixed Calls to pass. * Added a skip test and fixed the producing executor to recreate the list once for every input line. * More tests. Also added a custom AqlCall printer function for GTest. Tests still red, need to fix fullCount and hardLimit. * Implemented ostream operator for AQLCall. * Properly implemented fullCount incl. UnitTest * Added test for Callforwarding. They still have some todos, but these can only be solved by upgrading the OutputRow. Which should be part of separate PR * Added another test for CallForwarding in passthrough state * Added a Test Implementation for an Executor that uses a dynamic skip implementation. * Fixed skip with HARDLIMIT. * Startet to implement call forwarding test. However we need to improve Outputrow first this will be done in seperate branch * Removed designated initializers. Thanks for not supporting it MSVC! * Removed non-passthrough non-skip Lambda Executor again. We decided to disallow this. * Fixed merge * Update tests/Aql/ExecutionBlockImplTest.cpp Co-Authored-By: Markus Pfeiffer <markuspf@users.noreply.github.com> * Reverted accidental change * Started to add implementation of passthrough block allocation * Added a comparator to AqlCall. Mostly for tests * Fixed an issue in skip-passthrough version. Updated the tests. * Allow to 'overSkip' if we do fullCount * Enabled the first set of tests now. Only one set to go * Applied all fixes to get Integration testsuite green * Added some comments on the TestCases executed in the ExecutionBlockImpl * Added tes descriptions and removed a duplicate test * Added some comments on LamdbaExecutors * Added description of ExecutionBlockImple execute logic * Assert no input rows in skip * Began writing a test framework for LimitExecutor * Removed unused code * Continued test framework * Applied review comments, thanks to reviewers * Fixed modulo 2 off by one error * Renamed getLambda() => getProduceLambda() in test code, as we have produce and skip * Make AqlItemBlockInputRange hold a skipped count * Added == for LimitStats, moved code to .cpp file * Added Stats return value to skipRowsRange * WIP: Implement LimitExecutor test framework * Fixed namespace * Added missing advanceRow call * Added parametrized test cases * Dump * Made test output more readable * Fixed merge conflicts * Set engine in createFakeQuery() * [WIP] Dump: not working * Fixed merge conflicts * Switch from boost::variant to std::variant * Fix comparison of differently sized blocks * Fixed usage of temporary * Fixed skip handling in ExecutionBlockImpl * Add upstream fullCount in LimitExecutor * Fix assertion that was too narrow * Again, fixed skip handling in ExecutionBlockImpl * Extended ExecutorTestHelper, mainly a loop to handle passthrough Executors * Fixed expected skipped count * Fixed bugs in LimitExecutor * Another bugfix in LimitExecutor * Fixed expected output calculation * Removed some comments * Improve test speed by not recreating MockAqlServer for every case * Allow additional variants in ExecutorTestHelper * Tried to fix a bug in LimitExecutor, not quite working * A little cleanup * Toggle lieAboutHasMore * Reverted broken fix, fixed WaitingExecutionBlockMock * Fixed merge conflicts * Add LimitStats to expected checks in tests * Fixed another leftover merge conflict * Fixed fullCount stats * Removed old code from LimitExecutor * Updated ExecutionBlockImpl to allow removal of old executor methods * Removed old tests * Simplified test code * Added comments and constructors to AqlCall * Instantiate test cases with multiple different calls * Fixed a bug in run(), made looping optional * Fixed a bug in DependencyProxy * Fixed a bug and an assertion in LimitExecutor * Added assertions and a fix in WaitingExecutionBlockMock * Enabled stats tests again * Made the executor compile under Mac * Let AqlHelper compare profiled stats as well * Prepare reusability of LOG output for ExecutionBlock. * Added first version of DEBUG output to ExecutionBlockImpl * Modified skip logic in LimitExecutor to simplify what has to be reported to downStream and what is handled by internal skip. Also allow to do skip and fullCount in one go, if limit is 0 * Adapted the Expected skipped output in Limit test * Forward fullCount from client through LIMIT. Simplyfied produceLogic. * Reduced the amount of Tests in LimitExecutor. Need to decide on a suitable subset * Removed lieAboutHasMore by simply adding an empty block to the end that is blocking return in the WaitingExecutor * Fixed SingleRowFetcher to allow a Range with only skip information. * Intermed version of ExecutionBlockImpl. Needs to be continued * Allow hardLimit to pass getSome check * Intermed commit * Unified FastForward and FullCount code path * Produce and Skip go to FastForward if the Executor reports done and the output row is full. This way we can handle more FUllCount cases straight away. * Fixed the expected value for FullCount stats in test. This is rather awkward to automatically compute... * Fixed lying of the ExecutionBlockMock. Did not lie correctly if the limit was reached by softLimit. * Make the rules for supporting the OldAPI more relaxed. * Added a missmatch of outputColumns and internal output columns in TestHelper * Fixed AqlItemBlockRangeTests * Implemented unreachable code, otherwise our Jenkins does not compile it ;( * Fixed SortedCollect, it might count the last group twice on fullCount, if produce ends exactly in this group * Use AqlTestCase in ReturnExecutorTest * Return Executor needs to report skipped from upstream properly * Fixed fullCount call behaviour * Fixed expected numberSkipped in LimitExecutor test * Added some DEBUG level log messages in FastForward * Fixed reporting of skip in skipSome case and removed dead code * Fixed Impl test, we cannot let the WaitingBlockMock do counting on ShadowRows, it is not implemented there. * Removed unused function * Pointeless comment * Fixed skipSome of Filter * Avoid endless Loop in indexExecutor fullCount, where it tried to skipped 0 documents forever * Fixed return state of IndexExecutor * Fake DONE on hardLimit for oldAPI, apply only on toplevel. Improve shortcutting behviour to jump into FastForward everytime HARDLimit is reached, even if output is full. * Added an assertion that multiple UPSTREAM call rounds do not mess up the callStack by leaving the _upstreamCall on top. Now the executor itself pops off the call, this is subject to change if we manage the Stack differently * Was erlaube skip? Adjsuted tests to honor that we can do skip and limit now in one go, for all executors that are updated already * Fixed ConstFetcher, to not modify the CallStack, it only has a reference, where the other Fetchers copy it to the Executors, and modify it there * Do not overfetch on passthrough if the block is full. * We cannot create DONE on oldStyle if we need fullcount... * Unfortunately we teporarily need more calls in fullCount case, as long as one of the Executors above is not yet on execute * Mhm gcc and clang do not aggree on which types are equal... * Fixed compile issue. Co-authored-by: Michael Hackstein <michael@arangodb.com> Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> * Feature/aql subquery operations stack calculation exec enumerate exec (#11098) * removed unused import * re-ordered asserts to see which register was wrongly used * added implementation of produceRows with inputRange * added first test to test new produceRows including inputRange * rm skip from calc executor * rm not correct assert, added second test * added produceRows inputrange hasmore test * added new produceRows + test * added skipRowsRange enumerate collection executor * added skipRowsRange enumerate collection executor test * added assert * adde fake query helper, one without auto trx but with prepare plan, the other one without prepare but with trx - also removed fakeit mocks from enumeratecollectionexecutor tests * fixed skipRowsRange in enumerateCollectionExecutor, added test as well * removed todo * added skipAll * added EnumerateCollection as new style executor * added produce and skip rows, not ready yet * added new tests, still not ready, need unsorted compare operators in helper, waiting * rm logs * intermidiate commit - may not work * removed unused import * added implementation of produceRows with inputRange * added first test to test new produceRows including inputRange * rm skip from calc executor * rm not correct assert, added second test * added produceRows inputrange hasmore test * added new produceRows + test * added skipRowsRange enumerate collection executor * added skipRowsRange enumerate collection executor test * added assert * adde fake query helper, one without auto trx but with prepare plan, the other one without prepare but with trx - also removed fakeit mocks from enumeratecollectionexecutor tests * fixed skipRowsRange in enumerateCollectionExecutor, added test as well * removed todo * added skipAll * added EnumerateCollection as new style executor * added produce and skip rows, not ready yet * added new tests, still not ready, need unsorted compare operators in helper, waiting * rm logs * intermidiate commit - may not work * make branch compile again * mv assert * protected server * make enum collection tests work again * fixed ExecBlockImpl * todo optimization * fix gtest calc executor * header file enumcollexec, helper method * fix shrink method in aqlitemblock * impl * nodiscard * calc executor * rm empty line * fix calc exec * tmp re-enable old method * tmp make tests green again, needs cleanup later * use uint64_t instead of u_int64_t * removed unused code * Update arangod/Aql/CalculationExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * Update arangod/Aql/CalculationExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * remove implementation code of old api * Update arangod/Aql/EnumerateCollectionExecutor.cpp Co-Authored-By: Michael Hackstein <michael@arangodb.com> * buildable tests again * rm unused code * added assert * make tests compile again ... * disable few more tests, need to be optimized after subquery end start merges Co-authored-by: Michael Hackstein <michael@arangodb.com> * Quickfix for compilation error due to merge * Feature/aql subquery execution block impl execute implementation k shortest path executor (#10940) * Add new produceRow API code for KShortestPathsExecutor * Add KShortestPathsExecutor test to build * Add KShortestPathsExecutor test * Test and bugfix for KShortestPathExecutor * Update k-shortest-paths executor for new interface * Update tests for k-shortest-paths executor * Make ShortestPathExecutor and KShortestPathExecutor more similar * Fix merge messup * Refactor KShortestPathsExecutorInfos * Move KShortestPathsExecutor to new executor interface * Fix tests * Resolve naming conflict * Address some review comments * Fix skip * Fix for new signatures * Fix KShortestPathsExecutor * Fix KShortestPaths regression Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation traversal executor (#11023) * Move to new executor interface * Fix tests and some bugs * new skip signature * new skip signature * Fix Traversal Executor Skipping * Fix compiler complaint Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move SubqueryStartExecutor to new interface (#11025) * Move SubqueryStartExecutor to new interface * Diddle * Tests pass for SubqueryStartExecutor * Delete most of the SubqueryEndExecutor tests They are irrelevant now as they only tested shadowrow behaviour. Shadow Rows are now handled by ExecutionBlockImpl. * Skeleton tests * Junk * Cleanup and make work * Tweak ExecutorTestHelper to test pipelines * SubqueryStart/SubqueryEnd * Blafummel. * Modifications * Fixup tests * Address some comments * Fix. * Fix tests * Various fixes. * Made the branch compile on Mac * Intermediate commit, does not compile, required for merge * Fixed one of the SplicedSubqueryTests * Seperated ShadowRow handling for SubqueryStart/End executors to get a better overview and understanding on how it should behave. * Allow to use a fullCallStack in ExecutorTestHelper. * Added asserts on SubqueryEnd that is not allowed to be called with Skipped from upstream * Use AqlTestCase in SubqueryStartExecutorTest. Transformed first test to TestHelper pipeline * Let ExecutorTestHelper also test for shadowRows * Adapted SingleInput=>Data+Shadow test to testing Helepr and make it pass * Transformed existing SubqueryStart tests to Helper, fixed a test where there was no place for the ShadowRow to be written * Fixed Skip in SubqueryStartExecutor * Allow the Pipeline to add Consumer and dependency without fiddling with the Deque from outside * Added Skip adn ShadowRowForwarding tests to SubqueryStartExecutor * Fixed fullCount of SubqueryStart to be called properly. * Renamed isModificationSubquery -> isModificatioNode again * Consume more than one subquery en-block * Fixed debugReporting in SplicedSubqueryTest. Disabled the skip in subquery tests for now * Removed AQL debug log level from testing config * First version of Subquery Skip bypassing. Including 1 passing test. * Added disabled tests for skip, and proposed to do it later * Fixed reporting of FILTERED in fullCount of EnumerateCollectionExecutor * Enable calculationExecutor execute interface * Fixed compile issues * To Velocypack the modification Subquery value * Fixed illegal assert and reactivated failure tests * Fixed compile issue * Deactivated Calculation again * Added Compatibility Mode to AqlCallStack * Removed IdExecutor<void> the only feature it has can be moved into IDexecutor in general. This still has to be done * When falling back in BlocksWithClients use compatibility stack * ConstFetcher now correctly reports if it skipped all input * Fixed a bug in ExecutionBlockImpl, which would go into FetchMore state, even if the Executor still has data and is happy to produce more, but the output is full. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute remote node api flag (#11159) * Clarified comment * Added api property to RemoteNode * Added missing include * Feature/aql subquery operations stack i research view executor (#11140) * added arangosearch view executor to new style * added new produceRows and skipRowsRange * ires * test jenkins * modify aql call * cleanup branch * Update arangod/Aql/IResearchViewExecutor.cpp Co-Authored-By: Dronplane <andrei.lobov@arangodb.com> * applied suggested changes * also enable IResearchViewMergeExecutor * Fixed an issue in skipRange, where the input was not moved forward in all cases. * Removed unused local variables Co-authored-by: Dronplane <andrei.lobov@arangodb.com> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move SingleRemoteModificationExecutor to new interface (#11166) * Move SingleRemoteModificationExecutor to new interface * Activate SingleRemoteModificationExecutor * Inspect the result of remote modification Co-authored-by: Michael Hackstein <michael@arangodb.com> * New style DistinctCollect (#11096) * First implementation of new style executor for DistinctCollect. * Tests for distinct collect. * Remove debug output. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation noresults (#11180) * Use ExecutorHelper in tests * Implemented no Results fetcher * Moved Constrained sort to execute API (#11173) * Moved Constrained sort to execute API * The LIMIT in ConstrainedSort can actually fetch TOO much, if there is too small input from upstream * Improved profiler tests, we need less calls now \o/ * Fixed unlimited call to limit executor * Feature/aql subquery operations stack aql item block input matrix (#11167) * added new file to cmakelist * added new method skipAllRemainingDataRows * added first implementation of execute - AllRowsFetcher * impl is now using skipAllRemainingDataRows method, also execute methods now using Fetcher::DataRange instead of inputRange * impl is now using skipAllRemainingDataRows method, also execute methods now using Fetcher::DataRange instead of inputRange - header file * AllRowsFetcher now using AqlItemBlockInputMatrix as DataRange, first implementation of execute * first implementation of AqlItemBlockInputMatrix * more cleanup * optimized execut elogic * optimized logic in inputmatrix, still a few todos left * added SortExecutor as newStyle * added produceRows and skipRows * logs (to be removed), skipAllRemainingRows * logs * inputMatrix * changed return type of skipAllRemainingDataRows * impl logs * logs, added clear method * sort test * typo, log * latest changes * cleanup, assert * rm logs * added more todos * todo * new method names, compile again * Reset ExecutorReturnedDone for SubqueryStart and SubqueryEnd. * InputMatrix needs to return state DONE on every input request. It guarantees that we have all input * Removed invalid Assertion, and removed early return if output is full. * Removed DebugLogging from CalculationTest * Fixed skipping in SortExecutor * Fixed upstreamState return of AqlItemMatrix. It is only finalState iff there is no data * Let the InputMatrix drop all rows after FullCount * Allow bypassing with hardLimit 0 in InputMatrix * Disabled the old-code assertion. This way the SortExecutor tests we have will be red, but not crash, allowing the jenkins to produce meaningfull results * Fix assert in WAITING case * Add rudimentary sort executor GTests * Fixed assertion on InputMatrix that starts right away with a shadowRow * Removed declared but not implemented function * changed from: peekDataRowAndState -> to: peekDataRow Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute in restaqlhandler (#11175) * Updated comment * Added two static strings * Avoid unintentional streaming operator of string_view as a container * Added parser for execute rest call * Fixed ambiguous overload in ResultT * Added execute and executeForClient to ExecutionEngine * Implemented executeForClient in RestAqlHandler * Add default value * Updated comment * Changed error attribute * Fixed duplicate log id * Try to make MSVC work * Moved AqlExecuteResult to a separate file * Tried to fix MSVC compile error * Revert "Tried to fix MSVC compile error" This reverts commit f6f43b2c7e4bfe17dafe65f48bcd5c0b6e5c69a9. * Next try to fix MSVC * Revert "Next try to fix MSVC" This reverts commit 528c4c795d10ee0aa6686d143be1f6faa6e9b553. * Revert "Try to make MSVC work" This reverts commit ba7d9c072fccb969da1cdf045eadf49c297da8e9. * Work around MSVC shortcoming * Support executors with multiple dependencies (#11181) * Use DataRange in some places * Add MultiAqlItemBlockInputRange * First implementation of life, universe, and everything * implement skip for UnsortedGather * Some more wiring * Attempt at hacking up execute for DependencyProxy * Compilation fixes! * Fixed AqlCall compilation and unified execute and executeForDependency * Fixed WAITIGN assertion * Sonderlocken. * Foo * Make compile * Make tests pass. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Move ModificationExecutors to new interface (#11165) * Move ModificationExecutors to new interface * added inputRange method to inputMatrix class, added switch between DataRange Input <-> Matrix for ModificationExecutors * forgot to enable allrows modification executor in fastForwardType * removed obsolete comment * removed obsolete comment Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery operations stack materialize executor (#11192) * removed obsolete comment, added produceRows and skipRowsRange, enabled MaterializeExecutor as newStyleExecutor * forgot to use advanceRow, fetch input during skipRowsRange * removed old implementations * removed comment, fixed a logical loop error * skip could be simplified as well * temp workaround Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute remote (#11197) * Clarified comment * Updated comment * Added two static strings * Avoid unintentional streaming operator of string_view as a container * Added parser for execute rest call * Fixed ambiguous overload in ResultT * Added execute and executeForClient to ExecutionEngine * Implemented executeForClient in RestAqlHandler * Add default value * Updated comment * Changed error attribute * Added API toggle to RemoteExecutor * Fixed compile error * Fixed duplicate log id * Try to make MSVC work * Moved AqlExecuteResult to a separate file * Moved AqlExecuteResult to a separate file * Implemtented execute in Remote; except for (de)serialization, which is still missing * Tried to fix MSVC compile error * Revert "Tried to fix MSVC compile error" This reverts commit f6f43b2c7e4bfe17dafe65f48bcd5c0b6e5c69a9. * Next try to fix MSVC * Implemented (de)serialization for RemoteExecutor::execute * Fixes * Added minimal AqlCall (de)serialization tests * Bugfix * Added minimal (de)serialization tests for AqlCallStack and AqlExecuteResult * Fixed test * Changed Serialization format a little * Bugfix * Initialize SingletonBlock immediately with an empty row * Revert "Next try to fix MSVC" This reverts commit 528c4c795d10ee0aa6686d143be1f6faa6e9b553. * Revert "Try to make MSVC work" This reverts commit ba7d9c072fccb969da1cdf045eadf49c297da8e9. * Work around MSVC shortcoming * Work around MSVC shortcoming * Attempt to fix windows compile issue Co-authored-by: Michael Hackstein <michael@arangodb.com> * Added test for distinct collect with random order. (#11184) * Feature/aql subquery execution block impl execute implementation subquery executor (#11196) * Intermediate commit, does not compile. * New style for SubqueryExecutor. * Removed bad assert. * Reset subsqueryInitialised in skip. Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execution block impl execute implementation count collect (#11172) * Move SubqueryStartExecutor to new interface * Diddle * Tests pass for SubqueryStartExecutor * Delete most of the SubqueryEndExecutor tests They are irrelevant now as they only tested shadowrow behaviour. Shadow Rows are now handled by ExecutionBlockImpl. * Skeleton tests * Junk * Cleanup and make work * Tweak ExecutorTestHelper to test pipelines * SubqueryStart/SubqueryEnd * Blafummel. * Modifications * Fixup tests * Address some comments * Fix. * Fix tests * Various fixes. * Made the branch compile on Mac * Intermediate commit, does not compile, required for merge * Fixed one of the SplicedSubqueryTests * Seperated ShadowRow handling for SubqueryStart/End executors to get a better overview and understanding on how it should behave. * Allow to use a fullCallStack in ExecutorTestHelper. * Added asserts on SubqueryEnd that is not allowed to be called with Skipped from upstream * Use AqlTestCase in SubqueryStartExecutorTest. Transformed first test to TestHelper pipeline * Let ExecutorTestHelper also test for shadowRows * Adapted SingleInput=>Data+Shadow test to testing Helepr and make it pass * Transformed existing SubqueryStart tests to Helper, fixed a test where there was no place for the ShadowRow to be written * Fixed Skip in SubqueryStartExecutor * Allow the Pipeline to add Consumer and dependency without fiddling with the Deque from outside * Added Skip adn ShadowRowForwarding tests to SubqueryStartExecutor * Fixed fullCount of SubqueryStart to be called properly. * Renamed isModificationSubquery -> isModificatioNode again * Consume more than one subquery en-block * Fixed debugReporting in SplicedSubqueryTest. Disabled the skip in subquery tests for now * Removed AQL debug log level from testing config * First version of Subquery Skip bypassing. Including 1 passing test. * Added disabled tests for skip, and proposed to do it later * Fixed reporting of FILTERED in fullCount of EnumerateCollectionExecutor * Enable calculationExecutor execute interface * Fixed compile issues * To Velocypack the modification Subquery value * Fixed illegal assert and reactivated failure tests * Fixed compile issue * Deactivated Calculation again * Added a testSuite for countCollectExecutor * Fixed Subquery Test for CountCollect * Added CountCollectImplementation * Added Compatibility Mode to AqlCallStack * Removed IdExecutor<void> the only feature it has can be moved into IDexecutor in general. This still has to be done * When falling back in BlocksWithClients use compatibility stack * Fixed EnumerateCollectionExecutor * Fixed fullCountBehaviour of EnumerateList. Also removed a unlimited,fullCount combination in IndexExecutor * Fixed endless loop while skipAl in OperationCursor * Fixed View Executor, which sends a no-limit no-offset upstream. * Fixed forwarding in NoResultsExecutor Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com> * Activate CalculationExecutors (#11177) * Fixed compile bug-due to hidden merge conflict * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Started removing old getSome/skipSome API. This version does not compile yet * Harvested more, however the old tests still exist and do not compile now. * Feature/aql subquery execution block impl execute implementation base bugfixes (#11201) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Fix memory leak in traversal (#11210) * AqlSubqueryExecutionBlockImplExecuteImplementation use moveValueInto instead of cloneValueInto (#11213) * Fix memory leak in traversal * Use move instead of clone for AqlValues * Make compile * Feature/aql subquery execution block impl execute implementation gather executors (#11202) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Feature/aql subquery execute parallel gather (#11215) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather Co-authored-by: Michael Hackstein <michael@arangodb.com> * Feature/aql subquery execute parallel gather 2 (#11221) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather * [WIP] Changed Api for MultiDepExecutors in ExecBlockImpl (not yet in the actual executors) * Moved AqlCallSet into a separate file * Changed SortingGather to use the new API * Changed ParallelUnsortedGather to use the new API * Changed UnsortedGather to use the new API * Moved AqlCall operator<< into .cpp file * Implement operator<< for AqlCallSet * Fix boolean mix-up * Fixed state machine: go to UPSTREAM when the AqlCallSet is not empty * Fixed assertion * Bugfix * SortingGather bugfixes * Added init() method to fix an assertion and cleanup * Removed unused variable * Fixed constrained sort * Fixed constrained sort #2 * Fix boolean mix-up * Remove old interface * Use call parameter for upstream request in produceRows * Remove more old interface code * Add skip methods to MultiAqlItemBlockInputRange * Skip in UnsortedGather * skip for UnsortedGather * Fix skip and upstream calls for UnsortedGather * skipRowsRange change * Remove useless comments * Moved multi-dep related code from ExeBlockImpl to MultiFetcher * Cleanup in SortingGather, implemented parallel fullCount * Try to fix a windows compile error * Simplify and extend skipRowsRange for UnsortedGatherExecutor * Made ParallelUnsortedGather actually parallel * Removed erroneous assertion * Undid erroneous change * Fixed MacOs compile. Also disabled tests for non-relevant AqlCallStacks. They will be removed * Fixed initialize Cursor for multi dependency blocks * Fixed fullCount case in parallel unsorted gather * Fixed fullCount upstream call of ParallelUnsortedGatherExecutor * Fixed fullCount in SortingGather * Windows \o/ if you cannot work properly with constexpr and static asserts, we do not let you do it! * Do not advance in Unsorted gather if there are still rows to skip * Add more comparison operators for AqlCall limits * Send clientCall limits to upstream in SortingGather * Improved fullCount in SortingGatherExectur * Disabled a cluster profile test. We now ask the RemoteNode more often if it already has data. It is a bit unclear to me if this is now better performance wise (<< i think so) or triggers undesired side effects * Helpless attempt to work around issues in stonage Visual Studio Compiler we are using. * Clearly adding an operator on a well defined type causes ambigousness on random basic types using the operator Co-authored-by: Michael Hackstein <michael@arangodb.com> Co-authored-by: Markus Pfeiffer <markus@arangodb.com> * Feature/aql subquery execution block impl execute implementation bypass skip (#11203) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subq…
jsteemann
added a commit
that referenced
this issue
May 25, 2020
…FilterNode) while planning registers
jsteemann
added a commit
that referenced
this issue
May 25, 2020
…FilterNode) while planning registers
KVS85
pushed a commit
that referenced
this issue
May 25, 2020
…FilterNode) while planning registers (#11676)
jsteemann
added a commit
that referenced
this issue
May 25, 2020
…FilterNode) while planning registers (#11675)
ObiWahn
added a commit
that referenced
this issue
May 26, 2020
9 tasks
jsteemann
added a commit
that referenced
this issue
Mar 12, 2021
This change can improve the performance of local sorts operations, e.g.
Baseline (3.7.9):
Query String (94 chars, cacheable: false):
FOR i IN 1..500000 LET value = CONCAT('testvalue-to-be-sorted', i) SORT value ASC RETURN value
Execution plan:
Id NodeType Calls Items Runtime [s] Comment
1 SingletonNode 1 1 0.00003 * ROOT
2 CalculationNode 1 1 0.00003 - LET #2 = 1 .. 500000 /* range */ /* simple expression */
3 EnumerateListNode 500 500000 0.08725 - FOR i IN #2 /* list iteration */
4 CalculationNode 500 500000 0.22722 - LET value = CONCAT("testvalue-to-be-sorted", i) /* simple expression */
5 SortNode 500 500000 2.05180 - SORT value ASC /* sorting strategy: standard */
6 ReturnNode 500 500000 0.02911 - RETURN value
Query Statistics:
Writes Exec Writes Ign Scan Full Scan Index Filtered Exec Time [s]
0 0 0 0 0 2.39644
With sort optimization (3.7.10):
Query String (94 chars, cacheable: false):
FOR i IN 1..500000 LET value = CONCAT('testvalue-to-be-sorted', i) SORT value ASC RETURN value
Execution plan:
Id NodeType Calls Items Runtime [s] Comment
1 SingletonNode 1 1 0.00002 * ROOT
2 CalculationNode 1 1 0.00003 - LET #2 = 1 .. 500000 /* range */ /* simple expression */
3 EnumerateListNode 500 500000 0.08755 - FOR i IN #2 /* list iteration */
4 CalculationNode 500 500000 0.26161 - LET value = CONCAT("testvalue-to-be-sorted", i) /* simple expression */
5 SortNode 500 500000 1.36070 - SORT value ASC /* sorting strategy: standard */
6 ReturnNode 500 500000 0.02864 - RETURN value
Query Statistics:
Writes Exec Writes Ign Scan Full Scan Index Filtered Exec Time [s]
0 0 0 0 0 1.73940
jsteemann
added a commit
that referenced
this issue
Mar 12, 2021
This change can improve the performance of local sorts operations, e.g.
Baseline (3.7.9):
Query String (94 chars, cacheable: false):
FOR i IN 1..500000 LET value = CONCAT('testvalue-to-be-sorted', i) SORT value ASC RETURN value
Execution plan:
Id NodeType Calls Items Runtime [s] Comment
1 SingletonNode 1 1 0.00003 * ROOT
2 CalculationNode 1 1 0.00003 - LET #2 = 1 .. 500000 /* range */ /* simple expression */
3 EnumerateListNode 500 500000 0.08725 - FOR i IN #2 /* list iteration */
4 CalculationNode 500 500000 0.22722 - LET value = CONCAT("testvalue-to-be-sorted", i) /* simple expression */
5 SortNode 500 500000 2.05180 - SORT value ASC /* sorting strategy: standard */
6 ReturnNode 500 500000 0.02911 - RETURN value
Query Statistics:
Writes Exec Writes Ign Scan Full Scan Index Filtered Exec Time [s]
0 0 0 0 0 2.39644
With sort optimization (3.7.10):
Query String (94 chars, cacheable: false):
FOR i IN 1..500000 LET value = CONCAT('testvalue-to-be-sorted', i) SORT value ASC RETURN value
Execution plan:
Id NodeType Calls Items Runtime [s] Comment
1 SingletonNode 1 1 0.00002 * ROOT
2 CalculationNode 1 1 0.00003 - LET #2 = 1 .. 500000 /* range */ /* simple expression */
3 EnumerateListNode 500 500000 0.08755 - FOR i IN #2 /* list iteration */
4 CalculationNode 500 500000 0.26161 - LET value = CONCAT("testvalue-to-be-sorted", i) /* simple expression */
5 SortNode 500 500000 1.36070 - SORT value ASC /* sorting strategy: standard */
6 ReturnNode 500 500000 0.02864 - RETURN value
Query Statistics:
Writes Exec Writes Ign Scan Full Scan Index Filtered Exec Time [s]
0 0 0 0 0 1.73940
maierlars
pushed a commit
that referenced
this issue
Nov 4, 2021
…) (#14907) * Took over some refactoring from a previous branches * Took over some more changes * Fixes * Fixed catchToResultT * Re-added missing function * Moved default initializations * Fixed C++ tests * Improved the assertion on single server a little * Suppressed warning * Fixed remaining merge conflicts * Added comments, assertions, and moved initializations in transaction::Options from member initializer list to default member initializers * Fixed compile error * [WIP] Try to fix google test error * Fix test * Fix non-maintainer build. Co-authored-by: mpoeter <manuel@manuel-poeter.at>
hkernbach
added a commit
that referenced
this issue
Jul 20, 2023
CryptoNinjaGeek
pushed a commit
that referenced
this issue
Jul 25, 2023
* improve mem tracking * remove unused function, add mem tracking for projections * more tracking * clang format * just a test * next attempt * fix dummy code * clang format * properly init _vertexShards * properly pass allocator * adjust comment * rollback some changes as they do not work * prepare Expression for resourceMonitor to be able to track memory here * attempt to fix ci failures * clang format * this should not be required * take care of mem use with parallelism * remove logging * init commit for expressions and aqlnamepaths * fix issues found by tests * fix tests * Apply suggestions from code review Remove not needed comments / cod e * Update arangod/Graph/BaseOptions.cpp * Update arangod/Graph/BaseOptions.h * clang format * Update arangod/Graph/BaseOptions.cpp Co-authored-by: Jan <jsteemann@users.noreply.github.com> * use std::move * code review: try catch * documentFastPathLocal now uses std::string_view instead of std::string const& - this change unfortunately leads to a lot of additional function signature changes * clang format * clang format * memtrack AttributeNamePath * move _usedBytesByData out of the union * code review: do not use a scope guard if we do have a try catch block already. directly clear structs after decrease of memory * more use of absl::StrCat, prevent string copy where not required * fix typo * prevent string copy. Implementation now support std::string_view * adjust code to code conventions * remove not required includes to speed up compilation * re-establish default constructor (deleted) * prevent string copy in CollLevelMap * stick to code conventions * fix build, fix wrong projections build up in the EdgeCursor * fix wrong projections build up in the EdgeCursor again * fix wrong projections build up in the pregel based EdgeCursor ... again * fix default constructor in AttributeNamePath, re-establish default deleted constructor * fix test binary build * do NOT use current to get the resMonitor reference - as current ptr is changed dynamically and runs into nullptr * extended memory tracking usage * do not create a vector of strings - use string_view instead - also removed some not required includes * reset _usedBytesByData in error case * clang format * clang format * move helper method into anonymous namespace * attempt to fix windows build * Revert " attempt to fix windows build" This reverts commit c80999c. * attempt to fix windows build #2 * fix false positive cppcheck * do not inplace build std::vector of string_views * clang format * move p instead of rebuilding it * simpler constructor * clangormat --------- Co-authored-by: Jan <jsteemann@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Leider segfaulted die neuste Version bei mir Habe gepulled, make clean, autoconf, ./configure..., make gemacht. Alles ohne Probleme. Aber wenn ich dann mit ./avocado startet kommt:
"Segmentation fault: 11"
Habe es dann mit valgrind gestartet. Erstaunlicherweise fährt er da den Server hoch. Aber im Log ist eine riesige Anzahl von Fehlern. Weißt du, was das zu bedeuten hat?
https://gist.github.com/a6b7b8bd5aea05541c17
The text was updated successfully, but these errors were encountered: