Implement reduce and dependency task logic

[sandreae]

reduce

• fetch operations by document_id
• fetch operations by document_view_id
• materialise document
• materialise document_views
• store specific document view
• store documents current view
• return next dependency tasks
• error handling
• tests
• make requested review changes

dependency

• retrieve relation graph from db (including parent and children) check child pinned relations are materialised
• return reduce tasks for every missing child dependency
• error on deleted documents
• error handling
• tests

📋 Checklist

• Add tests that cover your changes
• Add this PR to the Unreleased section in CHANGELOG.md
• Link this PR to any issues it closes
• New files contain a SPDX license header

[codecov]

Codecov Report

Merging #144 (b0d0124) into development (32d12e7) will increase coverage by 0.95%.
The diff coverage is 96.05%.

@@               Coverage Diff               @@
##           development     #144      +/-   ##
===============================================
+ Coverage        87.63%   88.59%   +0.95%     
===============================================
  Files               42       42              
  Lines             2055     2280     +225     
===============================================
+ Hits              1801     2020     +219     
- Misses             254      260       +6     

Impacted Files	Coverage Δ
aquadoggo/src/db/provider.rs	87.50% <ø> (ø)
aquadoggo/src/db/stores/document.rs	93.46% <ø> (-0.04%)	⬇️
aquadoggo/src/materializer/input.rs	100.00% <ø> (+100.00%)	⬆️
aquadoggo/src/materializer/tasks/schema.rs	0.00% <ø> (ø)
aquadoggo/src/materializer/worker.rs	88.34% <ø> (ø)
aquadoggo/src/materializer/tasks/reduce.rs	93.80% <93.80%> (+93.80%)	⬆️
aquadoggo/src/materializer/tasks/dependency.rs	98.26% <98.26%> (+98.26%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 32d12e7...b0d0124. Read the comment docs.

[adzialocha]

Just writing some comments while we're going as I was very curious 🤩

This is really really really beautiful, the tasks are so small and powerful!

[adzialocha]

The error handling is funny as we probably end up only having one critical error type or Ok(None) 🤣

There could be two approaches to the semantics of the return values:

• Ok(...) Task finished without any critical problem
• Err(Critical) Task failed with a critical problem

.. or:

• Ok(...) Task finished and did it's job
• Err(Failure) Task finished but couldn't do it's job
• Err(Critical) Task failed with a critical problem

[sandreae]

Thanks for all the comments @adzialocha! I will make those changes.

When you have time, you can also look over the dependency task logic. It seems a little more nuanced than I originally imagined. The current logic is:

• for "parent" dependencies which have been materialised (the only kind) => dispatch a dependency task
• for "child" dependencies which have been materialised => do nothing
• for "child" dependencies which have not been materialised => dispatch a reduce task
• for documents or document views which have all their child dependencies met => dispatch a schema task

"parent dependency" = a document relating to the document being processed by the task
"child dependency" = a document which the tasks' document relates to

[adzialocha]

I haven't reflected yet on the new schema store in this context but I think we currently don't need a schema task with this db layout.

I'd add it as soon as we build the service 👍

For now we can build the schema through schema store on every request. That's naive of course but it will be fine for now!

Later we can add the schema service which will be used to query schema instances in memory (as a caching layer), it will be populated during node startup and filled with new schemas in a schema task (only if the wanted schema was missing during startup). Later we can deprecate / delete schemas in the task when the related document got deleted.

[sandreae]

but I think we currently don't need a schema task with this db layout

Yep, that makes sense.

Later we can add the schema service which will be used to query schema instances in memory (as a caching layer), it will be populated during node startup and filled with new schemas in a schema task (only if the wanted schema was missing during startup). Later we can deprecate / delete schemas in the task when the related document got deleted.

Cool, this is what I imagined we'd be doing as well 👍

[adzialocha]

• for "parent" dependencies which have been materialised (the only kind) => dispatch a dependency task

This looks really close to the specification we have in the Whimsical 👍 except of this one point where I would expect this to dispatch a reduce task (which again will dispatch a dependency task), we might otherwise miss out on materialising the specific document view of this parent (it might be pinned as well from another parent). (that should itself being wrong after meeting with @sandreae, we actually don't need to check for parents at all)

[cafca]

I added some documentation for the dependency task, I think the same could be very helpful for the reducer (but dinner is ready so I stop now ;) ). The control flow is quite deeply nested with multiple matches. Flattening that out would probably help make this maintainable.

Overall this is good to go though :)

[sandreae]

Thanks for the review @cafca, I improved the docs and did a little flattening, I like it! Think I covered everything now.

[adzialocha]

Alright! I did some crazy rebase-kung-fu and now this is ready for merging into development! Also, I had a go on @cafca's idea to improve the control flow nesting in reduce_task by factoring all into smaller methods. Please have a (hopefully) last look, especially @sandreae, I hope I didn't mess with your code too much.

[sandreae]

Thanks! That's great 👍 for some reason I thought @cafca 's comment was about the dependency task....