Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(Cosmos): eqx query, Category.TryHydrateTip #434

Merged
merged 10 commits into from
Jan 12, 2024
Merged

Conversation

bartelink
Copy link
Collaborator

@bartelink bartelink commented Dec 30, 2023

  • Add --index-unfolds / -U option to eqx init to index unfolds
  • Add eqx query facility to validate costs / correctness of the indexes
  • Add CosmosStoreCategory.TryHydrateTip to generate a memento based on u (and optionally the _etag if it's for Transact purposes)
  • Add TryLoad shortcut (based on TryHydrateTip, but solely returns the state)
  • Reinstate Load = LoadOption.FromMemento from feat!: Remove ISyncContext.CreateMemento #413 (but not ISyncContext.CreateMemento)

Future/unanswered questions:

  • Should CosmosStoreContainer expose a Querying mechanism (and should that let you express filters via IQueryable<UnfoldBody>)

@bartelink bartelink force-pushed the cosmos-query branch 2 times, most recently from a3ad9b3 to 762eaf3 Compare December 31, 2023 01:23
@bartelink bartelink changed the base branch from master to shouldcompress December 31, 2023 01:24
@bartelink bartelink changed the title feat(Cosmos): eqx query feat(Cosmos): eqx query + (Query|Tansact)Many Dec 31, 2023
@bartelink bartelink changed the title feat(Cosmos): eqx query + (Query|Tansact)Many feat(Cosmos): eqx query + (Query|Transact)Many Dec 31, 2023
@nordfjord
Copy link
Contributor

Would be good to see an example of how to use it in actual applications as well

@bartelink bartelink changed the base branch from shouldcompress to syncexp January 8, 2024 23:50
@bartelink bartelink force-pushed the syncexp branch 4 times, most recently from d2ee470 to af658f5 Compare January 9, 2024 16:55
@bartelink bartelink force-pushed the cosmos-query branch 2 times, most recently from 13adb22 to 52630c6 Compare January 9, 2024 17:22
@bartelink bartelink changed the title feat(Cosmos): eqx query + (Query|Transact)Many feat(Cosmos): eqx query + Category.TryHydrate/Load Jan 9, 2024
@bartelink bartelink force-pushed the syncexp branch 2 times, most recently from e0cba15 to 1d11644 Compare January 9, 2024 17:40
Base automatically changed from syncexp to master January 9, 2024 23:05
@bartelink bartelink force-pushed the cosmos-query branch 2 times, most recently from 3296c62 to dde8bd6 Compare January 10, 2024 22:44
@bartelink bartelink force-pushed the cosmos-query branch 3 times, most recently from ba142d1 to f05f3d3 Compare January 11, 2024 13:34
@bartelink bartelink changed the base branch from master to fix-438 January 11, 2024 13:43
@bartelink
Copy link
Collaborator Author

bartelink commented Jan 11, 2024

Learnings/decisions relative to the original overview at the outset

  • Main difference/learning/variance is that providing a function to generate a memento means no TransactMany/QueryMany API variant.
  • Searching based on u/[]/c means no need to filter based on id = "-1"
  • eqx query shows 3 modes (inc retrieving the p values); How to use the p value is an external concern

Original proposal:

We can:

  • pay more on writes to maintain 2 more indexed things (affects write ops, but there's only one per stream and the first path should remain stable)
    • no, have not assessed the damage in terms of write costs
    • note that the /d/* means only data in uncompressed folds, so should be relatively low
  • add uncompressed unfolds (so we can query against u.d.<FieldName>)
  • identify streams to read or transact against by category name + an unfold name + arbitrary criteria targeting uncompressed unfolds for 2.8RU
  • QueryMany: pull back u (and, optionally p) and provide a paged result based on hydrating the State from the (potentially compressed) Snapshot and rendering that (can convey Version and State)
    • p lets you identify the stream, but that's optional if it's state is sufficient
  • TransactMany: pull back u, p, _etag and then (potentially concurrently) do a Transact read and/or write roundtrip based on only that input
    • could also do a null update to freshen the Cache
  • for streams that are AccessStrategy.Unoptimized (events only), it would also work
    • would have to get e in lieu of u
    • as there is no u, there's no ability to filter based on uncompressed folds
    • would also need to filter by id = "-1" (to rule out other event batches), or retrieve those too and sort by c.i (if you know you'll hydrate the bulk of them)

@bartelink bartelink changed the title feat(Cosmos): eqx query + Category.TryHydrate/Load feat(Cosmos): eqx query + Category.TryHydrate Jan 11, 2024
@bartelink bartelink changed the title feat(Cosmos): eqx query + Category.TryHydrate feat(Cosmos): eqx query + Category.TryHydrate/TryLoad Jan 11, 2024
@bartelink bartelink marked this pull request as ready for review January 11, 2024 15:26
bartelink added a commit that referenced this pull request Jan 11, 2024
bartelink added a commit that referenced this pull request Jan 11, 2024
Base automatically changed from fix-438 to master January 11, 2024 15:52
@bartelink bartelink changed the title feat(Cosmos): eqx query + Category.TryHydrate/TryLoad feat(Cosmos): eqx query + Category.TryHydrateTip Jan 11, 2024
@bartelink bartelink changed the title feat(Cosmos): eqx query + Category.TryHydrateTip feat(Cosmos): eqx query, Category.TryHydrateTip Jan 12, 2024
@bartelink bartelink merged commit 6bf5e83 into master Jan 12, 2024
4 of 5 checks passed
@bartelink bartelink deleted the cosmos-query branch January 12, 2024 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants