Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(flow): shared in-memory state for dataflow operator #3508

Merged
merged 12 commits into from Mar 19, 2024

Conversation

discord9
Copy link
Contributor

@discord9 discord9 commented Mar 14, 2024

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

#3187

What's changed and what's your intention?

A shared state of key-value pair for various state
in dataflow execution, called Arrangement

i.e: Mfp operator with temporal filter need to store it's future output so that it can add now, and delete later. To get all needed updates in a time span, use get_updates_in_range

And reduce operator need full state of it's output, so that it can query(and modify by calling apply_updates)
existing state, also need a way to expire keys. To get a key's current value, use get with time being now
so it's like:
mfp operator -> arrange(store futures only, no expire) -> reduce operator <-> arrange(full, with key expiring time) -> output

Note the two way arrow between reduce operator and arrange, it's because reduce operator need to query existing state
and also need to update existing state.

This state will be used in building&execution dataflow graph

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR does not require documentation updates.

@github-actions github-actions bot added the Doc not needed This PR do not impact docs label Mar 14, 2024
@discord9 discord9 marked this pull request as ready for review March 14, 2024 06:34
Copy link

codecov bot commented Mar 14, 2024

Codecov Report

Attention: Patch coverage is 95.29703% with 19 lines in your changes are missing coverage. Please review.

Project coverage is 85.00%. Comparing base (b85d7bb) to head (256a510).
Report is 13 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3508      +/-   ##
==========================================
- Coverage   85.30%   85.00%   -0.31%     
==========================================
  Files         904      907       +3     
  Lines      150170   151406    +1236     
==========================================
+ Hits       128106   128704     +598     
- Misses      22064    22702     +638     

src/flow/src/repr.rs Outdated Show resolved Hide resolved
src/flow/src/utils.rs Outdated Show resolved Hide resolved
src/flow/src/utils.rs Outdated Show resolved Hide resolved
src/flow/src/utils.rs Show resolved Hide resolved
src/flow/src/utils.rs Outdated Show resolved Hide resolved
src/flow/src/repr.rs Outdated Show resolved Hide resolved
src/flow/src/utils.rs Show resolved Hide resolved
src/flow/src/utils.rs Outdated Show resolved Hide resolved
src/flow/src/utils.rs Outdated Show resolved Hide resolved
src/flow/src/utils.rs Outdated Show resolved Hide resolved
src/flow/src/utils.rs Outdated Show resolved Hide resolved
src/flow/src/utils.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@zhongzc zhongzc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@waynexia waynexia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@waynexia waynexia added this pull request to the merge queue Mar 19, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 19, 2024
@zhongzc zhongzc added this pull request to the merge queue Mar 19, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 19, 2024
@fengjiachun fengjiachun added this pull request to the merge queue Mar 19, 2024
Merged via the queue into GreptimeTeam:main with commit 2c115bc Mar 19, 2024
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Doc not needed This PR do not impact docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants