Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce the amount of duplicate code in per-dataflow-worker metrics #7690

Merged

Conversation

antifuchs
Copy link
Contributor

This PR reduces the number of duplicate names you have to type if you introduce a Command to the dataflow server, and reduces the number of metrics that are present but never get updated. We do this by making the cache/counters live on an array and index that array by the Command kind that was processed.

@antifuchs
Copy link
Contributor Author

One thing this requires is to de-strictify the rejection of missing_docs on the dataflow crate, until the PR at Soft/enum-kinds#3 goes in.

@antifuchs antifuchs force-pushed the asf/oaoo-commands-processed-metrics branch from add0492 to 30b1298 Compare August 4, 2021 17:36
Copy link
Member

@benesch benesch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is sooo good! Tyvm!

Copy link
Contributor

@elindsey elindsey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome!

src/dataflow/src/server.rs Outdated Show resolved Hide resolved
@antifuchs antifuchs force-pushed the asf/oaoo-commands-processed-metrics branch 2 times, most recently from 7cbcb6e to 82d57e7 Compare August 4, 2021 18:56
@antifuchs antifuchs changed the title Reduce the number of duplication in per-dataflow-worker metrics Reduce the amount of duplicate code in per-dataflow-worker metrics Aug 4, 2021
By combining enum_kinds and num_enum, we can change the worker command
representation to be an array of counters & values by which we increment
them. That reduces the number of names we have to duplicate in code, and
requires one fewer match statement on Command variants.

It also gets rid of some metrics that were represented but never got
incremented.
@antifuchs antifuchs force-pushed the asf/oaoo-commands-processed-metrics branch from 82d57e7 to 8dfab6a Compare August 4, 2021 20:12
@antifuchs antifuchs merged commit 310dfa1 into MaterializeInc:main Aug 4, 2021
@antifuchs antifuchs deleted the asf/oaoo-commands-processed-metrics branch August 4, 2021 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants