Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove COUNT from logging sources/views #2208

Closed
frankmcsherry opened this issue Mar 4, 2020 · 1 comment
Closed

Remove COUNT from logging sources/views #2208

frankmcsherry opened this issue Mar 4, 2020 · 1 comment
Assignees
Labels
C-musing Category: not-yet-actionable discussions T-memory Theme: memory
Projects

Comments

@frankmcsherry
Copy link
Contributor

Many of the logging sources and views have a COUNT, associating a number with various other attributes. For example, we track the elapsed number of nanoseconds for each operator, the counts for each power-of-two elapsed scheduling event, counts of number of records and batches in arrangements.

We could not do that, and just leave the uncounted data as the collection.

The benefit here is that the count adds additional work, additional state, and additional churn when data change. If the user wants to count the results they can, and instead if the user wants to project out certain columns (often: worker) they wouldn't have to write the SUM to perform the accumulation.

There are several things we do badly at the moment that might prevent this from working well yet, mostly about how well we could use raw arrangements in places where we currently use arrangements with a value (e.g. a count).

@frankmcsherry frankmcsherry added the C-musing Category: not-yet-actionable discussions label Mar 4, 2020
@frankmcsherry frankmcsherry added the T-memory Theme: memory label Sep 3, 2020
@antiguru antiguru self-assigned this Sep 1, 2021
@uce uce added this to Needs Triage in Compute via automation Sep 1, 2021
@uce uce moved this from Needs Triage to To do in Compute Sep 1, 2021
@antiguru
Copy link
Member

antiguru commented Oct 7, 2021

All but some materialized logs do not explicitly count anymore. I'll go ahead and close this issue because I don't expect the remaining logs to have a large volume.

@antiguru antiguru closed this as completed Oct 7, 2021
Compute automation moved this from To do to Done Oct 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-musing Category: not-yet-actionable discussions T-memory Theme: memory
Projects
No open projects
Development

No branches or pull requests

2 participants