Add sum of squared deviation estimate #341
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dashpole
force-pushed
the
sum_of_squared_deviation
branch
from
2405768
to
d185aec
Compare
dashpole
force-pushed
the
sum_of_squared_deviation
branch
9 times, most recently
from
b94d90e
to
523ece6
Compare
|
This is ready for review |
aabmass
reviewed
Apr 18, 2022
|
The assumptions the GMP algorithm make seem unreasonable. I'm thinking for our regular GCM exporter, we should just not send SOSD at all, what do you think? |
|
The best case I can make for generally setting SOSD is that it isn't an optional field. We are essentially always setting it to 0. The estimate here seems better than setting it to 0, although could possibly be confusing and misleading if it is too far off... Alternatively, I can make this optional and off by default. But users may want to set this if they want to graph it for some reason. I do find it somewhat odd that GMP sets it given that it isn't part of a prom histogram... Let me know what direction you think is best, and we can do that. |
dashpole
force-pushed
the
sum_of_squared_deviation
branch
from
523ece6
to
fc0d34b
Compare
dashpole
force-pushed
the
sum_of_squared_deviation
branch
from
fc0d34b
to
4b60db0
Compare
|
Example from the integration test run. The higher lines (~6) are with the estimated SOSD. The lower line is with a zero SOSD. The higher line is generally closer to accurate, although both are technically not correct. SOSD does not appear to impact any percentiles or the mean or count aggregations. It only matters when graphing the standard deviation.
|
|
This doesn't enable it by default anymore. |
|
Updated to not set SOSD by default. The enable config option is only useable by other exporters (e.g. GMP), and is not settable by users. |
aabmass
approved these changes
Apr 20, 2022
damemi
added a commit
that referenced
this pull request
Apr 29, 2022
* Do not use IsZero on timestamps obtained from pdata (#336) * do not use IsZero on timestamps obtained from pdata * send summary counts as doubles instead of integers (#338) * drop data points with no value set (#339) * handle NaN sums in histograms and exponential histograms (#340) * Fix broken links, typos in docs (#345) * Update fixtures based on upstream changes to the prometheus receiver (#346) * update fixtures based on upstream changes to the prometheus receiver * feat(propagator): create a new one-way propagator (#343) * feat(propagator): create a new one-way propagator CloudTraceOneWayPropagator moves trace info from X-cloud-trace-context into 'traceparent'. * Apply suggestions from code review Co-authored-by: David Ashpole <dashpole@google.com> * Update propagator to extract both, emit only traceparent Users will no longer need to create a composite propagator, and there's a clear precedence order between trace headers. Co-authored-by: David Ashpole <dashpole@google.com> * Demonstrate that integration tests aren't catching failures (#349) * demonstrate that integration tests aren't catching failures * Update dependencies (#347) update dependencies * address remaining nits from #347 (#351) * prepare for release (#353) * add integration test with metrics from the prometheus receiver (#348) * add make commands for release (#354) * Send timeseries synchronously (#358) * send timeseries syncronously * update docs, and fix existing broken test * Drop histograms without a sum (#359) * update opentelemetry collector to v0.49.0 * drop histograms without a sum * Add sum of squared deviation estimate (#341) * add sum of squared deviation estimate * Add missing otel attributes to Spans exported to Cloud Trace (#365) * Add instrumentation library to span attributes. * Add test to ensure resource + instrumentation-scope labels show up in spans. * Expand testing. * Move resource mapping into shared module, and add monitored resource labels to trace spans. * Do the go mod tidy shuffle. * Fix docker build to include new shared module. * Bump back to go 1.17 * Migrate to SDK version of semantic convention constants instead of Collector's. * Clean up documents. * Normalize histograms, summaries, and exponential histograms (#360) * normalize remaining cumulative data points * use the timeout passed to the metrics exporter (#369) * make gotidy * Resolve conflicts from missed file Co-authored-by: David Ashpole <dashpole@google.com> Co-authored-by: Marc Dougherty <muncus@users.noreply.github.com> Co-authored-by: Josh Suereth <joshuasuereth@google.com>
damemi
added a commit
that referenced
this pull request
Apr 29, 2022
* Do not use IsZero on timestamps obtained from pdata (#336) * do not use IsZero on timestamps obtained from pdata * send summary counts as doubles instead of integers (#338) * drop data points with no value set (#339) * handle NaN sums in histograms and exponential histograms (#340) * Fix broken links, typos in docs (#345) * Update fixtures based on upstream changes to the prometheus receiver (#346) * update fixtures based on upstream changes to the prometheus receiver * feat(propagator): create a new one-way propagator (#343) * feat(propagator): create a new one-way propagator CloudTraceOneWayPropagator moves trace info from X-cloud-trace-context into 'traceparent'. * Apply suggestions from code review Co-authored-by: David Ashpole <dashpole@google.com> * Update propagator to extract both, emit only traceparent Users will no longer need to create a composite propagator, and there's a clear precedence order between trace headers. Co-authored-by: David Ashpole <dashpole@google.com> * Demonstrate that integration tests aren't catching failures (#349) * demonstrate that integration tests aren't catching failures * Update dependencies (#347) update dependencies * address remaining nits from #347 (#351) * prepare for release (#353) * add integration test with metrics from the prometheus receiver (#348) * add make commands for release (#354) * Send timeseries synchronously (#358) * send timeseries syncronously * update docs, and fix existing broken test * Drop histograms without a sum (#359) * update opentelemetry collector to v0.49.0 * drop histograms without a sum * Add sum of squared deviation estimate (#341) * add sum of squared deviation estimate * Add missing otel attributes to Spans exported to Cloud Trace (#365) * Add instrumentation library to span attributes. * Add test to ensure resource + instrumentation-scope labels show up in spans. * Expand testing. * Move resource mapping into shared module, and add monitored resource labels to trace spans. * Do the go mod tidy shuffle. * Fix docker build to include new shared module. * Bump back to go 1.17 * Migrate to SDK version of semantic convention constants instead of Collector's. * Clean up documents. * Normalize histograms, summaries, and exponential histograms (#360) * normalize remaining cumulative data points * use the timeout passed to the metrics exporter (#369) Co-authored-by: David Ashpole <dashpole@google.com> Co-authored-by: Marc Dougherty <muncus@users.noreply.github.com> Co-authored-by: Josh Suereth <joshuasuereth@google.com>
This was referenced May 12, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is based on how google prometheus-engine calculates the sum of squared deviations. I need to double-check the implications of sending an estimate without having an exact SOSD.