Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gauges can't be created from dynamic metrics #15718

Closed
blazember opened this issue Oct 8, 2019 · 0 comments · Fixed by #15782
Closed

Gauges can't be created from dynamic metrics #15718

blazember opened this issue Oct 8, 2019 · 0 comments · Fixed by #15782
Assignees
Milestone

Comments

@blazember
Copy link
Contributor

@blazember blazember commented Oct 8, 2019

Gauges read their values only from ProbeInstances that don't exist for dynamic metrics, leading to rendering 0 value for the affected gauges. This was noticed in the HealthMetric logs that reported clientCount as zero despite the non-zero number of connected clients.

See this extracted snippet from a HealthMonitor log:

clientEndpoint.count=944, connection.active.count=0, client.connection.count=0
@blazember blazember added this to the 4.0 milestone Oct 8, 2019
@blazember blazember self-assigned this Oct 8, 2019
blazember added a commit to blazember/hazelcast that referenced this issue Oct 16, 2019
This change adds dynamic metric support for gauges. Prior to this change
gauges were updated from static metrics only.

Dynamic metrics are supported by introducing MetricValueCatcher that
catches the values of the dynamic metrics during the collection cycle.

The dynamic metrics provided as concrete values by calling
`collect(MetricTagger, String, ProbeLevel, ProbeUnit, long)` or
`collect(MetricTagger, String, ProbeLevel, ProbeUnit, double)` on
`MetricsCollectionContext` are cached by values, so the `gauge.read()`
calls return the last updated value for the metric.

The dynamic metrics provided through collected objects by calling
`MetricsCollectionContext#collect(MetricTagger, Object)` are cached by
source object. Once the given metric source object is discovered in a
collection cycle, the object gets cached inside the gauge and every
`gauge#read()` returns the most current value of the metric. This
caching doesn't prevent the source object to get GCd since the gauges
reference it through `WeakReference`s.

Besides allowing the metric source object to get GCd, the gauges that
are not "visited" during a metric collection cycle get reset therefore
no stale values are read once the backing dynamic  metrics sources
disappear. This is true even if the metric source object is still alive.

Fixes hazelcast#15718
blazember added a commit to blazember/hazelcast that referenced this issue Oct 16, 2019
This change adds dynamic metric support for gauges. Prior to this change
gauges were updated from static metrics only.

Dynamic metrics are supported by introducing MetricValueCatcher that
catches the values of the dynamic metrics during the collection cycle.

The dynamic metrics provided as concrete values by calling
`collect(MetricTagger, String, ProbeLevel, ProbeUnit, long)` or
`collect(MetricTagger, String, ProbeLevel, ProbeUnit, double)` on
`MetricsCollectionContext` are cached by values, so the `gauge.read()`
calls return the last updated value for the metric.

The dynamic metrics provided through collected objects by calling
`MetricsCollectionContext#collect(MetricTagger, Object)` are cached by
source object. Once the given metric source object is discovered in a
collection cycle, the object gets cached inside the gauge and every
`gauge#read()` returns the most current value of the metric. This
caching doesn't prevent the source object to get GCd since the gauges
reference it through `WeakReference`s.

Besides allowing the metric source object to get GCd, the gauges that
are not "visited" during a metric collection cycle get reset therefore
no stale values are read once the backing dynamic  metrics sources
disappear. This is true even if the metric source object is still alive.

Fixes hazelcast#15718
blazember added a commit to blazember/hazelcast that referenced this issue Oct 18, 2019
This change adds dynamic metric support for gauges. Prior to this change
gauges were updated from static metrics only.

Dynamic metrics are supported by introducing MetricValueCatcher that
catches the values of the dynamic metrics during the collection cycle.

The dynamic metrics provided as concrete values by calling
`collect(MetricTagger, String, ProbeLevel, ProbeUnit, long)` or
`collect(MetricTagger, String, ProbeLevel, ProbeUnit, double)` on
`MetricsCollectionContext` are cached by values, so the `gauge.read()`
calls return the last updated value for the metric.

The dynamic metrics provided through collected objects by calling
`MetricsCollectionContext#collect(MetricTagger, Object)` are cached by
source object. Once the given metric source object is discovered in a
collection cycle, the object gets cached inside the gauge and every
`gauge#read()` returns the most current value of the metric. This
caching doesn't prevent the source object to get GCd since the gauges
reference it through `WeakReference`s.

Besides allowing the metric source object to get GCd, the gauges that
are not "visited" during a metric collection cycle get reset therefore
no stale values are read once the backing dynamic  metrics sources
disappear. This is true even if the metric source object is still alive.

Fixes hazelcast#15718
blazember added a commit that referenced this issue Oct 24, 2019
This change adds dynamic metric support for gauges. Prior to this change
gauges were updated from static metrics only.

Dynamic metrics are supported by introducing `MetricValueCatcher` that
catches the values of the dynamic metrics during the collection cycle.

The dynamic metrics provided as concrete values by calling
`collect(MetricTagger, String, ProbeLevel, ProbeUnit, long)` or
`collect(MetricTagger, String, ProbeLevel, ProbeUnit, double)` on
`MetricsCollectionContext` are cached by values, so the `gauge.read()`
calls return the last updated value for the metric.

The dynamic metrics provided through collected objects by calling
`MetricsCollectionContext#collect(MetricTagger, Object)` are cached by
source object. Once the given metric source object is discovered in a
collection cycle, the object gets cached inside the gauge and every
`gauge#read()` returns the most current value of the metric. This
caching doesn't prevent the source object to get GCd since the gauges
reference it through `WeakReference`s.

Besides allowing the metric source object to get GCd, the gauges that
are not "visited" during a metric collection cycle get reset therefore
no stale values are read once the backing dynamic  metrics sources
disappear. This is true even if the metric source object is still alive.

Fixes #15718
Fixes hazelcast/hazelcast-enterprise/issues/3216
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.