v1.102.0-rc1

Released at 2024-06-07

Update note 1: the -remoteWrite.multitenantURL command-line flag at vmagent was removed starting from this release. This flag was deprecated since v1.96.0. Use -enableMultitenantHandlers instead, as it is easier to use and combine with multitenant URL at vminsert. See these docs for details.

Update note 2: the -streamAggr.dropInputLabels command-line flag at vmagent was renamed to -remoteWrite.streamAggr.dropInputLabels. -streamAggr.dropInputLabels is now used for global streaming aggregation.

Update note 3: the -maxLabelValueLen command-line flag default value was changed from 16kB to 1kB. It may lead to truncating of labels with enormous values.

• SECURITY: upgrade Go builder from Go1.22.2 to Go1.22.4. See the list of issues addressed in Go1.22.3 and Go1.22.4.

• SECURITY: upgrade base docker image (Alpine) from 3.19.1 to 3.20.0. See alpine 3.20.0 release notes.

• SECURITY: add release images built from scratch image. Such images could be more preferable for using in environments with higher security standards. See this pull request.

• FEATURE: dashboards/single: support selecting of multiple instances on the dashboard. See this issue for details.

• FEATURE: dashboards/single: properly display version in the Stats row for the custom builds of VictoriaMetrics.

• FEATURE: dashboards/single: add Network Usage panel to Resource Usage row.

• FEATURE: dashboards/operator, dashboards/backupmanager and dashboard/tenant-statistic: update dashboard to be compatible with Grafana 10+ version.

• FEATURE: dashboards/cluster: add new panel Concurrent selects to vmstorage row. The panel will show how many ongoing select queries are processed by vmstorage and should help to identify resource bottlenecks. See panel description for more details.

• FEATURE: dashboards: use $__interval variable for offsets and look-behind windows in annotations. This should improve precision of restarts and version change annotations when zooming-in/zooming-out on the dashboards.

• FEATURE: vmagent: support aggregation and deduplication configs before replicating data to configured -remoteWrite.url destinations. This saves CPU and memory resources when incoming data needs to be aggregated or deduplicated once and then replicated to multiple destinations. See this issue.

• FEATURE: vmagent: add service discovery support for Vultr. See these docs and this issue.

• FEATURE: vmagent: allow configuring -remoteWrite.disableOnDiskQueue and -remoteWrite.dropSamplesOnOverload cmd-line flags per each -remoteWrite.url. See this pull request. Thanks to @rbizos for implementation!

• FEATURE: vmagent: add labels path and url to metrics vmagent_remotewrite_push_failures_total and vmagent_remotewrite_samples_dropped_total. Now number of failed pushes and dropped samples can be tracked per -remoteWrite.url.

• FEATURE: stream aggregation: add rate_sum and rate_avg aggregation outputs.

• FEATURE: stream aggregation: reduce the number of allocated objects in heap during deduplication and aggregation. The change supposed to reduce pressure on Garbage Collector, as it will need to scan less objects. See this pull request.

• FEATURE: vmalert: add datasource.idleConnTimeout, remoteWrite.idleConnTimeout and remoteRead.idleConnTimeout flags. These flags are set to 50s by default and should reduce the probability of broken pipe or connection reset by peer errors in vmalert logs. See this issue for details.

• FEATURE: vmalert: add auto request retry for trivial network errors, such as broken pipe and connection reset for requests to remoteRead, remoteWrite and datasource URLs. See this issue for details.

• FEATURE: vmalert: reduce CPU usage when evaluating high number of alerting and recording rules.

• FEATURE: vmalert: speed up retrieving rules files from object storages by skipping unchanged objects during reloading. See this issue.

• FEATURE: vmalert: support reading DNS SRV records in -datasource.url, -remoteWrite.url and -remoteRead.url command-line option. For example, -remoteWrite.url=http://srv+victoria-metrics automatically resolves the victoria-metrics DNS SRV to a list of hostnames with TCP ports and then sends data to one of the addresses. See this feature request.

• FEATURE: vmbackup, vmrestore, vmbackupmanager: add -s3TLSInsecureSkipVerify command-line flag for skipping TLS certificates verification when connecting to S3 endpoint.

• FEATURE: expose metric vm_indexdb_items_dropped_total to track the number of IndexDB records that had to be dropped during ingestion. The reason of dropping the record will be annotated in reason label of the exposed metric. This change also comes with a new alerting rule to track changes of this metric.

• FEATURE: alerts-health: add new alerting rules TooLongLabelValues and TooLongLabelNames to notify about truncation of label values or names respectively.

• FEATURE: stream aggregation: expose vm_streamaggr_ignored_samples_total counters at /metrics page, which can be used for detecting amount of too old or NaN valued ignored samples. Expose also vm_streamaggr_samples_lag_seconds histogram to monitor aggregated samples lag.

• FEATURE: stream aggregation: improve filtering speed of the received data samples if match field is matching only metric name.

• BUGFIX: vmui: fix bug that prevents the first query trace from expanding on click event. See this issue. The issue was introduced in v1.100.0 release.

• BUGFIX: vmui: fix calendar display when UTC+00:00 timezone is set. See this issue.

• BUGFIX: vmui: remove redundant requests on the Explore Cardinality page. See this issue.

• BUGFIX: vmui: fix handling of URL params for browser history navigation (back and forward buttons). See this issue and this comment.

• BUGFIX: vmagent: prevent potential panic during stream aggregation if more than one --remoteWrite.streamAggr.dedupInterval is configured. See this issue.

• BUGFIX: vmagent: skip empty data blocks before sending to the remote write destination. Thanks to @viperstars for the pull request.

• BUGFIX: stream aggregation: set correct suffix <output>_prometheus for aggregation outputs increase_prometheus and total_prometheus. Before, outputs total and total_prometheus or increase and increase_prometheus had the same suffix.

• BUGFIX: stream aggregation: prevent from excessive resource usage when stream aggregation config file is empty.

• BUGFIX: properly estimate the needed memory for query execution if it has the format aggr_func(rollup_func[d] (for example, sum(rate(request_duration_seconds_bucket[5m]))). This should allow performing aggregations over bigger number of time series when VictoriaMetrics runs in environments with small amounts of available memory. The issue has been introduced in this commit in v1.83.0.

• BUGFIX: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: correctly apply -inmemoryDataFlushInterval when it's set to minimum supported value 1s.

• BUGFIX: Single-node VictoriaMetrics and vminsert in VictoriaMetrics cluster: reduce the default value for -maxLabelValueLen command-line flag from 16KiB to 1KiB. This should prevent from issues like this one when time series with too long labels are ingested into VictoriaMetrics.

• BUGFIX: vmauth: properly release memory used for metrics during config reload. See this issue.

• BUGFIX: dashboards: fix AnnotationQueryRunner error in Grafana when executing annotations query against Prometheus backend. See this issue for details.

• BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: filter deleted label names and values from /api/v1/labels and /api/v1/label/.../values responses when match[] filter matches small number of time series. The issue was introduced v1.81.0.

• BUGFIX: vmalert-tool: fix float values template in input_series. See this issue.

• BUGFIX: vmalert: fix misleading error logs in vmalert's stdout when unsupported HTTP path is requested.

• BUGFIX: retry files delete attempts on vXFS file system for EEXIST error type. See the related issue. Thanks to @pludov for the pull request.

• DEPRECATION: vmagent: removed deprecated -remoteWrite.multitenantURL flag from vmagent. This flag was deprecated since v1.96.0. Use -enableMultitenantHandlers instead, as it is easier to use and combine with multitenant URL at vminsert. See these docs for details.