All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
- Update
e2e-resourceschart by @juanjjaramillo #1018
- Updated kubernetes packages to v0.29.3
- Updated github.com/prometheus/common to v0.50.0 - Changelog 🔗
- Automatically release E2E resources chart by @juanjjaramillo #1013
- Give GitHub token permission to release chart by @juanjjaramillo #1014
- Updated golang.org/x/crypto to v0.21.0
- Updated google.golang.org/protobuf to v1.33.0
- Updated kubernetes packages to v0.29.2
- Updated github.com/prometheus/client_model to v0.6.0 - Changelog 🔗
- Updated github.com/prometheus/common to v0.48.0 - Changelog 🔗
- Add linux node selector @dbudziwojskiNR #1000
- Updated golang.org/x/crypto to v0.19.0
- Updated github.com/newrelic/infra-integrations-sdk to v3.8.2+incompatible
- Updated github.com/newrelic/infra-integrations-sdk to v3.8.0+incompatible
- Add Codecov @dbudziwojskiNR #980
- Updated kubernetes packages to v0.29.1
- Updated alpine to v3.19.1
- Updated github.com/prometheus/common to v0.46.0 - Changelog 🔗
- Updated go to v1.21.6
- Add pod startup metrics by @w21froster #964
- Updated golang.org/x/crypto to v0.18.0
- Updated kubernetes packages to v0.29.0
- Updated github.com/spf13/viper to v1.18.2 - Changelog 🔗
- Updated google.golang.org/protobuf to v1.32.0
- Trigger release creation by @juanjjaramillo #958
- Updated alpine to v3.19.0
- Updated github.com/spf13/viper to v1.18.1 - Changelog 🔗
- Updated golang.org/x/crypto to v0.16.0
- Updated go to v1.21.5
- Update reusable workflow dependency by @juanjjaramillo #951
- Improve E2E resources chart by @juanjjaramillo in #946
- Update k8s.yaml by @juanjjaramillo in #947
- Automate local E2E test runs by @juanjjaramillo in #938
- Add PV, PVC dashboards tests by @dbudziwojskiNR in #829
- Add statefulset dashboard tests by @dbudziwojskiNR in #830
- Add deployment dashboard tests by @dbudziwojskiNR in #832
- Add failed job dashboard tests by @dbudziwojskiNR in #855
- Updated kubernetes packages to v0.28.4
- Update E2E resources by @juanjjaramillo in #926
- Replace k8s v1.28.0-rc.1 with k8s 1.28.3 by @svetlanabrennan in #936
- Add failed pod container pending e2e tests by @dbudziwojskiNR in #849
- Add failed pod container creating e2e tests by @dbudziwojskiNR in #848
- Add cronjob dashboard tests by @dbudziwojskiNR in #827
- Add daemonset dashboard tests by @dbudziwojskiNR in #828
- Updated golang.org/x/crypto to v0.15.0
- Add k8s v1.28.0-rc.1 support by @svetlanabrennan in #919
- Fix
renovateconfiguration by juanjjaramillo in PR #921
- Updated kubernetes packages to v0.28.3
- Updated github.com/prometheus/common to v0.45.0 - Changelog 🔗
- Address CVE-2023-44487 and CVE-2023-39325 by juanjjaramillo in PR #910
- Updated github.com/google/go-cmp to v0.6.0 - Changelog 🔗
- Updated github.com/spf13/viper to v1.17.0 - Changelog 🔗
- Upgraded alpine from 3.18.3 to 3.18.4
- Updated go to 1.21
- Updated golang.org/x/crypto to v0.14.0
- Updated github.com/prometheus/client_model to v0.5.0 - Changelog 🔗
- chore(deps): bump docker/setup-qemu-action from 2 to 3 by @dependabot in #881
- chore(deps): bump docker/login-action from 2 to 3 by @dependabot in #880
- chore(deps): bump manusa/actions-setup-minikube from 2.7.2 to 2.9.0 by @dependabot in #879
- chore(deps): bump docker/setup-buildx-action from 2 to 3 by @dependabot in #878
- chore(deps): update actions/checkout action to v4 by @renovate in #864
- fix(deps): update kubernetes packages to v0.28.2 by @renovate in #861
- chore(deps): update newrelic/infrastructure-bundle docker tag to v3.2.16 by @renovate in #859
- chore(deps): update newrelic/k8s-events-forwarder docker tag to v1.47.0 by @renovate in #860
- chore(deps): update module golang.org/x/crypto to v0.13.0 by @renovate in #866
- chore(deps): update aquasecurity/trivy-action action to v0.12.0 by @renovate in #862
- Update KSM version and chart, deprecate incompatible Kubernetes versions by @juanjjaramillo in #867
- Update unit testing data by @juanjjaramillo in #868
- Update Helm chart lint job by @juanjjaramillo in #869
- Update cpuLimitCores metric not available log to debug level in #870
- Updated KSM unit tests by @svetlanabrennan in #876
- Use a KSM stable metric instead of an experimental one by @juanjjaramillo in #872
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.16.0...v3.17.0
- Add changelog workflow by @svetlanabrennan in #837
- Update changelog workflow @svetlanabrennan in #843
- Add k8s 1.27 support by @csongnr in #845
- @davidgit made their first contribution in #826
- @nr-security-github made their first contribution in #846
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.15.3...v3.15.4
- bump app and chart version by @csongnr in #819
- NR-139168: Fix k8s.container.cpuCoresUtilization metric calculation by @sachin-shankar in #817
- @sachin-shankar made their first contribution in #817
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.15.2...v3.15.3
- Bump Chart Versions by @xqi-nr in #810
- Update Changelog by @xqi-nr in #809
- chore(deps): Directly Use Prometheus Parser - remove prom2json dep by @isaacadeleke-nr in #799
- fix(parsing): Log an error instead of fully failing on partial parsing failure by @isaacadeleke-nr in #802
- fix(deps): update module google.golang.org/protobuf to v1.31.0 by @renovate in #811
- chore(deps): update newrelic/k8s-events-forwarder docker tag to v1.45.0 by @renovate in #814
- chore(deps): update newrelic/infrastructure-bundle docker tag to v3.2.11 by @renovate in #816
- fix(deps): update kubernetes packages to v0.27.4 by @renovate in #815
- chore(deps): update module golang.org/x/crypto to v0.11.0 by @renovate in #813
- chore(deps): update newrelic/infrastructure-bundle docker tag to v3.2.12 by @renovate in #818
@isaacadeleke-nr made their first contribution in #799
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.15.1...v3.15.2
- update changelog by @csongnr in #804
- update chart version and reference latest docker image by @csongnr in #805
- chore(deps): update newrelic/k8s-events-forwarder docker tag to v1.43.1 by @renovate in #806
- chore(deps): update newrelic/infrastructure-bundle docker tag to v3.2.9 by @renovate in #807
- Upgrade Go Version to 1.20 by @xqi-nr in #808
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.15.0...v3.15.1
- Update CHANGELOG.md by @juanjjaramillo in #783
- Bump versions by @juanjjaramillo in #784
- fix(deps): update module github.com/sirupsen/logrus to v1.9.3 by @renovate in #785
- chore(deps): update aquasecurity/trivy-action action to v0.11.0 by @renovate in #787
- chore(deps): update newrelic/k8s-events-forwarder docker tag to v1.42.5 by @renovate in #789
- chore(deps): update newrelic/infrastructure-bundle docker tag to v3.2.7 by @renovate in #788
- Automate generating static test data for all supported Kubernetes versions by @juanjjaramillo in #792
- Update E2E readme file by @juanjjaramillo in #801
- chore(deps): update aquasecurity/trivy-action action to v0.11.2 by @renovate in #794
- chore(deps): update module golang.org/x/crypto to v0.10.0 by @renovate in #795
- fix(deps): update kubernetes packages to v0.27.3 by @renovate in #797
- bump alpine from 3.18.0 to 3.18.2 by @csongnr in #803
- chore(deps): update newrelic/k8s-events-forwarder docker tag to v1.43.0 by @renovate in #800
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.14.0...v3.15.0
- Update chart and image versions by @juanjjaramillo in #769
- Update static test data by @juanjjaramillo in #776
- Update Prometheus dependencies by @juanjjaramillo in #775
- Increase the number of parallel E2E tests by @juanjjaramillo in #780
- Add demo mode for testing resources by @juanjjaramillo in #779
- Update
datagen.shdocumentation by @juanjjaramillo in #777 - chore(deps): update newrelic/k8s-events-forwarder docker tag to v1.42.3 by @renovate in #781
- chore(deps): update newrelic/infrastructure-bundle docker tag to v3.2.4 by @renovate in #782
- fix(deps): update module github.com/spf13/viper to v1.16.0 by @renovate in #773
- fix(deps): update module github.com/stretchr/testify to v1.8.4 by @renovate in #772
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.13.0...v3.14.0
- Bump chart by @htroisi in #750
- Update kubelet static testing exclusions. by @htroisi in #759
- Update
datagen.shby @juanjjaramillo in #762 - Silence log messages that mask testing errors by @juanjjaramillo in #763
- fix(deps): update module github.com/stretchr/testify to v1.8.3 by @renovate in #756
- chore(deps): update newrelic/k8s-events-forwarder docker tag to v1.42.1 by @renovate in #755
- chore(deps): update newrelic/infrastructure-bundle docker tag to v3.2.2 by @renovate in #751
- chore(deps): bump github.com/sirupsen/logrus from 1.9.0 to 1.9.2 by @juanjjaramillo in #765
- fix(deps): update kubernetes packages to v0.27.2 by @renovate in #740
- NEWRELIC-5968 Mount containerd socket by @DavSanchez in #734
- Integration tests for k8s 1.27 by @csongnr in #719
- chore(deps): update module golang.org/x/crypto to v0.9.0 by @renovate in #739
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.12.0...v3.13.0
- Bump app and chart version by @juanjjaramillo in #736
- Update renovatebot by @htroisi in #732
- Add more workload name fields to K8sContainerSample, K8sPodSample by @htroisi in #733
- chore(deps): update newrelic/k8s-events-forwarder docker tag to v1.41.0 by @renovate in #748
- chore(deps): update alpine docker tag to v3.18.0 by @renovate in #746
- Add PersistentVolume and PersistentVolumeClaim KSM metrics by @htroisi in #729
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.11.0...v3.12.0
- chore(deps): update newrelic/k8s-events-forwarder docker tag to v1.40.0 by @renovate in #712
- Bump app and chart version by @htroisi in #710
- chore(deps): update newrelic/infrastructure-bundle docker tag to v3.1.6 by @renovate in #716
- Improve Cronjob chart for testing by @juanjjaramillo in #715
- CronJob chart improvement by @juanjjaramillo in #718
- chore(deps): update newrelic/infrastructure-bundle docker tag to v3.1.7 by @renovate in #717
- Remove manual go cache since setup-go/v4 automatically caches by @htroisi in #713
- chore(deps): bump actions/github-script from 6.4.0 to 6.4.1 by @dependabot in #714
- Add spec for backoffLimit to help with testing cronjobs by @juanjjaramillo in #720
- Add KSM metrics promoted to stable category by @juanjjaramillo in #724
- Fix push_pr workflow by @htroisi in #726
- Improve testing for deployment object by @juanjjaramillo in #730
- Adding new attribute
ReplicaFailureto deployment workload by @juanjjaramillo in #725 - chore(deps): bump aquasecurity/trivy-action from 0.9.2 to 0.10.0 by @htroisi in #727
- Fix Helm unittests by @htroisi in #728
- Update infrastructure-bundle to v3.1.8 and k8s-events-forwarder to v1.40.1 by @htroisi in #731
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.10.0...v3.11.0
- Add CronJob and Job kube-state-metrics collection
- Make ProbeTimeout and ProbeBackoff Configurable
- Updated dependencies
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.9.0...v3.10.0
- Updated dependencies
- Update Kubernetes image registry
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.7.0...v3.9.0
- Fix an issue where when the Kubelet becomes temporarily unavailable the agent fails: #633
- Update e2e testing chart templates
- Updated dependencies
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.6.0...v3.7.0
- Add support for kube-state-metrics v2
- Update static test data to use KSM v2
- Update kube-state-metrics version in e2e testing chart
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.5.0...v3.6.0
Updated go version and several dependencies
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.4.1...v3.5.0
In version above 1.21 having the apiServer flag service-account-extend-token-expiration set to false was causing the kubelet scraper pod to be restarted each time the token expired.
In AWS having environments due to its implementation caused a pod restart each 90days
Updated several dependencies
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.4.0...v3.4.1
- Add k8s v1.23 & v1.24 new metrics [#485, #507]:
apiserverCurrentInflightRequestsMutatingapiserverCurrentInflightRequestsReadOnlycontainerOOMEventsDeltanodeCollectorEvictionsDeltaschedulerPendingPodsActiveschedulerPendingPodsBackoffschedulerPendingPodsUnschedulable
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.3.1...v3.4.0
- Add nrFiltered attribute to K8sNamespaceSamples when using namespace filtering #496
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.3.0...v3.3.1
- Allow filter to only scrape selected namespaces in ksm and kubelet by @alvarocabanas and @marcsanmi in #457, #476 and #487
- Use Go version 1.18 in the pipelines @roobre #472
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.2.1...v3.3.0
- fix: round up CPU allocatable and capacity metrics by @gsanchezgavier in #412
- Dockerfile: use COPY instead of ADD by @roobre in #433
- chore(deps): bump alpine from 3.15.4 to 3.16.0 by @dependabot in #458
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.2.0...v3.2.1
- Initial, internal-only implementation of mTLS-enabled sink by @roobre in #338
restartCountmetric for pods is now also available asrestartCountDeltaby @sigilioso in #382
isReadymetric is now correctly reported asfalse(rather thanNULL) for pending pods by @paologallinaharbur in #404
- chore(deps): bump github.com/google/go-cmp from 0.5.7 to 0.5.8 by @dependabot in #406
- chore(deps): bump alpine from 3.15.0 to 3.15.4 by @dependabot in #391
- chore(deps): bump github.com/newrelic/infra-integrations-sdk from 3.7.1+incompatible to 3.7.2+incompatible by @dependabot in #372
- chore(deps): bump github.com/sethgrid/pester from 1.1.0 to 1.2.0 by @dependabot in #358
- Update dependencies to solve security issues pointed by trivy by @kang-makes in #403
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.1.0...v3.1.1
- chore(deps): bump aquasecurity/trivy-action from 0.2.1 to 0.2.2 by @dependabot in #355
- controlplane/authenticator: allow to use
kubernetes.io/tlssecrets by @roobre in #344 - config: document options by @roobre in #363
Full Changelog: https://github.com/newrelic/nri-kubernetes/compare/v3.0.0...v3.1.0
This new version makes significant changes to the number of components that are deployed to the cluster, and introduces many new configuration options to tune the behavior to your environment. We encourage you to take a look at what's changed in full detail here.
The number and format of the metrics reported by version 3 of the integration have not changed with respect to earlier versions.- The format of the
values.ymlfile has changed to accommodate the newly added configuration options. Please take a look at our migration guide to see how to change your configuration.
- Our solution is now deployed in three components:
- A
DaemonSetto monitor the Kubelet, deployed in all nodes of the cluster. - A second
DaemonSetto monitor the control plane, deployed in master nodes only. - A
Deploymentto collect metrics from kube-state-metrics, deployed in the same node as the latter.
- A
- We now offer better control for CPU and memory limits and requests, which can be now configured for the three components individually.
- Impact of discovery and collection operations on the API server has been greatly reduced, thanks to the use of kubernetes informers.
- Logs messages have been greatly revamped to surface problems more clearly.
- Comprehensive configuration options have been added to provide fine-grain control to how the integration discovers and connects to metric providers. Remarkably:
- Discovery options for control plane components have been improved. You can check the details on how discovery is configured here.
- It is now possible to collect metrics from control plane components running outside of the cluster.
- Discovery options for KSM and the kubelet have also been added.
- The interval at which metrics are collected is now configurable.
- Moved default config.sample to V4, added a dependency for infra-agent version 1.20.0
Please notice that old V3 configuration format is deprecated, but still supported.
- Updated agent and integrations to their latest version.
- Updated agent and integrations to their latest version.
- Node status and conditions are now fetched from the API Server rather than KSM, which fixes some inconsistencies in the samples. This does not change which data is reported, and should be an invisible change. (#194).
- Add a series of parameters which allow to configure a jitter to be applied to API Server response caching, which might help to spread the load on large clusters. (#185).
Note: This is an out-of-order release which brings some hotfixes to the 2.7.x branch
- Node status and conditions are now fetched from the API Server rather than KSM, which fixes some inconsistencies in the samples. This does not change which data is reported, and should be an invisible change.
- Kubernetes v1.22.0 Support
- Upgrade infrastructure-bundle to 2.6.4 (#123)
- See https://github.com/newrelic/infrastructure-bundle/releases/tag/2.6.4 for more details about the upgraded integrations in this release of the infrastructure-bundle
- Integration now reports node status and conditions, as
condition.{Name}(e.g.condition.Ready,condition.PIDPressure). - Added new KubeStateMetricsNamespace parameter to restrict discovery of KSM pod to a particular namespace.
- This should help reduce load in the control plane for clusters with many pods and/or nodes.
- Integration version shown in the samples.
- Upgrade infrastructure-bundle to 2.6.0 (#123)
- See https://github.com/newrelic/infrastructure-bundle/releases/tag/2.6.0 for more details about the upgraded integrations in this release of the infrastructure-bundle
- Bumped all dependencies and moved to /v2 in go.mod #111
- Improved e2e tests with more coverage and support for Helm3 and k8s 1.20-1.21 #110 #108
- Improved KSM discovery logic #104
- Support for multiarch docker images
- Correctly identifying k8s server version with characters (#81)
- The base image of
newrelic/infrastructure-k8shas been updated to2.2.3. More info regarding all the integrations upgraded can be found in the release notes of the base image. - Changed scale of node
cpuRequestedCoresto cores from millis
- Added metrics pertaining to Horizontal Pod Autoscaler. More information about the collected metrics can be found in the official documentation
- LoadBalancerIP was not being collected properly. It is now fetched from KSM metric
kube_service_status_load_balancer_ingress
- The base image of
newrelic/infrastructure-k8shas been updated to2.2.1. This base image has fixed an issue wherenrjmxwas not properly running due to the bundled java version. More info regarding all the integration upgraded can be found in the release notes of the base image.
- Added aggregate cpu and memory requests for nodes
- The base image of
newrelic/infrastructure-k8shas been updated to2.0.0. That base image is bundling the integrationnri-nginx3.0.2that contains a breaking change. More info regarding all the integration upgraded can be found in the release notes of the base image.
- Added release pipeline to Github Actions.
- Upgraded Docker base image
newrelic/infrastructure-bundleto v1.6.0. For more information on the release please see the New Relic Infrastructure Bundle release notes.
- When configuring API_SERVER_ENDPOINT_URL with https endpoint, the api-server component will use that that instead of the default. More info at issue #41
- When discovering kube-state-metrics behind a headless service, the DNS discovery will return an error. Before it would be considered successful and return "None" as endpoint.
- Container id's are correctly parsed when using system driver
- Added
restartCountto containers in thewantingstate- In case the container is in a crash loop the restart count would not be shown
- Added
restartCountto containers in theterminatedstate- In case the container is in a crash loop the restart count would not be shown
- Upgrade Docker image to use Tini entrypoint solving:
- reaps orphaned zombie process attached to PID 1
- correctly forwards signals to CMD process Read for more details: https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/
- Upgraded Docker base image
newrelic/infrastructure-bundleto v1.5.1. For more information on the release please see the New Relic Infrastructure Bundle release notes.
- Upgraded Docker base image
newrelic/infrastructure-bundleto v1.5.0. For more information on the release please see the New Relic Infrastructure Bundle release notes.
- When querying the summary endpoint from Kubelet to get the Node or Pod network metrics, if the default network interface is not eth0 then summary endpoint for Kubelet doesn't return the metrics as we expect them. We rely on them being a direct member of the "network" object. See rxBytes, txBytes and rxErrors in the following example metrics:
"network": {
"time": "2020-06-04T10:01:15Z",
"name": "eth0",
"rxBytes": 207909096,
"rxErrors": 0,
"txBytes": 8970981,
"txErrors": 0,
"interfaces": [
{
"name": "eth0",
"rxBytes": 207909096,
"rxErrors": 0,
"txBytes": 8970981,
"txErrors": 0
},
{
"name": "ip6tnl0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "tunl0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
}
]
}
This scenario only happens when the default interface is eth0. Kubernetes source code has it hardcoded that eth0 is the default. In the following example you can see that we only have network metrics inside the interfaces list, in this case there is no eth0 on the and the default interface is ens5:
"network": {
"time": "2020-06-04T10:01:15Z",
"name": "",
"interfaces": [
{
"name": "ens5",
"rxBytes": 207909096,
"rxErrors": 42,
"txBytes": 8970981,
"txErrors": 24
},
{
"name": "ip6tnl0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "tunl0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
}
]
In cases like this, the integration will look for the default interface inside the interfaces list and use those values. The default interface name is retrieved from the network route file (default /proc/net/route).
When running the unprivileged version of the integration we don't have access to the route file, the integration won't be able to get the default interface name and won't send network metrics for the unless there's a network interface called eth0.
For Pods, this issue is mainly present when using hostNetwok since they shared the same network interfaces with the Node.
- Upgraded Docker base image
newrelic/infrastructure-bundleto v1.4.2. For more information on the release please see the New Relic Infrastructure Bundle release notes.
- Support for Kubernetes versions 1.17.X
-
Upgraded Docker base image
newrelic/infrastructure-bundleto v1.4.1. For more information on the release please see the New Relic Infrastructure Bundle release notes. -
The API server is queried by default on the Secure Port using the service account's bearer authentication. If the query on the Secure Port fails, it will fallback automatically to the non-secure one. This should preserve the same behavior as previous versions.
- Upgraded Docker base image
newrelic/infrastructure-bundleto v1.4.0. For more information on the release please see the New Relic Infrastructure Bundle release notes.
- Bug that swapped values of node allocatable resources with node capacity resources.
- Kubernetes 1.16 is now officially supported.
- The minimum supported version of kube-state-metrics for this release is 1.9.5, according to the KSM compatibility matrix
- Added container throttling metrics to the K8sContainerSample:
containerCpuCfsPeriodsDelta: Delta change of elapsed enforcement period intervals.containerCpuCfsThrottledPeriodsDelta: Delta change of throttled period intervals.containerCpuCfsThrottledSecondsDelta: Delta change of duration the container has been throttled.containerCpuCfsPeriodsTotal: Number of elapsed enforcement period intervals.containerCpuCfsThrottledPeriodsTotal: Number of throttled period intervals.containerCpuCfsThrottledSecondsTotal: Total time duration the container has been throttled.
- Added container mmap byte usage metrics to the K8sContainerSample:
containerMemoryMappedFileBytes: Size of memory mapped files in bytes.
- Upgraded Docker base image
newrelic/infrastructure-bundleto v1.3.9. For more information on the release please see the New Relic Infrastructure Bundle release notes.
- Resources allocatable and capacity are retrieved from the apiserver and
added to the
K8sNodeSampleascapacity<ResourceName>andallocatable<ResourceName>. - The Kubernetes server version is retrieved from the apiserver and cached
with the
APIServerCacheK8SVersionTTLconfig option. The Kubernetes version is added to theK8sClusterSampleasclusterK8sVersionand to inventory. - Add support for static pods status for Kubernetes server versions 1.15 or newer.
- Upgraded Docker base image
newrelic/infrastructure-bundleto v1.3.8. For more information on the release please see the New Relic Infrastructure Bundle release notes.
- Upgraded Docker base image
newrelic/infrastructure-bundleto v1.3.5. For more information on the release please see the New Relic Infrastructure Bundle release notes.
- New label combination to discover the Kubernetes controller manager:
app=controller-managercontroller-manager=true
- Upgraded Docker base image
newrelic/infrastructure-bundleto v1.3.4. For more information on the release please see the New Relic Infrastructure Bundle release notes.
- Upgraded Docker base image
newrelic/infrastructure-bundleto v1.3.2. For more information on the release please see the New Relic Infrastructure Bundle release notes.
- Added the necessary files for building a windows image of the integration. The windows image needs to be manually created and it's still not in our CI/CD pipeline. We have the files for building it but we are not publishing it. The latest supported image for Windows, at the time of writing, is 1.16.0.
- Upgraded Docker base image
newrelic/infrastructure-bundleto v1.3.0. For more information on the release please see the New Relic Infrastructure Bundle release notes.
- Support for completely avoid querying Kube State Metrics. It's behind the
DISABLE_KUBE_STATE_METRICSenvironment variable and its default value isfalse. Note that disabling this will imply in missing metrics that are collected from KSM and possibly missing features in the Kubernetes Cluster Explorer. Please, refer to our official documentation on this configuration option for more information.
- Support for querying Kube State Metrics instances behind kube-rbac-proxy.
It ONLY works when the label-based KSM discovery is enabled through the
KUBE_STATE_METRICS_POD_LABELenvironment variable. 2 new configuration environment variables are added:- KUBE_STATE_METRICS_SCHEME: defaults to
http. Valid values arehttpandhttps. - KUBE_STATE_METRICS_PORT: defaults to
8080. On a standard setup of kube-rbac-proxy this should be set to8443.
- KUBE_STATE_METRICS_SCHEME: defaults to
- OpenShift Control Plane components are now automatically discovered.
- Added 4 new environment variables to explicitly set the Control Plane components URLs:
- SCHEDULER_ENDPOINT_URL
- ETCD_ENDPOINT_URL
- CONTROLLER_MANAGER_ENDPOINT_URL
- API_SERVER_ENDPOINT_URL
- Fix a bug that was preventing
selector.<key>type attributes to not be added to some of theK8sServiceSample. - The integration now uses
newrelic/infrastructure-bundleas the base image. The version used is1.2.0, for more information on the release please see the New Relic Infrastructure Bundle release notes.
- The integration now uses the infrastructure agent v1.9.0-bundle. For more information refer to the infrastructure agent release notes between versions v1.8.32 and v1.9.0.
- Added daemonsetName field to the K8sDaemonsetSample
-
Added samples for Statefulsets, Daemonsets, Endpoints and Services.
-
API Server metrics can now be queried using the secure port. Configure the port using the
API_SERVER_SECURE_PORTenvironment variable. The ClusterRole has been updated to allow this query to happen. -
The integration now uses the infrastructure agent v1.8.32-bundle. For more information refer to the infrastructure agent release notes between versions v1.8.23 and v1.8.32.
The bundle container contains a subset of On-host integrations that are supported by New Relic. This also includes the ability to "Auto Discover" services running on Kubernetes in a similar way to our Container auto-discovery
-
The integration has been renamed from
nr-kubernetestonri-kubernetes.
- Control Plane components can now also be discovered using the
tierandcomponentlabels, besidesk8s-app. - The integration now uses the infrastructure agent v1.8.23. For more information refer to the infrastructure agent release notes between versions v1.5.75 and v1.8.23.
-
The old way of determining Leader/Follower status has been switched to a job based architecture. The old Leader/Follower was needed to detect which nri-kubernetes Pod should query Kube State Metrics (a.k.a. the Leader), but it was hard to add additional scrape targets (e.g. control plane). Important notice: The output logs have been changed. Before the integration logged whether it was Follower or a Leader, and this has been changed to show which jobs are executed.
Following are 2 examples of logs that are being output by the integration before and after this update.
Before, logs for leaders:
Auto-discovered role: LeaderAfter, equivalent example, now using jobs:
Running job: kubelet Running job: kube state metricsBefore, logs for followers:
Auto-disovered role: FollowerAfter, equivalent example, now using jobs:
Running job: kubeletThese 2 before & after examples are identical in the targets & information they scrape.
-
The e2e test package has been updated to work with this refactor.
-
Control Plane Monitoring: the integration will automatically detect if it's running on a master node using its Kubernetes pod's labels, which are retrieved from the API Server. If it finds itself running on a master node, these additional jobs will run:
- ETCD
- API Server
- Controller Manager
- Scheduler
All jobs, except ETCD, will work out of the box with no further configuration needed. ETCD exposes its metrics using Mutual TLS, which can be configured as follows.
First, create a secret containing the following fields:
key: <private_key_data, PEM format> cert: <certificate_belonging_to_private_key, PEM format> cacert: <optional, the ETCD cacert, PEM format> insecureSkipVerify: <optional, bool 'true' or 'false', default: 'false'>Which can be created like this (expecting the files
key,certandcacertto be present):kubectl create secret generic etcd-server-tls --from-file=./key --from-file=./cert --from-file=./cacertThen, configure the integration to use this secret using these environment variables.
ETCD_TLS_SECRET_NAME: etcd-server-tls ETCD_TLS_SECRET_NAMESPACE: defaultIf everything is configured properly the integration should start collecting ETCD metrics.
-
A new command, called kubernetes-static has been added, which enables the integration to be run locally on your machine, without deploying it to k8s. It uses a static set of exports from kubelet & KSM.
-
A new way to query a specific Kube State Metrics (KSM) pod when running multiple redundant pods: by Label Selector. If you want to target a certain KSM instance, you can now use the
KUBE_STATE_METRICS_POD_LABELenvironment variable. If this variable has been set (and KUBE_STATE_METRICS_URL is unset) the integration will find the KSM pod by this variable.For example:
```shell script # Label a specific KSM pod. Always set the value to the string "true". kubectl label pod kube-state-metrics please-use-this-ksm-pod=true ```Configure
nri-kubernetesto use this KSM pod:env: - name: KUBE_STATE_METRICS_POD_LABEL value: please-use-this-ksm-pod
- The integration now uses the infrastructure agent v1.5.75. For more information refer to the infrastructure agent release notes between versions v1.5.31 and v1.5.75.
- Rollback agent version to v1.5.31 because there is an issue with nodes reporting inventory using the node ip as entity key, this causes the nodes to be indexed as clusters.
-
Node labes are added to the
K8sNodeSample. They are retrieved from the k8s api and cached. -
The integration now uses the infrastructure agent v1.5.51. For more information refer to the infrastructure agent release notes between versions v1.5.31 and v1.5.51.
- The integration now uses the Infrastructu Agent v1.5.31. The biggest changes were major improvements to logging and to the StorageSampler. For more information refer to the infrastructure agent release notes between versions v1.3.18 and v1.5.31.
- No code changes have been made. This fixes a regression at Docker image level related to moby/moby#35443.
- Support for discovering KSMs when running with the label
app.kubernetes.io/name.
- No code changes has been made. The fix is at docker image level. We got affected by moby/moby#35443.
- The unprivileged integration runs always as
nri-agentuser. Fixes kubernetes/kubernetes#78308. - Infrastructure agent is now behaving in secure-forwarder mode.
- Autodiscovery cache directory permissions got changed from 644 to 744 in order to let the nri-agent user write inside.
- The integraion now uses the infrastructure agent v1.3.18 instead of 1.1.14. Refer to the infrastructure agent release notes for more information about all the changes from this upgrade.
-
The integration reports the name of the cluster as Infrastructure inventory.
-
The integration reports a new event type
K8sClusterSample. At this moment, these events contain only the cluster name as an attribute.
-
Support for kube-state-metrics v1.5.
-
Pod's status reason and status message are now sent in the
K8sPodSampleasreasonandmessagefields. -
Container's
memory_working_set_bytesis now sent in theK8sContainerSampleasworkingSetBytes.
- Always request metrics from kube-state-metrics in the text format. In kube-state-metrics v1.5 this is the default regardless of the format requested.
namespaceNamemetric attribute was added to all the samples wherenamespaceattribute is present.
namespacemetric attribute will be removed soon. Please usenamespaceNamefrom now on.
- Due to an issue in Kubelet, we stopped reporting the Status of static pods. See kubernetes/kubernetes#61717.
- Update base image in the dockerfile to use latest newrelic/infrastructure version: 0.0.62 (Infrastructure agent v1.1.14, released at: 2018-12-20)
- Add clusterName custom attribute to manifest file. This helps users correlate Kubernetes integration data with Infrastructure agent data.
KUBE_STATE_METRICS_URLenvironment variable can be specified containing only host & port or it can be the complete URL including also the/metricspath (ex:http://my-service.my-ns.svc.cluster.local:8080/metrics).
-
Fix how the usage percentage is calculated for container filesystem metrics.
-
Fix how the usage percentage is calculated for volumes.
-
Add metrics for volumes (persistent and non-persistent volumes).
-
Add container filesystem metrics.
- Add
reasonmetric for terminated containers
- Support for specifying the K8s API Host and Port by setting the
KUBERNETES_SERVICE_HOSTandKUBERNETES_SERVICE_PORTenv vars.
- Improve readability of log messages, when verbose mode is enabled.
- Kubernetes API url discovery failed sometimes giving errors like "error trying to connect to...". Now this should be fixed.
- The agent tag installed within the integration docker image is now fixed to 0.0.24.
- Add
hostNetwork: trueoption and the required dns policy to daemonset file. This is a requirement for the Infrastructure Agent to report the proper hostname in New Relic.
-
Update newrelic-infra.yaml to force our objects to be deployed in
defaultnamespace. -
Add NoExecute toleration ensuring that our pod is being deployed when the NoExecute node taint is set.
-
Add missing metric:
podsMaxUnavailablefor deployment -
Fix some of the metrics for pods in pending status
- Adding missing metrics:
startTime,isReady - Unifying
isScheduledandisReadyto be reported as1and0fortrueandfalserespectively.
- Adding missing metrics:
-
Fix pod metrics (
statusandisReady): non-scheduled or pending pods were not reported correctly.
-
Add configurable flag for kube-state-metrics endpoint (only HTTP).
-
Add additional label
appfor discovering kube-state-metrics endpoint.
- Kubelet discovery process fetches now the nodeName directly from the spec using downward API.
- Fix bug in error handling where recoverable errors made the integration to panic.
- Allow direct connection to cAdvisor by specifying the port.
- Call to CAdvisor was failing when Kubelet was secure.
- nodes/metrics resource was added to the newrelic cluster role.
- CAdvisor call is now bypassing Kubelet endpoint talking then directly to CAdvisor port
Initial public beta release.
-
TransformFunc now handles errors.
-
Add checks for missing data coming from kube-state-metrics.
-
Boolean values have changed from
"true"and"false"to1and0respectively from the following metrics:- isReady and isScheduled for pods.
- isReady for containers.
-
Update metrics
errorCountPerSecondtoerrorsPerSecondfor pods and nodes.usageCoreSecondstocpuUsedCoreMillisecondsfor nodes.memoryMajorPageFaultstomemoryMajorPageFaultsPerSecondfor nodes.
- Calculate properly RATE metrics.
-
TypeGenerator for entities.
-
Caching discovered endpoints on disk.
-
Implementation of Time-To-Live (TTL) cache expiry functionality.
-
Added the concept of Leader and Follower roles.
- Leader represents the node where Kube State Metrics is installed (so only 1 by cluster).
- Follower represents any other node.
-
Both Follower and Leader call kubelet /pods endpoint in order to get metrics that were previously fetched from KSM.
-
Fetch metrics from KSM about pods with status "Pending".
-
Prometheus TextToProtoHandleFunc as http.HandlerFunc. Useful for serving a Prometheus payload in protobuf format from a plain text reader.
-
Both Follower and Leader call kubelet /metrics/cadvisor endpoint in order to fill some missing metrics coming from Kubelet.
-
Rename
endpointspackage toclientpackage. -
Moved a bunch of functions related to
Prometheusfromksmpackage toprometheusone. -
Renamed the recently moved
Prometheusfunctions. Removed Prometheus word as it is considered redundant. -
Containers objects reported as their own entities (not as part of pod entities).
-
NewRelic infra Daemonset updateStrategy set to RollingUpdate in newrelic-infra.yaml.
-
Prometheus CounterValue type changed from uint to float64.
-
Change our daemonset file to deploy the integration in "default" namespace.
-
Prometheus queries now require to use an operator.
-
Prometheus Do method now requires a metrics endpoint.
-
Follower does not call KSM endpoints anymore.
-
Config package with default unknown namespace value
-
Removed legacy Kubernetes spec files.
-
Replace
log.Fatal()bylog.Panic()in order to call all defer statements. -
Skip missing data from /stats/summary endpoint, instead of reporting them as zero values.
-
Entities not reported in case of problem with setting their name or type.
-
Adding node metrics. Data is fetched from Kubelet and kube-state-metrics.
-
Adding toleration for the "NoSchedule" taint, so the integration is deployed on all nodes.
-
Adding new autodiscovery flow with authentication and authorization mechanisms.
- Custom arguments for kubelet and kube-state-metrics endpoints.
- Integration stops on KSM or Kubelet connection error, instead of continuing.
-
updatedAtmetric was renamed topodsUpdated. -
cpuUsedCoreshas been divided by 10^9, to show actual cores instead of nanocores. -
Update configurable timeout flag using it to connect to kubelet and kube-state-metrics.
- Fix debug log level when verbose. Some parts of the code didn't log debug information.
- Metrics for unscheduled Pods.
-
Fix format of inherited labels. Remove unnecessary prefix
label_included by kube-state-metrics. -
Fix labels inheritance. Labels weren't propagating between "entities" correctly.
- Initial version reporting metrics about Namespaces, Deployments, ReplicaSets, Pods and Containers. This data is fetched from two different sources: Kubelet and kube-state-metrics.