Exporting CF Common tags results in broken metrics when appending tags to metric path

[bhoogter]

The recent change #644 broke our metrics exporting. PCF 2.5. Spring 2.1.xx (recent). Exporting via Micrometer to Graphite. Graphite config was to append all tags/values into the path. Paths were then displayed on a Grafana dashboard.

We ended up with:

[prefix].tomcat....cf.account.VALUE.cf.application.VALUE.cf.cluster.VALUE..........Count

Which resulted from the appending of the following tags and values to the path:

• cf account
• cf application
• cf cluster
• cf instance index
• cf space

Took us a while to figure out if it was our code, PCF itself, some other dependency. Finally tracked it down to the build pack. The values still reported to Graphite, but to a different location due to config. This caused all the Grafana dashboards to no longer report data, as the data is now in a new location.

Workaround was to lock in the buildpack version in the manifest.yml to 4.20 with:

buildpacks:

• https://github.com/cloudfoundry/java-buildpack.git#v4.20

This stopped getting the current master for java-buildpack, and got us out of this change.

It isn't immediately clear how to disable these tags from being exported. If there is one, my appologies for not being familiar. If there isn't, it would be helpful if one could be provided, since Graphite, at least our graphite, doesn't provide a satisfactory way of handling tags that we export, and so we simply concatenate them onto the key in our case.

[nebhale]

@bhoogter Thanks for taking the chance of being on master and testing out the new features as we develop them. Sorry to hear that it took a while to find the change, but good that you were able to get back going again.

There is a configuration to disable the feature by doing something like cf set-env <APP> JBP_CONFIG_METRIC_WRITER '{ enabled: false }'. If you'd like this applied to your entire enterprise, you can set a staging environment variable group to the same.

Let me know if you have any issues with this strategy.

[briansurratt]

@nebhale - thanks for the clue on the environment variable.

[neiljpowell]

This is a serious issue and has impacted others. If released, you risk impacting many others who may not thoroughly test build packs. Even if they do test, they won't be expecting to have application monitoring impacts such as broken dashboards and alerts if using hierarchical filesystem-based monitoring tools like statsd and graphite. The customers who have more modern and forgiving monitoring systems, still may not want the additional 7 attributes attached to every metric produced by all of their CF deployed Spring Boot/Micrometer apps (100+ metrics per app?). Depending on monitoring system/provider, it could cost something.

Also, your use of @ Order may not allow anyone to confidently override your tagging behavior, selectively by app. Order defaults to lowest precedence, which applies your filter last (or competes for last), per org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryConfigurer.

Please choose an Order value that is not Lowest and introduce this as an opt-in monitoring feature.