App running with both Java Agent and Micrometer registry shows duplicate entity

[dhiren-mudgil-olx]

App contains both java agent and micrometer registry created duplicate entity in new relic.

Here is the code for configuring the registry

@Configuration
@AutoConfigureBefore(
    CompositeMeterRegistryAutoConfiguration::class, SimpleMetricsExportAutoConfiguration::class)
@AutoConfigureAfter(MetricsAutoConfiguration::class)
@ConditionalOnClass(NewRelicRegistry::class)
@ConfigurationPropertiesScan
@Profile("production")
class NewRelicMetricsConfiguration {

  @Bean
  fun newRelicConfig(newRelicConfiguration: NewRelicConfiguration): NewRelicRegistryConfig {
    return object : NewRelicRegistryConfig {

      override fun apiKey(): String? {
        return newRelicConfiguration.apiKey
      }

      override fun get(key: String): String? {
        return null
      }

      override fun step(): Duration {
        return Duration.ofSeconds(10)
      }

      override fun serviceName(): String? {
        return newRelicConfiguration.appName
      }
    }
  }

  @Bean
  fun newRelicMeterRegistry(config: NewRelicRegistryConfig): NewRelicRegistry {
    val newRelicRegistry = NewRelicRegistry.builder(config).build()
    newRelicRegistry.start(NamedThreadFactory("newrelic.micrometer.registry"))
    return newRelicRegistry
  }

  @ConstructorBinding
  @ConfigurationProperties(prefix = "newrelic")
  data class NewRelicConfiguration(val apiKey: String?, val appName: String?)
}

appName = comms-push-token-consumer-staging-ireland

[georgej-github]

I'm noticing the same issue, any interim workaround available ?

[georgej-github]

NerdGraph returns this info about the entities with the same name (in case this helps):

{
  "data": {
    "actor": {
      "entitySearch": {
        "results": {
          "entities": [
            {
              "accountId": 012,
              "domain": "APM",
              "entityType": "APM_APPLICATION_ENTITY",
              "guid": "...",
              "name": "myapp",
              "type": "APPLICATION"
            },
            {
              "accountId": 012,
              "domain": "EXT",
              "entityType": "THIRD_PARTY_SERVICE_ENTITY",
              "guid": "...",
              "name": "myapp",
              "type": "SERVICE"
            }
          ]
        }
      }
    }
  }
}

[breedx-nr]

Ok, so this is a bit of a drag and, frankly, this is probably a shortcoming that we should work on fixing/addressing.

In the meantime, however, I think there may be a workaround. This involves getting the entity.guid common attribute for the EXT entity to point to the same entity.guid as the APM entity. It's a little complicated, but:

1. Go to the APM entity in NR1 and click the (i) info icon next to the entity name in the upper left. This will open a panel on the righthand side that shows some metadata about the entity.
2. Click the clipboard copy icon next to the entity guid.
3. Depending on how you are wiring up the registry, this next step might be a lot of work...but you will somehow want to leverage the NewRelicRegistryBuilder to add some common attributes. At minimum, it should contain the entity guid copied above, like new Attributes().put('entity.guid', 'abc1234thingcopiedabove7890').

I know that's a lot of hand waiving around #3, but I haven't had a chance to try doing that yet. If you can get it wired up that way as a workaround, please let us know, and please keep in mind that this is NOT the long-term expected solution. Also, this will not impact data that has already been ingested with the other entity, but should apply to newly ingested data going forward.

We should provide something more useful. Thanks again for working with us on this.

[breedx-nr]

A better approach might be to poll JMX for the linking metadata mbean that can be exposed by the agent (although that also requires agent configuration).

[breedx-nr]

There is an example of setting up the NewRelicRegistry with common attributes here: https://github.com/newrelic/micrometer-registry-newrelic/wiki/Spring-Config-Example
That could be modified to contain the desired entity.guid.

[XiXiaPdx]

Hi @georgej-github , any luck in trying the workaround @breedx-nr suggested? I know it is not ideal 😞 If you have a moment, please let us know if it worked. Thank you!

[georgej-github]

Hi @XiXiaPdx , unfortunately not, at this point we're going with the Micrometer NewRelic registry so we can integrate with the NR Java agent as we already use it in our environments. Does the roadmap for NewRelic's registry intend to support the features that Micrometer's registry currently provides, including NR agent integration ?

[XiXiaPdx]

@georgej-github Ah, I see. I'm glad you found a solution around this issue.

I'm curious what features in the Micrometer NewRelic registry (just to be clear, you are referring to this registry?) you would like the NewRelic registry to support? Or to ask in another way, what problems do our NewRelic registry not solve for you right now? Your use case will help our internal product manager understand this issue better.

I have a general understanding of the Micrometer one. It either takes a NR insights api key to send events or if the NR Agent is installed, it delegates to the NR agent to send custom events, which very likely get sampled, and you don't need the insights api key.

Thank you for responding and helping us improve our offering!

[jerrychen-afterpay]

Hi team,

I've also run into this issue. I've tried the workaround mentioned above (i.e. Attributes().put('entity.guid', 'guid...')) and the result was that metrics did not show up in neither the duplicate service nor the original service.
Not sure if related, but one difference I see between the two is that the duplicate from micrometer has a Integration-reported services instead of the usual Services on the top left of the entity dashboard (image attached).

Are there any other potential workarounds, or will this be just pending a fix?

[rfelgent]

I would also prefer a workaround instead of switching to micrometers-new-relic libraries. I would appreciate any help for providing a working example

[rfelgent]

Today I was informed by @XiXiaPdx that the telemetry-sdk supports the license key (have a look at: newrelic/newrelic-telemetry-sdk-java#257).

The micrometer-registry-newrelic must now be aligned with that new feature of the SDK in order to get rid of the duplicate entity, right?

[XiXiaPdx]

Today I was informed by @XiXiaPdx that the telemetry-sdk supports the license key (have a look at: newrelic/newrelic-telemetry-sdk-java#257).

The micrometer-registry-newrelic must now be aligned with that new feature of the SDK in order to get rid of the duplicate entity, right?

@rfelgent micrometer-registry-newrelic will need to be updated before it will accept a license key. That issue is being tracked here #126

[rfelgent]

thank you very much for the link @XiXiaPdx.

one more question: would the support for license key also solve this issue. As far as I unerstand - yes?!

[georgej-github]

@georgej-github Ah, I see. I'm glad you found a solution around this issue.

I'm curious what features in the Micrometer NewRelic registry (just to be clear, you are referring to this registry?) you would like the NewRelic registry to support? Or to ask in another way, what problems do our NewRelic registry not solve for you right now? Your use case will help our internal product manager understand this issue better.

I have a general understanding of the Micrometer one. It either takes a NR insights api key to send events or if the NR Agent is installed, it delegates to the NR agent to send custom events, which very likely get sampled, and you don't need the insights api key.

Thank you for responding and helping us improve our offering!

Hi @XiXiaPdx, apologies for the late response,
We currently include the NR java agent in our app containers and dont plan on removing this for the foreseeable future. Instead of the registry sending data via its own mechanism in addition to the NR agent, having micrometer delegate to the NR agent to send custom events gives us the benefit of a smaller network (and possibly CPU,mem) footprint, as our apps have some important requirements around performance.

[kford-newrelic]

@dhiren-mudgil-olx thank you for bringing this issue to our attention. Unfortunately, we do not have the engineering capacity to take up this issue.

If any contributors would like to do some agent coding however, we would gladly review any PR(s) submitted.