New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MetricsFilter may create an unbounded number of metrics for requests with a templated URI that are not handled by Spring MVC #5875

Closed
heroicefforts opened this Issue May 5, 2016 · 15 comments

Comments

Projects
None yet
10 participants
@heroicefforts

heroicefforts commented May 5, 2016

The MetricsFilter is unsuitable for REST microservices that are backing a large number of entities. In our case, it was trying to generate gauges and counters for /widget/1 to /widget/60000000. It eventually soaks the heap and causes the server to slow to a crawl as it spends most of its CPU cycles in GC. There should be a big red "caveat" bubble in the documentation that details this scenario.

Ideally, a Spring MVC controller metrics controller should be implemented that works similar to the DropWizard Jersey 2 implementation (plus some RequestMapping introspection), then the MetricsFilter could be disabled by default.

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona May 5, 2016

Member

The behaviour you've described isn't intended, and it doesn't match what's tested.

In one of the tests there's a request mapping like this: @RequestMapping("templateVarTest/{someVariable}"). It's called with /templateVarTest/foo which results in status.200.templateVarTest.someVariable and response.templateVarTest.someVariable being updated. As you can see, it's the name of the variable in the request mapping rather than the value in the request URI that's used.

Can you please provide a sample or testcase that illustrates the behaviour you have described?

Member

wilkinsona commented May 5, 2016

The behaviour you've described isn't intended, and it doesn't match what's tested.

In one of the tests there's a request mapping like this: @RequestMapping("templateVarTest/{someVariable}"). It's called with /templateVarTest/foo which results in status.200.templateVarTest.someVariable and response.templateVarTest.someVariable being updated. As you can see, it's the name of the variable in the request mapping rather than the value in the request URI that's used.

Can you please provide a sample or testcase that illustrates the behaviour you have described?

@heroicefforts

This comment has been minimized.

Show comment
Hide comment
@heroicefforts

heroicefforts May 5, 2016

Then, perhaps, it is a consequence of using Jersey with SpringBoot (don't get me started on why we're using both...):

@GET
@Path("/parent/{pid}/child/{cid}")
@Produces("application/json")
@Timed(name="child.reads")
public Response readChild(@Context UriInfo uriInfo, @PathParam("pid") String parentId,
        @PathParam("cid") String childId) { ... } 

heroicefforts commented May 5, 2016

Then, perhaps, it is a consequence of using Jersey with SpringBoot (don't get me started on why we're using both...):

@GET
@Path("/parent/{pid}/child/{cid}")
@Produces("application/json")
@Timed(name="child.reads")
public Response readChild(@Context UriInfo uriInfo, @PathParam("pid") String parentId,
        @PathParam("cid") String childId) { ... } 
@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona May 5, 2016

Member

Thanks. That'll be it. We need to use Jersey's equivalent of Spring MVC's best matching pattern attribute (assuming it has such a thing).

Member

wilkinsona commented May 5, 2016

Thanks. That'll be it. We need to use Jersey's equivalent of Spring MVC's best matching pattern attribute (assuming it has such a thing).

@wilkinsona wilkinsona changed the title from Document caveats of using MetricsFilter to MetricsFilter doesn't consider path variables when using Jersey May 5, 2016

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona May 18, 2016

Member

@beckje01 No, I don't think so. It think it's something similar to this that we need to do for Jersey.

Member

wilkinsona commented May 18, 2016

@beckje01 No, I don't think so. It think it's something similar to this that we need to do for Jersey.

@beckje01

This comment has been minimized.

Show comment
Hide comment
@beckje01

beckje01 May 18, 2016

Ok so we just need to make sure things are putting that attribute in, this same issue effect Grails 3 as well. So that way anything built on boot the metrics will work for assuming the attribute is set.

beckje01 commented May 18, 2016

Ok so we just need to make sure things are putting that attribute in, this same issue effect Grails 3 as well. So that way anything built on boot the metrics will work for assuming the attribute is set.

@jbellmann

This comment has been minimized.

Show comment
Hide comment
@jbellmann

jbellmann May 24, 2016

Hi,
maybe this snippet could help here?

jbellmann commented May 24, 2016

Hi,
maybe this snippet could help here?

@pdavidson

This comment has been minimized.

Show comment
Hide comment
@pdavidson

pdavidson Nov 18, 2016

I just ran into this with jersey as well. It didn't saturate our jvm. Just filled up 380GB on our graphite server. I'm disabling the filter using:

endpoints:
  metrics:
    filter:
      enabled: false

pdavidson commented Nov 18, 2016

I just ran into this with jersey as well. It didn't saturate our jvm. Just filled up 380GB on our graphite server. I'm disabling the filter using:

endpoints:
  metrics:
    filter:
      enabled: false
@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona Nov 28, 2016

Member

Adding the following bean to an application will correct the metrics for requests handled by Jersey:

@Bean
public ResourceConfigCustomizer bestMatchingPatternResourceConfigCustomizer(
		final JerseyProperties jerseyProperties) {
	return new ResourceConfigCustomizer() {

		@Override
		public void customize(ResourceConfig config) {
			String applicationPath = determineApplicationPath(jerseyProperties,
					config);
			config.register(new BestMatchingPatternContainerRequestFilter(
					applicationPath));
		}

		private String determineApplicationPath(JerseyProperties jerseyProperties,
				ResourceConfig config) {
			if (StringUtils.hasText(jerseyProperties.getApplicationPath())) {
				return jerseyProperties.getApplicationPath();
			}
			ApplicationPath applicationPath = AnnotationUtils
					.findAnnotation(config.getClass(), ApplicationPath.class);
			return applicationPath == null ? "" : applicationPath.value();
		}

	};

}

/**
 * A {@link ContainerRequestFilter} that sets the
 * {@link HandlerMapping#BEST_MATCHING_PATTERN_ATTRIBUTE} request attribute.
 */
private static final class BestMatchingPatternContainerRequestFilter
		implements ContainerRequestFilter {

	private final String applicationPath;

	private BestMatchingPatternContainerRequestFilter(String applicationPath) {
		this.applicationPath = applicationPath.startsWith("/") ? applicationPath
				: "/" + applicationPath;
	}

	@Override
	public void filter(ContainerRequestContext requestContext)
			throws IOException {
		String matchingPattern = extractMatchingPattern(requestContext);
		requestContext.setProperty(HandlerMapping.BEST_MATCHING_PATTERN_ATTRIBUTE,
				matchingPattern);
	}

	private String extractMatchingPattern(
			ContainerRequestContext requestContext) {
		ExtendedUriInfo extendedUriInfo = (ExtendedUriInfo) requestContext
				.getUriInfo();
		List<String> templates = new ArrayList<String>();
		for (UriTemplate matchedTemplate : extendedUriInfo
				.getMatchedTemplates()) {
			String template = matchedTemplate.getTemplate();
			templates.add(template.startsWith("/") ? template : "/" + template);
		}
		if (StringUtils.hasText(this.applicationPath)) {
			templates.add(this.applicationPath);
		}
		Collections.reverse(templates);
		return StringUtils.collectionToDelimitedString(templates, "");
	}

}

Some caveats:

  • I don't like that it uses Spring MVC's request attribute (see #7505)
  • Regular expressions in @Path don't work particularly well (see #7503)
Member

wilkinsona commented Nov 28, 2016

Adding the following bean to an application will correct the metrics for requests handled by Jersey:

@Bean
public ResourceConfigCustomizer bestMatchingPatternResourceConfigCustomizer(
		final JerseyProperties jerseyProperties) {
	return new ResourceConfigCustomizer() {

		@Override
		public void customize(ResourceConfig config) {
			String applicationPath = determineApplicationPath(jerseyProperties,
					config);
			config.register(new BestMatchingPatternContainerRequestFilter(
					applicationPath));
		}

		private String determineApplicationPath(JerseyProperties jerseyProperties,
				ResourceConfig config) {
			if (StringUtils.hasText(jerseyProperties.getApplicationPath())) {
				return jerseyProperties.getApplicationPath();
			}
			ApplicationPath applicationPath = AnnotationUtils
					.findAnnotation(config.getClass(), ApplicationPath.class);
			return applicationPath == null ? "" : applicationPath.value();
		}

	};

}

/**
 * A {@link ContainerRequestFilter} that sets the
 * {@link HandlerMapping#BEST_MATCHING_PATTERN_ATTRIBUTE} request attribute.
 */
private static final class BestMatchingPatternContainerRequestFilter
		implements ContainerRequestFilter {

	private final String applicationPath;

	private BestMatchingPatternContainerRequestFilter(String applicationPath) {
		this.applicationPath = applicationPath.startsWith("/") ? applicationPath
				: "/" + applicationPath;
	}

	@Override
	public void filter(ContainerRequestContext requestContext)
			throws IOException {
		String matchingPattern = extractMatchingPattern(requestContext);
		requestContext.setProperty(HandlerMapping.BEST_MATCHING_PATTERN_ATTRIBUTE,
				matchingPattern);
	}

	private String extractMatchingPattern(
			ContainerRequestContext requestContext) {
		ExtendedUriInfo extendedUriInfo = (ExtendedUriInfo) requestContext
				.getUriInfo();
		List<String> templates = new ArrayList<String>();
		for (UriTemplate matchedTemplate : extendedUriInfo
				.getMatchedTemplates()) {
			String template = matchedTemplate.getTemplate();
			templates.add(template.startsWith("/") ? template : "/" + template);
		}
		if (StringUtils.hasText(this.applicationPath)) {
			templates.add(this.applicationPath);
		}
		Collections.reverse(templates);
		return StringUtils.collectionToDelimitedString(templates, "");
	}

}

Some caveats:

  • I don't like that it uses Spring MVC's request attribute (see #7505)
  • Regular expressions in @Path don't work particularly well (see #7503)

@wilkinsona wilkinsona added this to the 1.5.0 RC1 milestone Nov 30, 2016

@wilkinsona wilkinsona self-assigned this Nov 30, 2016

@soldierkam

This comment has been minimized.

Show comment
Hide comment
@soldierkam

soldierkam Dec 1, 2016

The same happens for almost every HttpServlet used in Spring Boot application. MetricsFilter should create new keys only if it knows what mapping was used.

soldierkam commented Dec 1, 2016

The same happens for almost every HttpServlet used in Spring Boot application. MetricsFilter should create new keys only if it knows what mapping was used.

@r0nin

This comment has been minimized.

Show comment
Hide comment
@r0nin

r0nin Apr 21, 2017

@wilkinsona Just as a comment. I used this bean and it indeed solved the problem, but i got strange gauge metrics a la gauge.response..v1.controllerName.apiEndPoint, because the Jersey application path is /, the Controller has a @Path("/controllerName") and the method has a @Path("apiEndPoint").

The problem is probably on our side since the /controllerName might be slightly incorrect, but anyway in the returned concatenation of matching templates this leads to //v1/controllerName/apiEndPoint which obviously causing the .. part in the gauge metric. So to be safe against that, I simply replaced // by / before returning the result.

r0nin commented Apr 21, 2017

@wilkinsona Just as a comment. I used this bean and it indeed solved the problem, but i got strange gauge metrics a la gauge.response..v1.controllerName.apiEndPoint, because the Jersey application path is /, the Controller has a @Path("/controllerName") and the method has a @Path("apiEndPoint").

The problem is probably on our side since the /controllerName might be slightly incorrect, but anyway in the returned concatenation of matching templates this leads to //v1/controllerName/apiEndPoint which obviously causing the .. part in the gauge metric. So to be safe against that, I simply replaced // by / before returning the result.

@deki

This comment has been minimized.

Show comment
Hide comment
@deki

deki Apr 24, 2017

Contributor

The title of this issue should be changed as this also affects Apache CXF and other frameworks.

Contributor

deki commented Apr 24, 2017

The title of this issue should be changed as this also affects Apache CXF and other frameworks.

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona Aug 11, 2017

Member

The planned move to Micrometer-based metrics will get rid of the metrics filter.

Member

wilkinsona commented Aug 11, 2017

The planned move to Micrometer-based metrics will get rid of the metrics filter.

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona Jun 21, 2018

Member

We've decided to align 1.5.x with 2.0's current behaviour and fallback to unmapped when we can't get the best matching pattern from Spring MVC. This will avoid requests that aren't handled by Spring MVC from causing a potentially unbounded number of metrics to be created.

Member

wilkinsona commented Jun 21, 2018

We've decided to align 1.5.x with 2.0's current behaviour and fallback to unmapped when we can't get the best matching pattern from Spring MVC. This will avoid requests that aren't handled by Spring MVC from causing a potentially unbounded number of metrics to be created.

@wilkinsona wilkinsona changed the title from MetricsFilter doesn't consider path variables when using Jersey to MetricsFilter may create an unbounded number of metrics for requests with templated URI that are not handled by Spring MVC Jun 21, 2018

@wilkinsona wilkinsona changed the title from MetricsFilter may create an unbounded number of metrics for requests with templated URI that are not handled by Spring MVC to MetricsFilter may create an unbounded number of metrics for requests with a templated URI that are not handled by Spring MVC Jun 21, 2018

@wilkinsona wilkinsona added this to the 1.5.15 milestone Jun 21, 2018

@wilkinsona wilkinsona self-assigned this Jun 21, 2018

@deki

This comment has been minimized.

Show comment
Hide comment
@deki

deki Jun 21, 2018

Contributor

Great improvement for those who still stick to that version. Thanks!

Contributor

deki commented Jun 21, 2018

Great improvement for those who still stick to that version. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment