Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Post events from discovery server to ECAP that will allow me to generate the necessary histogram data #6

Closed
wants to merge 8 commits into from

5 participants

@wyan0220

... the necessary histogram data.

...ava/com/proofpoint/discovery/DynamicAnnouncement.java
@@ -157,4 +146,11 @@ public DynamicAnnouncement build()
return new DynamicAnnouncement(environment, pool, location, services);
}
}
+
+ @Override
+ public String toString()
+ {
+ return String.format("DynamicAnnouncement{environment:%s,location:%s,pool:%s,services:%s}", environment,

Use Guava Objects.toStringHelper

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...om/proofpoint/discovery/monitor/DiscoveryMonitor.java
((19 lines not shown))
+ {
+ this.eventClient = Preconditions.checkNotNull(eventClient);
+ this.stats = Preconditions.checkNotNull(stats);
+ }
+
+ public void monitorDiscoveryEvent(DiscoveryEventType type, boolean success, String remoteAddress, String requestUri, String requestBodyJson, long startTime)
+ {
+ eventClient.post(new DiscoveryEvent(type, success, remoteAddress, requestUri,
+ requestBodyJson, new Duration(System.nanoTime() - startTime, TimeUnit.NANOSECONDS)));
+ stats.addStats(type, success, startTime);
+ }
+
+ public void monitorDiscoveryFailureEvent(DiscoveryEventType type, Exception e, String requestUri)
+ {
+ eventClient.post(new DiscoveryFailureEvent(type, e, requestUri));
+ log.warn(e, String.format("discovery request [%s] failed", requestUri));

You shouldn't log things that are then re-thrown. This causes duplicate log messages and is confusing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
.../com/proofpoint/discovery/monitor/DiscoveryStats.java
((134 lines not shown))
+ public TimedStat getDynamicAnnouncementProcessingTime()
+ {
+ return dynamicAnnouncementProcessingTime;
+ }
+
+ @Managed
+ @Nested
+ public TimedStat getDynamicAnnouncementDeleteProcessingTime()
+ {
+ return dynamicAnnouncementDeleteProcessingTime;
+ }
+
+ public void addStats(DiscoveryEventType type, boolean success, long startTime)
+ {
+ if (success) {
+ switch (type) {

Use EnumMap

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
.../com/proofpoint/discovery/monitor/DiscoveryStats.java
@@ -0,0 +1,205 @@
+package com.proofpoint.discovery.monitor;
+
+import com.proofpoint.stats.TimedStat;
+import com.proofpoint.units.Duration;
+import org.weakref.jmx.Managed;
+import org.weakref.jmx.Nested;
+
+import java.util.concurrent.atomic.AtomicLong;
+
+public class DiscoveryStats
+{
+ private final AtomicLong serviceQuerySuccessCount = new AtomicLong();

There should be a way to use EnumMap for all of these and still have it work with JMX.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...proofpoint/discovery/DynamicAnnouncementResource.java
((19 lines not shown))
{
- if (!nodeInfo.getEnvironment().equals(announcement.getEnvironment())) {
- return Response.status(BAD_REQUEST)
- .entity(format("Environment mismatch. Expected: %s, Provided: %s", nodeInfo.getEnvironment(), announcement.getEnvironment()))
- .build();
- }
+ boolean success = true;

This is ugly having all this logic in the resource. It's much cleaner to wrap the store and put the logic in the wrapper. You can also consider using a dynamic proxy (java.lang.reflect.Proxy) in the wrapper to eliminate boilerplate.

@anandsomani Owner

Please ignore if you are commenting on the boiler plate code for events. I thought this was on the environment check...

Generally I agree that business logic belongs in store and not in resource. I have seen a tendency to add business logic to resource, which is a bad idea. Resource is just one way of exposing this API and should only contain API to internal <=> mapping.

In this case, I cannot pin point why, but I feel leaving this check here makes more sense since this in someways is the API contract.

I was referring to all the stats logic, not the environment mismatch check. The check is fine.

hi David, i implemented a wrapper class for event monitoring logic. could you kindly review it? Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@wyan0220 wyan0220 changes based on review:
1. use guava toStringHelper
2. use EnumMap
5577c22
@wyan0220

Hi David, thanks for your comments. I've made the changes for most of them. Regarding dynamic proxy, i was told we should stay away from it if possible. For me I feel like it's a little overkill in this case...

@electrum

This check doesn't seem necessary: it's fine to pass null to toStringHelper. If you do want it to be displayed as an empty string rather than null, then use Strings.nullToEmpty.

@electrum

It's cleaner to call this eventTypeStats and to make the type Map rather than EnumMap. Always prefer to use the most generic collection class when declaring variables (e.g. List vs ArrayList or Map vs HashMap).

@electrum

It can make the code a little shorter and easier to read if you static import the enum values.

@wyan0220

hi David, i implemented a wrapper class for event monitoring logic. could you kindly review it? Thanks

@anandsomani
@anandsomani

instead of monitor() maybe use execute(), since the purpose is not to monitor but to execute something that is being monitored

.../java/com/proofpoint/discovery/EventMonitorProxy.java
((13 lines not shown))
+ private final UriInfo uriInfo;
+ private final HttpServletRequest httpServletRequest;
+ private final String requestBody;
+
+ public EventMonitorProxy(DiscoveryMonitor discoveryMonitor, DiscoveryEventType type, UriInfo uriInfo, HttpServletRequest httpServletRequest, String requestBody)
+ {
+ this.discoveryMonitor = discoveryMonitor;
+ this.type = type;
+ this.uriInfo = uriInfo;
+ this.httpServletRequest = httpServletRequest;
+ this.requestBody = requestBody;
+ }
+
+ public T execute()
+ {
+ boolean success = true;

Prefer the opposite logic here. For example, if your code throws an Error, you'll log a success event. Better to initialize success to false, and set it to true after doWork returns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...proofpoint/discovery/DynamicAnnouncementResource.java
((19 lines not shown))
{
- if (!nodeInfo.getEnvironment().equals(announcement.getEnvironment())) {
- return Response.status(BAD_REQUEST)
- .entity(format("Environment mismatch. Expected: %s, Provided: %s", nodeInfo.getEnvironment(), announcement.getEnvironment()))
- .build();
- }
+ EventMonitorProxy<Response> eventMonitor = new EventMonitorProxy<Response>(discoveryMonitor, DiscoveryEventType.DYNAMICANNOUNCEMENT, uriInfo, httpServletRequest, announcement.toString())
+ {
+ @Override
+ public Response doWork()
+ {
+ if (!nodeInfo.getEnvironment().equals(announcement.getEnvironment())) {
+ return Response.status(BAD_REQUEST)
+ .entity(format("Environment mismatch. Expected: %s, Provided: %s", nodeInfo.getEnvironment(), announcement.getEnvironment()))

In the event of this BAD_REQUEST result, you will still log a success event in the monitor. Is that really what you want?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@markkent markkent commented on the diff
.../proofpoint/discovery/monitor/DiscoveryEventType.java
@@ -0,0 +1,11 @@
+package com.proofpoint.discovery.monitor;
+
+public enum DiscoveryEventType

Can't you make this the event object? Seems like it would be neater.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@markkent markkent commented on the diff
.../com/proofpoint/discovery/monitor/DiscoveryEvent.java
@@ -0,0 +1,127 @@
+package com.proofpoint.discovery.monitor;
+
+import com.google.common.base.Objects;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import com.proofpoint.event.client.EventField;
+import com.proofpoint.event.client.EventType;
+import com.proofpoint.units.Duration;
+
+@EventType("platform:type=discovery,name=discovery")

We're going to end up bifurcating discovery because of this - this event type name isn't consistent with the naming scheme for v2 events.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
.../proofpoint/discovery/StaticAnnouncementResource.java
((66 lines not shown))
{
- return new Services(nodeInfo.getEnvironment(), store.getAll());
+ EventMonitorProxy<Services> eventMonitor = new EventMonitorProxy<Services>(discoveryMonitor, DiscoveryEventType.STATICANNOUNCEMENTLIST, uriInfo, httpServletRequest, "")

Consider adding static factory methods to make this code easier to read.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...proofpoint/discovery/DynamicAnnouncementResource.java
((59 lines not shown))
{
- if (!dynamicStore.delete(nodeId)) {
- return Response.status(NOT_FOUND).build();
- }
+ EventMonitorProxy<Response> eventMonitor = new EventMonitorProxy<Response>(discoveryMonitor, DiscoveryEventType.DYNAMICANNOUNCEMENTDELETE, uriInfo, httpServletRequest, "")

This still looks kinda ugly. Is there a way to make it neater?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@markkent markkent commented on the diff
.../proofpoint/discovery/monitor/MonitorInterceptor.java
@@ -0,0 +1,40 @@
+package com.proofpoint.discovery.monitor;
+
+import com.google.inject.Inject;
+import org.aopalliance.intercept.MethodInterceptor;
+import org.aopalliance.intercept.MethodInvocation;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.core.UriInfo;
+
+public class MonitorInterceptor implements MethodInterceptor
+{
+ @Inject
+ DiscoveryMonitor discoveryMonitor;

Is field injection the only way to get this to work? I'm guessing so. Shame

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
.../proofpoint/discovery/monitor/MonitorInterceptor.java
((6 lines not shown))
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.core.UriInfo;
+
+public class MonitorInterceptor implements MethodInterceptor
+{
+ @Inject
+ DiscoveryMonitor discoveryMonitor;
+
+ @Override
+ public Object invoke(MethodInvocation methodInvocation)
+ throws Throwable
+ {
+ boolean success = false;
+ long startTime = System.nanoTime();
+ DiscoveryEventType type = methodInvocation.getMethod().getAnnotation(MonitorWith.class).value();

Hrm... I haven't really looked at this, but we probably don't want to be reading the annotations every time through this code if we can avoid it. Wouldn't that be wasteful?

Also, all this getArguments()[index] looks odd.

I need to read up on how this junk works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...a/com/proofpoint/discovery/DiscoveryServerModule.java
@@ -62,6 +69,15 @@ public void configure(Binder binder)
binder.bind(StaticStore.class).to(ReplicatedStaticStore.class).in(Scopes.SINGLETON);
binder.install(new ReplicatedStoreModule("static", ForStaticStore.class, PersistentStore.class));
bindConfig(binder).prefixedWith("static").to(PersistentStoreConfig.class);
+
+ // event
+ eventBinder(binder).bindEventClient(DiscoveryEvent.class);
+ eventBinder(binder).bindEventClient(DiscoveryFailureEvent.class);
+
+ // interceptor
+ MonitorInterceptor interceptor = new MonitorInterceptor();
+ binder.requestInjection(interceptor);
+ binder.bindInterceptor(Matchers.any(), Matchers.annotatedWith(MonitorWith.class), interceptor);

A couple of things here. Firstly, put the monitor-related things in a module that you bind separately.

Also, perhaps also require the class to be annotated (perhaps with @Monitored, and then the methods with @MonitorWithEvent()) Maybe not. I'm not sure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...proofpoint/discovery/DynamicAnnouncementResource.java
@@ -49,7 +53,8 @@ public DynamicAnnouncementResource(DynamicStore dynamicStore, NodeInfo nodeInfo)
@PUT
@Consumes(MediaType.APPLICATION_JSON)
- public Response put(@PathParam("node_id") Id<Node> nodeId, @Context UriInfo uriInfo, DynamicAnnouncement announcement)
+ @MonitorWith(DYNAMICANNOUNCEMENT)

yeah, I don't like the way "MonitorWith" reads here. (yes, I know it was my name) Find a better name for the annotation? At the very least, the annotations appear to be stating properties of the method, so "MonitoredWith" seems better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...in/java/com/proofpoint/discovery/ServiceResource.java
((7 lines not shown))
{
return new Services(node.getEnvironment(), union(dynamicStore.get(type), staticStore.get(type)));
}
@GET
@Produces(MediaType.APPLICATION_JSON)
- public Services getServices()
+ @MonitorWith(SERVICEQUERY)
+ public Services getServices(@Context HttpServletRequest httpServletRequest, @Context UriInfo uriInfo)

I think it's really really clunky that you have to add these two parameters, which the IDE will claim aren't used in the function, just because the interceptor expects to rely on their value. Is there a better way of doing this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@johngmyers
Owner

Not going to use events for instrumenting discovery server. Closing out.

@johngmyers johngmyers closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jul 9, 2012
  1. @wyan0220

    Post events from discovery server to ECAP that will allow me to gener…

    wyan0220 authored
    …ate the necessary histogram data.
Commits on Jul 10, 2012
  1. @wyan0220

    changes based on review:

    wyan0220 authored
    1. use guava toStringHelper
    2. use EnumMap
Commits on Jul 11, 2012
  1. @wyan0220

    added missing binding for events. also change @EventField return type…

    wyan0220 authored
    … to be String or primitive
Commits on Jul 12, 2012
  1. @wyan0220

    changes based on review

    wyan0220 authored
Commits on Jul 13, 2012
  1. @wyan0220
Commits on Jul 17, 2012
  1. @wyan0220
Commits on Jul 18, 2012
  1. @wyan0220
Commits on Aug 13, 2012
  1. @wyan0220
Something went wrong with that request. Please try again.