log the Hystrix metrics to local file or DB

[silentAllay]

Can I log the Hystrix metrics to local file/DB with custom programming? Because Hystrix dashboard only show realtime metrix, what if I want to look back the history metrix?

[mukteshkrmishra]

You can do but you need to implement a custom metrics publisher.

[silentAllay]

I did the code as following, is it OK?

HystrixPlugins.getInstance().registerMetricsPublisher(HystrixServoMetricsPublisher.getInstance());
final List observers = new ArrayList();
observers.add(new MetricObserver() {
@OverRide
public void update(List metrics) {
// TODO Auto-generated method stub
for(Metric m:metrics) {
System.out.println("metrix update"+JSON.toJSONString(m));
}
}
@OverRide
public String getName() {
return null;
}
});
PollScheduler.getInstance().start();
PollRunnable task = new PollRunnable(new MonitorRegistryMetricPoller(), BasicMetricFilter.MATCH_ALL, true, observers);
PollScheduler.getInstance().addPoller(task, 5, TimeUnit.SECONDS);

[mattrjacobs]

Internally at Netflix, we log metrics to Atlas for historical time-series analysis. As @mukteshkrmishra observed, we use a custom metrics publisher for that (HystrixServoMetricsPublisher).

You should just create your own metrics publisher (extend from HystrixMetricsPublisher). Then, you're free to do anything you want with metrics, including writing them to files.

If you're interested in getting a review (or even submitting a PR), feel free to post it here - happy to review for you

[silentAllay]

@mattrjacobs thank you, I use HystrixServoMetricsPublisher and a customer observer to get the metrics,
Is there a doc to explain the metrics meaning?
metrics as following:

Metric{config=MonitorConfig{name=propertyValue_corePoolSize, tags=com.netflix.hystrix.contrib.servopublisher.HystrixServoMetricsPublisherThreadPool$1@50f556a3,com.netflix.hystrix.contrib.servopublisher.HystrixServoMetricsPublisherThreadPool$2@5e32a2cd, policy=DefaultPublishingPolicy}, timestamp=1470187468329, value=10}

[mukteshkrmishra]

You can refer servo java documents and dropwizard metrics api documents from git.

Sent from my iPhone

On Aug 2, 2016, at 9:38 PM, killjason <notifications@github.commailto:notifications@github.com> wrote:

@mattrjacobshttps://github.com/mattrjacobs thank you, I use HystrixServoMetricsPublisher and a customer observer to get the metrics,
Is there a doc to explain the metrics meaning?
metrics as following:

Metric{config=MonitorConfig{name=propertyValue_corePoolSize, tags=com.netflix.hystrix.contrib.servopublisher.HystrixServoMetricsPublisherThreadPool$1@50f556a3,com.netflix.hystrix.contrib.servopublisher.HystrixServoMetricsPublisherThreadPool$2@5e32a2cd, policy=DefaultPublishingPolicy}, timestamp=1470187468329, value=10}

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/1298#issuecomment-237135920, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKdlY3tDcIQod8n1kKANPuYvQoB7KqqNks5qcBs9gaJpZM4Jadq9.

[mukteshkrmishra]

@mattrjacobs : Do we have any code piece to show how we are pushing it to Atlas. PS: I have implemented a solution to push Hystrix Metrics to another time series and just wanted to see the Atlas implementation.

[mattrjacobs]

@killjason Again, I should repeat that using HystrixServoMetricsPublisher directly is not what you want to do. You should create a new metrics publisher that doesn't know anything about Servo (since you're not using it)

That data type is a Servo data type (used to publish to Atlas). Since you're not publishing to Atlas, that's not the right type to engage with. By extending HystrixMetricsPublisher, you will directly be dealing with HystrixCommandMetrics/HystrixThreadPoolMetrics/HystrixCollapserMetrics, which has no Servo dependency

[silentAllay]

@mattrjacobs OK, I got it ,I will try it and give a feed back later, thank you.

[silentAllay]

@mattrjacobs I created a new custom metrics publisher, and got the readable metrics,Thank you.

[mukteshkrmishra]

@killjason :: can tou share the implementation you did??

[silentAllay]

@mukteshkrmishra ,sure, if there is anything wrong ,please tell me, thank you.

public class CustomPublisher extends HystrixMetricsPublisher {
    @Override
    public HystrixMetricsPublisherCollapser getMetricsPublisherForCollapser(HystrixCollapserKey collapserKey,
            HystrixCollapserMetrics metrics, HystrixCollapserProperties properties) {

        return super.getMetricsPublisherForCollapser(collapserKey, metrics, properties);
    }
    @Override
    public HystrixMetricsPublisherCommand getMetricsPublisherForCommand(HystrixCommandKey commandKey,
            HystrixCommandGroupKey commandGroupKey, HystrixCommandMetrics metrics, HystrixCircuitBreaker circuitBreaker,
            HystrixCommandProperties properties) {
        return super.getMetricsPublisherForCommand(commandKey, commandGroupKey, metrics, circuitBreaker, properties);
    }
    @Override
    public HystrixMetricsPublisherThreadPool getMetricsPublisherForThreadPool(HystrixThreadPoolKey threadPoolKey,
            HystrixThreadPoolMetrics metrics, HystrixThreadPoolProperties properties) {
        return super.getMetricsPublisherForThreadPool(threadPoolKey, metrics, properties);
    }
}
private static void processMetrics() {
        // Registry plugin with hystrix
        HystrixPlugins.getInstance().registerMetricsPublisher(new CustomPublisher());
        final List<MetricObserver> observers = new ArrayList<MetricObserver>();
        observers.add(new CustomObserver("CustomObserver"));
        PollScheduler.getInstance().start();
        PollRunnable task = new PollRunnable(new MonitorRegistryMetricPoller(), BasicMetricFilter.MATCH_ALL, true, observers);
        PollScheduler.getInstance().addPoller(task, 5, TimeUnit.SECONDS);
    }
public class CustomObserver extends BaseMetricObserver {
    public CustomObserver(String name) {
        super(name);
    }
    @Override
    public void updateImpl(List<Metric> metrics) {
        for(Metric m:metrics) {
            // can write the metrics to local DB/file, or send to MQ like kafka
            System.out.println(m.toString());
        }
    }   
}

[mukteshkrmishra]

@killjason : Code looks good. Couple of suggestions: You can write own poller or observer as well (instead of using servo).

But overall this is the perfect strategy to pull out metrics from Hystrix.

[mukteshkrmishra]

@killjason :: One quick question, where are you registering your processMetrics() method, is it inside PostConstruct of init of a class or where?

Reason I am asking is to know what practice you are following..

[mukteshkrmishra]

@killjason : Any updates on this. As the above code seems to be bugy.
@mattrjacobs : Your thoughts on above code published by @killjason .

[silentAllay]

@mukteshkrmishra, processMetrics() is registering in Springboot main class.

[mukteshkrmishra]

@killjason : got it. In a typical web app env, I would prefer it to register as under @PostConstruct method. Never mind, its a good use case.

[silentAllay]

@mukteshkrmishra, thank you , glad to discuss with you.

[mattrjacobs]

Yeah, this seems fine

[mausamgautam]

@silentAllay can you share complete code of PostConstruct please? Even your complete code to write metric on console is not compiling for me.I have all dependancies.

[doanthai]

@mausamgautam This is postContructor in main class (change processMetrics is public static method)
@PostConstruct private void postConstruct() { CustomPublisher.processMetrics(); }