Mp Eviction Policy Implemenation

[lloydfischer]

I'm just starting to evaluate. My use case would be to process log messages. Each message has a session identifier. I'd like to have Dempsy instantiate a new MP for each session. But once the session completes successfully or fails I'd like the MP to publish an output event and cause the container to destroy the MP. Similarly, if the MP has not received a message for a "long time" is should publish an event and signal its destruction.

I can't see that this functionality exists. Did I miss it? If not, is this kind of thing out of scope, or on the road-map?

[jimfcarroll]

Hi @lloydfischer,

You're referring to what we've been calling "Eviction." I accidentally overlooked adding that as an issue from our current backlog when I setup us the project. My apologies. It's in the plans.

The way it will work is you will have a method on an MP, annotated with an @ Evictable (or something similar). The container will periodically ask each Mp if it is "evictable" by invoking that method. If the Mp indicates that it is, it will be passivated and removed from the container.

I have an MP lifecycle diagram I'll be posting that also includes eviction. I was going to include that with an update to the documentation this weekend.

However, for your particular use case, if you are not required to respond in real time to particular log messages, or are not doing real-time analytics, then I would recommend using tools specifically geared toward log consolidation and offline analytics in a distributed system like Flume (we are planning to use LogStash).

If however, you are trying to do analytics in real-time, then Dempsy makes sense.

Do you mind if I take this issue and edit the title so that it will become the enhancement entry for eviction?

Thanks
Jim

[lloydfischer]

Thanks for the answer. Real-analytics and dash boarding is the goal so
Dempsy seems a perfect fit.

Feel free to edit the issue or replace it, at your convenience.

Sent from my iPhone

On Apr 27, 2012, at 11:57 AM, Jim Carroll
reply@reply.github.com
wrote:

Hi @lloydfischer,

You're referring to what we've been calling "Eviction." I accidentally overlooked adding that as an issue from our current backlog when I setup us the project. My apologies. It's in the plans.

The way it will work is you will have a method on an MP, annotated with an @ Evictable (or something similar). The container will periodically ask each Mp if it is "evictable" by invoking that method. If the Mp indicates that it is, it will be passivated and removed from the container.

I have an MP lifecycle diagram I'll be posting that also includes eviction. I was going to include that with an update to the documentation this weekend.

However, for your particular use case, if you are not required to respond in real time to particular log messages, or are not doing real-time analytics, then I would recommend using tools specifically geared toward log consolidation and offline analytics in a distributed system like Flume (we are planning to use LogStash).

If however, you are trying to do analytics in real-time, then Dempsy makes sense.

Do you mind if I take this issue and edit the title so that it will become the enhancement entry for eviction?

Thanks
Jim

---

Reply to this email directly or view it on GitHub:
#22 (comment)

[jimfcarroll]

@lloydfischer , one other thing to keep in mind. Dempsy currently follows the original Apache S4 paradigm that guaranteed delivery is sacrificed for lag reduction. Dempsy is designed to allow this to be swapped in through the implementation of a different transport - but this is further out on the backlog.

[jimfcarroll]

Eviction, as well as some of the other future enhancements in this list, are now covered in the documentation under the section on the Mp Lifecycle.

I will be adding a section per lifecycle phase and probably reorganizing those under the section on the Application Developer Api.

If anyone has suggestions, feel free to comment here.

[jimfcarroll]

This was completed with the above PRs