-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mp Eviction Policy Implemenation #22
Comments
Hi @lloydfischer, You're referring to what we've been calling "Eviction." I accidentally overlooked adding that as an issue from our current backlog when I setup us the project. My apologies. It's in the plans. The way it will work is you will have a method on an MP, annotated with an @ Evictable (or something similar). The container will periodically ask each Mp if it is "evictable" by invoking that method. If the Mp indicates that it is, it will be passivated and removed from the container. I have an MP lifecycle diagram I'll be posting that also includes eviction. I was going to include that with an update to the documentation this weekend. However, for your particular use case, if you are not required to respond in real time to particular log messages, or are not doing real-time analytics, then I would recommend using tools specifically geared toward log consolidation and offline analytics in a distributed system like Flume (we are planning to use LogStash). If however, you are trying to do analytics in real-time, then Dempsy makes sense. Do you mind if I take this issue and edit the title so that it will become the enhancement entry for eviction? Thanks |
Thanks for the answer. Real-analytics and dash boarding is the goal so Feel free to edit the issue or replace it, at your convenience. Sent from my iPhone On Apr 27, 2012, at 11:57 AM, Jim Carroll
|
@lloydfischer , one other thing to keep in mind. Dempsy currently follows the original Apache S4 paradigm that guaranteed delivery is sacrificed for lag reduction. Dempsy is designed to allow this to be swapped in through the implementation of a different transport - but this is further out on the backlog. |
Eviction, as well as some of the other future enhancements in this list, are now covered in the documentation under the section on the Mp Lifecycle. I will be adding a section per lifecycle phase and probably reorganizing those under the section on the Application Developer Api. If anyone has suggestions, feel free to comment here. |
-- Validation for eviction method. -- handling eviction state message processors duing output and message handling. --To Do: Thread to check and clean message processors instances. Also references issue Dempsy#22
This was completed with the above PRs |
I'm just starting to evaluate. My use case would be to process log messages. Each message has a session identifier. I'd like to have Dempsy instantiate a new MP for each session. But once the session completes successfully or fails I'd like the MP to publish an output event and cause the container to destroy the MP. Similarly, if the MP has not received a message for a "long time" is should publish an event and signal its destruction.
I can't see that this functionality exists. Did I miss it? If not, is this kind of thing out of scope, or on the road-map?
The text was updated successfully, but these errors were encountered: