Threading

Loghub try to using as many core as possible.

To reach this goal, it's based on ØMQ that is used to move around events between it's components without using any synchronization primitives.

But still some precautions must be used and the threading model must be understood to fully reach Loghub's performance goals.

Input

Input object are mono threaded as usually they listen on a single network port that will be difficult to share. The developper must then take care of blocking code.

Output

Processors

Each processor step in a pipeline step exists as a single unit. But if it was created with the thread bean, the given number of threads will be created and share this instance to process events. It allows avoid some slow processors to hange the full pipeline. If a processor class is used in different pipeline steps will be different object. If many processor are chained without specifying a number of threads, they share the same single thread.

For example, with the following pipeline :

pipeline[test] { loghub.FastProcessor
                | loghub.FastProcessor2
                | loghub.SlowProcessor { threads: 4}
                | loghub.FastProcessor
               }

4 processor instance will be created, 3 loghub.FastProcessor and one loghub.SlowProcessor. But 6 threads will be used to process events. 1 for the first set of loghub.FastProcessor and loghub.FastProcessor2 instances, on for the last loghub.FastProcessor instance and 4 that will share the loghub.SlowProcessor . As a processor is generally stateless, that should not be a problem. But if some non thread safe object are to be used, extra care must be used, a java.lang.ThreadLocal is a very efficient solution to this problem.