Threading

Loghub uses threads transparently to do parallel processing and don't lock.

For developpement it's should me transparent, but anyway some precautions must be used and the threading model must be understood to fully reach Loghub's performance goals.

Input

Input object are mono threaded as usually they listen on a single network port that will be difficult to share. The developper must then take care of blocking code.

Output

Processors

Each processor step in a pipeline step exists as a single unit. But if it was created with the threads argument, the given number of threads will be created and share this instance to process events. It prevents some slow processors to hang the full pipeline. If a processor class is used in different pipeline steps, they will be different instances. If many processor are chained without specifying a number of threads, they share the same single thread.

For example, with the following pipeline :

pipeline[test] { loghub.FastProcessor
                | loghub.FastProcessor2
                | loghub.SlowProcessor { threads: 4}
                | loghub.FastProcessor
               }

4 processor instance will be created, 3 loghub.FastProcessor and one loghub.SlowProcessor. But 6 threads will be used to process events. 1 for the first set of loghub.FastProcessor and loghub.FastProcessor2 instances, on for the last loghub.FastProcessor instance and 4 that will share the loghub.SlowProcessor . As a processor is generally stateless, that should not be a problem. But if some non thread safe object are to be used, extra care must be used, a java.lang.ThreadLocal is a very efficient solution to this problem.