Move TransformFilter functionality into the inputs #166

Closed
rafrombrc opened this Issue May 2, 2013 · 2 comments

1 participant

@rafrombrc
Mozilla Services member

Currently the TransformFilter depends on the message router doing most of the hard work parsing text and extracting data. Unfortunately, since this is much slower than most of the other routing that has to happen, this creates a throughput back pressure that impacts all Heka traffic, when actually it should be possible to be processing non-text-parsing messages in parallel w/o much interference.

This inappropriate back pressure would be relieved by moving the text parsing into the inputs themselves. One approach would be to implement the same functionality as a decoder, which would require fleshing out a way for an input to handle multiple decoders for differently structured line formats. Related: #99.

@rafrombrc rafrombrc was assigned May 2, 2013
@rafrombrc
Mozilla Services member

Started this process in pull request #170 by converting the TransformFilter into a LoglineDecoder, so the back pressure from a slow decoder will only impact the inputs that are using the decoder. Might end up pulling more of the functionality out so that it can be shared by other decoder implementations, but this resolves the initial issue.

@rafrombrc
Mozilla Services member

#170 merged in f3aa9f1

@rafrombrc rafrombrc closed this May 13, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment