Home
Qiao (橋, the Chinese character for “bridge”, pronounced as Chiao) is a standalone service that streams event data from a log in real-time and delivers them to one or more destinations. Similar to Unix 'tail -F' command, Qiao keeps following a log file when it rotates. One notable difference of Qiao from other log streamers is that Qiao keeps track of read-cursor position and stores the offset on the disk efficiently. In the event that the agent restarts, Qiao is able to continue processing at the file position where it left off before it terminated even if the file was rotated out. Furthermore, for a space-conscious log producer, Qiao can process logs in a compacted binary format, in addition to regular text format (a single ‘line’ of text followed by line feed (‘\n’)). Qiao also supports Avro format out of box.
- Java 1.7+
- Small amount of disk space for cursor position cache and history cache
Qiao consists of one or more data funnels. Each data funnel transports data to a set of destinations. Data streams into the funnel via an injector and flows out via one or more emitters. The following diagram depicts its data flow:
- Injector - injects data from external source into data funnel
- Emitter - ejects data out of data funnel and delivers to the destination
- Data Pipe - an intermediate storage to hold data before ejection
- Data Spray - a mediator that forwards items in data pipe to one or more emitters