Skip to content

Log back appender to publish data to a processor with in nifi using site-to-site protocol

License

Notifications You must be signed in to change notification settings

pradeepmvn/logback-nifi-appender

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

logback-nifi-appender

A Simple logback appender to publish data to NIFI using site-to-site protocol. Site-to-Site Protocol is more scalable, efficient means of communication with Nifi ports. Nifi on the other end can be used as a central tool to push data to multiple destinations.

With current state, there are two type of Centralized logging solutions - Agent based to tail files and Agent less utilizing the appender mechanism.

Agent Based

Utilizes an Agent running along side of the application.

  • Pros

    • Executes out side of application
    • Data is hold even if the receiver is down
  • Cons

    • consumes more memory/disk space
    • Might not be efficient with docker and micro- environments.

Agent Less:

  • Pros
    • No additional memory or process is needed
  • Cons
    • Data can be lost if the receiver fails to accept data
    • Data can be lost if there is a network latency

This appender combines the best of the two style. It predominantly works as a Agent Less but can spin an Agent when the receiver fails to accept data. There are multiple Fallback Strategies that can be leveraged on Failure of receiver.

features

  • Light Weight
  • Leverages Site to Site protocol for auto discovery of Nifi cluster and transaction confirmations
  • Fallback to a file if Nifi is unable to receive data.

Full configuration example

Add logback-nifi-appender and nifi-site-to-site-client as dependencies to your project in pom.

Dependency in pom.xml
<dependency>
    <groupId>com.github.logback.nifi.appender</groupId>
    <artifactId>logback-nifi-appender</artifactId>
    <version>0.0.1</version>
    <scope>runtime</scope>
</dependency>
<dependency>
	<groupId>org.apache.nifi</groupId>
	<artifactId>nifi-site-to-site-client</artifactId>
	<version>1.4.0</version>
</dependency>
Configuration in logback.xml

Below are the only required properties for this appender.

<appender name="NIFI_APPENDER" class="com.github.logback.NifiAppender">
	<url>http://localhost:8000/nifi/</url>
	<receiverPortName>FromLogback</receiverPortName>
	<receiverPortId>3f5c5ece-4e8f-1c7f-ffff-ffff8a1f130b</receiverPortId>
	<pushInterval>1000</pushInterval>
</appender>
FAQ
  • Q: Why another appender? Why not UDP appender?
    A: UDP appender can be user to push data to Nifi single host. But it cannot automatically discover cluster and distribute the data.

  • Q: Any other encoders supported?
    A: You can use any other encoders that come with logback and place it in the configuration for the appender.

  • Q: Does this hold data when nifi is down?
    A: Yes. This appender has multiple fallback implementations. Default one is a Console appender. But there is also a File appender which gets used when a nifi is down or unable to accept requests.

  • Q: What is the retry mechanism?
    A: If Nifi server is down a fallback is used and main appender uses Exponential backoff Strategy for next retry.

References

About

Log back appender to publish data to a processor with in nifi using site-to-site protocol

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages