logstash-gelf-subsystem

This is a JBossAS7/Wildfly subsystem module to provide injection/JNDI bindings of GelfSender and Datenpumpe instances.

But wait: What is Datenpumpe?

TL;DR; Datenpumpe allows you to submit your own data structures to be processed by logstash, ElasticSearch and Kibana (the ELK stack)

And the longer version: Datenpumpe is a concept evolved from submitting log entries using GELF (Graylog Extended Log Format, see also https://github.com/mp911de/logstash-gelf). logstash-gelf was intended to allows simple submission of log events to logstash. With starting to use the ELK stack (see http://elasticsearch.org) you will explore the possibilities of centralized event and data management. This causes the need to be able to submit any data to your logstash for later resarch and statistics gathering. Datenpumpe takes proven working components and allows you to submit your own data strucures. This is useful for event processing of bussiness and technical events.

Example

public class MyServlet extends HttpServlet {

    @Resource(mappedName = "jndi:/jboss/datenpumpe")
    public Datenpumpe datenpumpe;
    
    @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp)  {
        
        Map<String, Object> message = new HashMap<>;
        message.put("uri", req.getRequestUri());
        message.put("resource", "MyServlet");
        message.put("event", "access");
        
        datenpumpe.submit(message);
    }
}

Or more sophisticated (using reflection to retrieve the fields from your model):

@Stateless
public class MyEjb {

    @Resource(mappedName = "jndi:/jboss/datenpumpe")
    public Datenpumpe datenpumpe;
    
    public void shoppingCartOrdered(ShoppingCart cart) {
        datenpumpe.submit(cart);
    }
}

public class ShoppingCart{

    private String cartId;
    private double amount;
    private String customerId;
    
    public String getCartId(){
        return cartId;
    }
    
    public double getAmount(){
        return amount;
    }
    
    public String getCustomerId(){
        return customerId;
    }
}

This results in a Gelf message like:

{ "timestamp": "1406797244.645",
  "facility": "logstash-gelf", 
  "_cartId": "the cart id", 
  "_amount": 9.27,
  "_customerId": "the customer id" }

How to get

JBoss AS7 and Wildfly modules are not binary compatible, therefore you need to download the right library.

• Maven Repo: http://oss.sonatype.org/content/repositories/snapshots/
• https://github.com/mp911de/logstash-gelf-subsystem/releases/tag/First-Snapshot
• JBoss AS7: https://oss.sonatype.org/content/repositories/snapshots/biz/paluch/logging/logstash-gelf-subsystem/7.0-SNAPSHOT/logstash-gelf-subsystem-7.0-20140801.182946-1-module.zip
• WildFly (AS8): https://oss.sonatype.org/content/repositories/snapshots/biz/paluch/logging/logstash-gelf-subsystem/8.0-SNAPSHOT/logstash-gelf-subsystem-8.0-20140801.184502-1-module.zip
• or via Maven:

JBoss AS7 Module Download:

<dependency>
    <groupId>biz.paluch.logging</groupId>
    <artifactId>logstash-gelf-subsystem</artifactId>
    <version>7.0.0-SNAPSHOT</version>
    <classifier>module</classifier>
</dependency>

Wildfly (AS8) Module Download:

<dependency>
    <groupId>biz.paluch.logging</groupId>
    <artifactId>logstash-gelf-subsystem</artifactId>
    <version>8.0.0-SNAPSHOT</version>
    <classifier>module</classifier>
</dependency>

How to install

Download the zipped module and drop the contents of the zip-file (without the top-level directory logstash-gelf-subsystem-(version)) into your JBoss' module directory (either below modules/system/layers/base, a new layer or below modules/ directly). The module zip contains also the dependency to logstash-gelf, so you don't need to download it separately.

Activating the subsystem

Add following lines to your JBoss config

In the extensions section (below server/extensions):

    <extension module="biz.paluch.logging.logstash-gelf-subsystem"/>

And anywhere in the profile secion (below server/profile):

    <subsystem xmlns="urn:biz.paluch.logging:logstash-gelf-subsystem:1.0">
    </subsystem>

Afterwards you can add Datenpumpe or GelfSender instances.

CLI Configuration

Adding a new Datenpumpe:

/subsystem=logstash-gelf-subsystem/datenpumpe="jndi:/jboss/datenpumpe"/:add(jndi-name="jndi:/jboss/datenpumpe", host="udp:logstash-host", port=12201)

Adding a new GelfSender:

/subsystem=logstash-gelf-subsystem/sender="jndi:/jboss/sender"/:add(jndi-name="jndi:/jboss/sender", host="udp:logstash-host", port=12201)

XML Configuration

Adding a new Datenpumpe:

<subsystem xmlns="urn:biz.paluch.logging:logstash-gelf-subsystem:1.0">
    <datenpumpe host="udp:logstash-host" port="12201" jndi-name="jndi:/jboss/datenpumpe" />
</subsystem>

Adding a new GelfSender:

<subsystem xmlns="urn:biz.paluch.logging:logstash-gelf-subsystem:1.0">
    <sender host="udp:logstash-host" port="12201" jndi-name="jndi:/jboss/sender" />
</subsystem>

Properties

• host: Hostname/IP-Address of the Logstash or Redis Host
  • tcp:(the host) for TCP, e.g. tcp:127.0.0.1 or tcp:some.host.com
  • udp:(the host) for UDP, e.g. udp:127.0.0.1 or udp:some.host.com
  • redis://[:REDISDB_PASSWORD@]REDISDB_HOST:REDISDB_PORT/REDISDB_NUMBER#REDISDB_LISTNAME , e.g. redis://:donttrustme@127.0.0.1:6379/0#myloglist or if no password needed redis://127.0.0.1:6379/0#myloglist
  • (the host) for UDP, e.g. 127.0.0.1 or some.host.com
• port: Port, default 12201
• jndi-name: JNDI Name of the binding

Notes on redis Connection

• IMPORTANT: for getting your logstash config right it is vital to know that we do LPUSH (list push and not channel method)
• The redis connection is done through jedis (https://github.com/xetorthio/jedis)
• The Url used as connection property is a java.net.URI , therefore it can have all nine components. we use only the following:
  • scheme (fixed: redis, directly used to determine the to be used sender class)
  • user-info (variable: only the password part is used since redis doesnt have users, indirectly used from jedis)
  • host (variable: the host your redis db runs on, indirectly used from jedis)
  • port (variable: the port your redis db runs on, indirectly used from jedis)
  • path (variable: only numbers - your redis db number, indirectly used from jedis)
  • fragment (variable: the listname we push the log messages via LPUSH, directly used)

License

• [The MIT License (MIT)] (http://opensource.org/licenses/MIT)

Contributing

Github is for social coding: if you want to write code, I encourage contributions through pull requests from forks of this repository. Create Github tickets for bugs and new features and comment on the ones that you are interested in.