Add journald input

[jordansissel]

(Requested at Fossetcon during @untergeek's talk by folks from Red Hat & CoreOS)

API docs here: http://www.freedesktop.org/software/systemd/man/sd-journal.html

Journal api seems fairly stable over the past few years, though any help understanding exactly which APIs are known stable and if that API is available in the systemd's shipped in RHEL 7, Fedora, etc.

Either way, systemd is popular, and streaming the journal into Logstash seems like a pretty great win for users :)

# Years where the first line a public method definition changed in sd-journal
% git blame journal/sd-journal.c | grep _public_ | fex '(2)1 {-4}-1' | sort | uniq -c
     35 2012
      6 2013

[stuart-warren]

I just use something like:

[Unit]
Description=CoreOS Logstash
After=docker.service
Requires=docker.service

[Service]
Restart=always
ExecStartPre=-/usr/bin/docker pull you/logstashcontainer
ExecStart=/bin/bash -c "/usr/bin/journalctl -b -o json -f | /usr/bin/docker run -i you/logstashcontainer"

[X-Fleet]
X-Conflicts=logstash.*.service

with similar config to:

input {
  stdin {
    codec => "json_lines"
    type => "stdin"
    tags => [ "coreos" ]
  }
}
filter {
  mutate {
    gsub => [ "__REALTIME_TIMESTAMP", ".{3}$", "" ]
    rename => [ "MESSAGE", "message" ]
  }
  # hacky way to check if json
  if ( [message] =~ /^\{".*\}$/ ) {
    json {
      source => "message"
      add_tag => [ "json" ]
    }
  }
  date {
    match => [ "__REALTIME_TIMESTAMP", "UNIX_MS"]
    timezone => "UTC"
  }
}
output {
  elasticsearch_http {
    host => "your.elasticsearch.com"
    index => "coreos-%{+YYYY.MM.dd}"
    port => 9200
    document_id => "%{__CURSOR}"
  }
}

[torrancew]

I'm happy to take a stab at this, as I've done some journald API work in the past. I'm pretty confident that the lion's share of journald API endpoints we'll need to use have been around for several releases, but I'll check that out.

Things to check:

• systemd install base (with version numbers)
• journald methods to be used

https://github.com/ledbettj/systemd-journal is most likely the prime solution, and if there are any legacy systemd compatibility issues we can ideally incorporate them upstream

[torrancew]

Worth noting, that for CoreOS users, this may not be an ideal solution, as it assumes that you can run logstash on the box you want to pull journals from. Perhaps in the long-term, integrating similar functionality into logstash-forwarder, or creating a protocol-compatible tool for journal, would be in order, since single-binary deployments are handy on that platform.

[stuart-warren]

You would run Logstash in a Docker container and probably need to use Docker volumes to provide access to the hosts journal files. I prefer this route to mine above using stdin.

Oh and the library is at https://github.com/ledbettj/systemd-journal instead, no d on the end

[torrancew]

@stuart-warren good catch on the typo, will edit and fix that now. Have you tried using the journald API in that way? I'm not sure if it will function or not, but can try to test.

[stuart-warren]

@torrancew Yup it works fine
On CoreOS:

docker run -ti -v /var/log/journal:/var/log/journal centos:centos7 bash
yum install -y rubygems ruby-devel
yum groupinstall -y 'Development Tools'
gem install systemd-journal ffi
irb
require 'systemd/journal'
j = Systemd::Journal.new
j.seek(:tail)
j.watch do |entry|
  puts entry.message
end

Logging into the CoreOS box using another shell outputs logs.

[torrancew]

@stuart-warren excellent! Thanks for your feedback!

[stuart-warren]

@torrancew

So I tried getting this working in Logstash 1.4.2 and hit the following issue:

# java -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -S gem install ffi systemd-journal
# java -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -S irb
irb(main):001:0> require 'systemd/journal'
NameError: undefined method 'read_uint64' for class 'FFI::MemoryPointer'
    from org/jruby/RubyModule.java:2283:in `alias_method'
    from /opt/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/systemd-journal-1.2.1/lib/systemd/ffi_size_t.rb:13:in `MemoryPointer'
    from /opt/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/systemd-journal-1.2.1/lib/systemd/ffi_size_t.rb:4:in `(root)'
    from org/jruby/RubyKernel.java:1085:in `require'
    from file:/opt/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:73:in `require'
    from file:/opt/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:71:in `require'
    from /opt/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/systemd-journal-1.2.1/lib/systemd/journal.rb:1:in `(root)'
    from org/jruby/RubyKernel.java:1085:in `require'
    from /opt/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/systemd-journal-1.2.1/lib/systemd/journal.rb:12:in `(root)'
    from file:/opt/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1:in `(root)'
    from org/jruby/RubyKernel.java:1121:in `eval'
    from file:/opt/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:135:in `require'
    from (irb):1:in `evaluate'
    from org/jruby/RubyKernel.java:1521:in `loop'
    from org/jruby/RubyKernel.java:1284:in `catch'
    from org/jruby/RubyKernel.java:1284:in `catch'
    from org/jruby/RubyKernel.java:1101:in `load'
    from jirb:1:in `(root)'irb(main):002:0> 

Seems to be a Jruby issue of some sort, but I haven't a clue what.
No change with jruby-complete-1.7.16.2.jar

[stuart-warren]

v1.2.2 of ledbettj/systemd-journal should work with jruby now :)

[duncaninnes]

I'd raised something similar in JIRA back in Feb (https://logstash.jira.com/browse/LOGSTASH-1807). Is this similar enough to be covering the same ground?

The main point of the JIRA ticket was to be coming up with a plugin for systemd-journal-upload that would export to Logstash in JSON format in an acceptable way. Also to allow custom fields to be added/calculated at the client side rather than trying to get Indexer to figure stuff out.

The matching ticket from the systemd side is: https://www.libreoffice.org/bugzilla/show_bug.cgi?id=77013

Wondering if a new ticket should be created for any of the functionality that might have been missed?

[stuart-warren]

Purely as a proof of concept i've got an input plugin for journald
https://github.com/stuart-warren/logstash-input-journald

Still need to actually test creating/installing it as a gem. Works for me using plugin-path parameter in logstash 1.4.2.

[stuart-warren]

I've finally got round to testing my plugin above with v1.5.0beta and it looks like it works :)
I'm not really a ruby guy though, so if anyone wants to take the reins, feel free!

Ruby debug output:

{
                      "priority" => "6",
                          "_uid" => "0",
                          "_gid" => "0",
                "_systemd_slice" => "system.slice",
                   "_machine_id" => "ba310359bbce4e3da218f2e07e10dc60",
                     "_hostname" => "debian-testing-amd64",
                "_cap_effective" => "3fffffffff",
                    "_transport" => "syslog",
               "syslog_facility" => "4",
             "syslog_identifier" => "sshd",
                         "_comm" => "sshd",
                          "_exe" => "/usr/sbin/sshd",
               "_systemd_cgroup" => "/system.slice/ssh.service",
                 "_systemd_unit" => "ssh.service",
                      "_cmdline" => "sshd: vagrant [priv]",
                      "_boot_id" => "387b9a0a4f2e4c069296855fe8da76f1",
                    "syslog_pid" => "4808",
                       "message" => "Accepted publickey for vagrant from 10.0.2.2 port 39034 ssh2: RSA 5b:72:fe:de:53:e9:5d:db:04:1f:35:cb:ea:41:b1:ea",
                          "_pid" => "4808",
    "_source_realtime_timestamp" => "1424264611395778",
                    "@timestamp" => "2015-02-18T13:03:31.395Z",
                          "host" => "debian-testing-amd64",
                        "cursor" => "s=23fb883f20af442c96a24892b3415177;i=af9;b=387b9a0a4f2e4c069296855fe8da76f1;m=553f8431;t=50f5c723a7973;x=c0fa9517856f2911",
                      "@version" => "1",
                          "type" => "systemd",
                          "tags" => [
        [0] "coreos"
    ]
}
{
                      "priority" => "6",
                          "_uid" => "0",
                          "_gid" => "0",
                "_systemd_slice" => "system.slice",
                   "_machine_id" => "ba310359bbce4e3da218f2e07e10dc60",
                     "_hostname" => "debian-testing-amd64",
                "_cap_effective" => "3fffffffff",
                    "_transport" => "syslog",
             "syslog_identifier" => "sshd",
                         "_comm" => "sshd",
                          "_exe" => "/usr/sbin/sshd",
               "_systemd_cgroup" => "/system.slice/ssh.service",
                 "_systemd_unit" => "ssh.service",
                      "_cmdline" => "sshd: vagrant [priv]",
               "syslog_facility" => "10",
                       "message" => "pam_unix(sshd:session): session opened for user vagrant by (uid=0)",
               "_audit_loginuid" => "900",
                "_audit_session" => "4",
                      "_boot_id" => "387b9a0a4f2e4c069296855fe8da76f1",
                    "syslog_pid" => "4808",
                          "_pid" => "4808",
    "_source_realtime_timestamp" => "1424264611408196",
                    "@timestamp" => "2015-02-18T13:03:31.408Z",
                          "host" => "debian-testing-amd64",
                        "cursor" => "s=23fb883f20af442c96a24892b3415177;i=afa;b=387b9a0a4f2e4c069296855fe8da76f1;m=553fb628;t=50f5c723aab6b;x=c00d9c540ab69aa4",
                      "@version" => "1",
                          "type" => "systemd",
                          "tags" => [
        [0] "coreos"
    ]
}

[jmreicha]

Anybody tried this with RC2 yet?

[tylerjl]

+1 interest to seeing this through. I'm currently working on a project that will hinge on this eventually, so I'll pitch in some testing + patches when I get time.

[cv]

Excellent work, @stuart-warren! I'm trying it out without issues so far.

[dhawal55]

@stuart-warren Any plans of integrating it with logstash or create a docker-image?

[novemberborn]

@dhawal55 I'll be publishing such an (example) image later today or tomorrow.

[novemberborn]

@dhawal55 et al please see https://github.com/State/logstash-journald which uses @stuart-warren's plugin.

[dhawal55]

Thank you @novemberborn

[ycombinator]

+1 Requested by visitor to our booth at OSCON.

[elvarb]

+1 a really useful feature that will become more and more useful

[jalev]

+1 Since I want to phase out our grok+mangling rules for something much much cleaner

[dawnstrider]

+1 since some derivates now come with journald as stock.

[andrewvc]

Has anyone tried this with systemd-journal-remote plus the logstash HTTP input yet?

I agree that we should make a purpose-built module here, but I'm curious if anyone has a PoC going with this yet.

[tdooner]

@andrewvc We've been using https://github.com/vaijab/logstash-input-journald in production for some time successfully. The only problem we have is this issue: ledbettj/systemd-journal#56 (comment) which prevents a fast graceful restart of logstash (though I don't believe log messages are dropped).

[andrewvc]

@tdooner awesome! Good info to bring back to the LS core team.

[L1ghtn1ng]

@andrewvc What is the status of this as this really should be implemented by now. As arch,ubuntu,centOS, fedora, debain all use journal. Yes ubuntu has syslog for backwards compatibility but others do not

[suyograo]

@stuart-warren Created https://github.com/logstash-plugins/logstash-input-journald. Can you please add some tests and create an initial PR here from your repo? Thanks!

[L1ghtn1ng]

@suyograo I have seen that but its not clear how to go about the config for it as this is my first time setting up an ELK stack. I have used an ELK stack before but never the one that setup till now.

[stuart-warren]

HI @suyograo, @vaijab is the new maintainer, primarily because ruby is not my strong suit and I've never written a ruby test.. 😞

[vaijab]

@suyograo I am happy to pass the repo onto logstash-plugins so that there is more visibility and people can contribute to it.

We've been using it in production for quite some time and it works really well.

[talset]

@vaijab Is it plan to merge the journald plugin inside logstash ?

[vaijab]

@talset probably not into logstash core, but I'd be happy to contribute this to logstash-plugins repo.

[suyograo]

@vaijab we would love that: Can you create a PR here: https://github.com/logstash-plugins/logstash-input-journald

[vaijab]

@suyograo would it not be easier if I reassigned the repo to logstash-plugins org? That way the original author and commit history and everything else would be preserved.

[suyograo]

@vaijab sure can you do this: https://help.github.com/articles/transferring-a-repository/#transferring-between-organizations

[vaijab]

@suyograo I get this logstash-plugins/logstash-input-journald already exists and You don’t have admin rights to logstash-plugins. I believe you have to delete the existing repo before I can transfer mine.

[suyograo]

@vaijab sent you an invite. Can you try now?

[vaijab]

@suyograo great, it's done now. Thanks.

[suyograo]

Thanks for the transfer @vaijab. You are an administrator for that repo. Let me know if you have any issues.

[vaijab]

@suyograo awesome. I will also transfer https://github.com/vaijab/logstash-filter-kubernetes so that more people have visibility, etc.

[vaijab]

By the way, I believe this issue can now be closed?

[suyograo]

@vaijab one other thing, can you add dev_ops@elastic.co to RubyGems owner? https://rubygems.org/gems/logstash-input-journald

[vaijab]

@suyograo done.