Feature/build fixes #3

tintoy · 2018-10-02T00:13:32Z

Update plugin to work against latest version(s) of Logstash.

Relates to #1 and #2.

Replaces #2.

CC: @mjnorman

@mjnorman

This incorporates changes from #2 (thanks, @mjnorman!).

#1

tintoy · 2018-10-02T00:17:17Z

@mjnorman - can you pull down the latest code from this branch (have rebased and squashed a couple of commits, so watch out for that) and see if it works for you?

coveralls · 2018-10-02T00:23:55Z

Coverage remained the same at 96.552% when pulling 24708ab on feature/build-fixes into c40ffb7 on master.

#1

mjnorman · 2018-10-02T01:51:15Z

This is the problem I was having this weekend. When I do a bundle exec rpsec it finds none of the tests in the project 0 examples found. When I do a bundle exec rake it fails with error no such file to load -- org/apache/logging/log4j/log4j-core/compile/log4j-core-compile-2.6.2 (LoadError).

However I am able to build the gem successfully. And it also installs and runs and actually exports data to seq. What is wrong with the build that it won't find the tests and run them?

tintoy · 2018-10-02T01:56:40Z

What version of the JRE / JDK do you have? I'm using JDK8 and JRuby v9.1.17.0 if that helps.

I spend 25 mins on a similar problem until I ran Get-Command bundle, and realised that regular Ruby was first in my $PATH, but I'm not sure whether that's the same problem you were having :)

On the off chance that it is, you'll need to ensure that JRuby is first in your $PATH and run:

Get-Command gem (check that it's the JRuby version)
gem install bundler
bundle exec rake vendor
bundle exec rake rspec spec

mjnorman · 2018-10-02T02:26:50Z

I'm sure it has to be something stupid I'm doing, but here is the output. I reran bundle install here also.

PS C:\git\repos\ruby\logstash-output-seq> Get-Command gem

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     gem.bat                                            0.0.0.0    c:\jruby-9.2.0.0\bin\gem.bat


PS C:\git\repos\ruby\logstash-output-seq> gem install bundler
Successfully installed bundler-1.16.5
1 gem installed
PS C:\git\repos\ruby\logstash-output-seq> bundle exec rake vendor
PS C:\git\repos\ruby\logstash-output-seq> Get-Command bundle

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     bundle.bat                                         0.0.0.0    c:\jruby-9.2.0.0\bin\bundle.bat


PS C:\git\repos\ruby\logstash-output-seq> bundle install
<bundle output spam, everything up to date>
Bundle complete! 9 Gemfile dependencies, 65 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
PS C:\git\repos\ruby\logstash-output-seq> bundle exec rake rspec spec
rake aborted!
Don't know how to build task 'rspec' (see --tasks)
c:\jruby-9.2.0.0\bin\rake:23:in `<main>'
(See full trace by running task with --trace)
PS C:\git\repos\ruby\logstash-output-seq>

Also ran bundle exec rake as before, and got the same error output about log4j above.
Double checked my path, and I do have jruby first in path.

PS C:\git\repos\ruby\logstash-output-seq> bundle exec rake spec
C:/jruby-9.2.0.0/bin/jruby.exe -I'C:/jruby-9.2.0.0/lib/ruby/gems/shared/gems/rspec-core-3.8.0/lib';'C:/jruby-9.2.0.0/lib/ruby/gems/shared/gems/rspec-support
-3.8.0/lib' 'C:/jruby-9.2.0.0/lib/ruby/gems/shared/gems/rspec-core-3.8.0/exe/rspec' --pattern 'spec/**{,/*/**}/*_spec.rb' --color --format documentation

An error occurred while loading ./spec/outputs/seq_spec.rb.
Failure/Error: # encoding: utf-8

RuntimeError:


        you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

  no such file to load -- org/apache/logging/log4j/log4j-core/compile/log4j-core-compile-2.6.2 (LoadError)
# ./spec/outputs/seq_spec.rb:1:in `(root)'
# ./spec/outputs/seq_spec.rb:2:in `(root)'
# ------------------
# --- Caused by: ---
# LoadError:
#   no such file to load -- org/apache/logging/log4j/log4j-core/compile/log4j-core-compile-2.6.2
#   ./spec/outputs/seq_spec.rb:1:in `(root)'
No examples found.

Finished in 0.013 seconds (files took 0.999 seconds to load)
0 examples, 0 failures, 1 error occurred outside of examples

Any other ideas?

tintoy · 2018-10-02T04:25:13Z

It's just bundle exec rspec spec, not bundle exec rake rspec spec :)

mjnorman · 2018-10-02T04:44:24Z

PS C:\git\repos\ruby\logstash-output-seq> bundle exec rspec spec

An error occurred while loading ./spec/outputs/seq_spec.rb.
Failure/Error: # encoding: utf-8

LoadError:
  load error: concurrent/utility/monotonic_time -- java.lang.NoSuchFieldError: SUPPORTS_FENCES
# ./spec/outputs/seq_spec.rb:1:in `<main>'
# c:\jruby-9.2.0.0\bin\rspec:23:in `<main>'
No examples found.


Finished in 0.011 seconds (files took 2.7 seconds to load)
0 examples, 0 failures, 1 error occurred outside of examples

:(

tintoy · 2018-10-02T11:46:38Z

Weird - maybe my editor added UTF8 preamble bytes to the file or something? Wonder why it works on my system though... Do you have a hex editor? :)

mjnorman · 2018-10-02T14:27:23Z

I can get one. How about this. Can you clone into a new directory, pull down your branch, and see if you get the same thing?

Can you paste all that output here and let me compare?

tintoy · 2018-10-03T03:16:35Z

Sure - will do that first thing tomorrow.

mjnorman · 2018-10-03T17:59:10Z

So this has to do with something in logstash's configurations. I was able to create a new standard ruby gem project--

bundle gem newgem
bundle install/update

Then I was able to run the specs normally--

PS C:\git\repos\ruby\newgem> bundle exec rspec spec

Newgem
  has a version number
  does something useful (FAILED - 1)

Failures:

  1) Newgem does something useful
     Failure/Error: expect(false).to eq(true)

       expected: true
            got: false

       (compared using ==)
     # ./spec/newgem_spec.rb:7:in `block in (root)'
     # c:\jruby-9.2.0.0\bin\rspec:23:in `<main>'

Finished in 0.203 seconds (files took 1.43 seconds to load)
2 examples, 1 failure

tintoy · 2018-10-03T21:29:21Z

I'm about to try fetching the cloning and building the repository from scratch. Will let you know how I go :)

tintoy · 2018-10-03T21:34:34Z

Yep, I get that error too when I check out from scratch! Let me see if I can figure out why...

#3

tintoy · 2018-10-03T21:47:45Z

Ok, this worked for me now:

logstash-output-seq
λ  git clone https://github.com/tintoy/logstash-output-seq.git . ; git checkout feature/build-fixes
Cloning into '.'...
remote: Enumerating objects: 39, done.
remote: Counting objects: 100% (39/39), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 201 (delta 11), reused 31 (delta 7), pack-reused 162
Receiving objects: 100% (201/201), 30.81 KiB | 595.00 KiB/s, done.
Resolving deltas: 100% (64/64), done.
Switched to a new branch 'feature/build-fixes'
Branch 'feature/build-fixes' set up to track remote branch 'feature/build-fixes' from 'origin'.
logstash-output-seq [feature/build-fixes ≡]
λ  bundle install ; bundle exec rake vendor ; bundle exec rspec spec
Fetching gem metadata from https://rubygems.org/............
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies.................................................................................................................
Using rake 11.3.0
Using bundler 1.16.5
Using numerizer 0.1.1
Using chronic_duration 0.10.6
Using clamp 0.6.5
Using coderay 1.1.2
Using concurrent-ruby 1.0.5 (java)
Using json 2.1.0 (java)
Using docile 1.3.1
Using simplecov-html 0.10.2
Using simplecov 0.16.1
Using tins 1.16.3
Using term-ansicolor 1.6.0
Using thor 0.19.4
Using coveralls 0.8.22
Using diff-lcs 1.3
Using multi_json 1.13.1
Using elasticsearch-api 5.0.5
Using multipart-post 2.0.0
Using faraday 0.15.3
Using elasticsearch-transport 5.0.5
Using elasticsearch 5.0.5
Using ffi 1.9.25 (java)
Using filesize 0.0.4
Using fivemat 1.3.7
Using gem_publisher 1.5.0
Using gems 0.8.3
Using i18n 0.6.9
Using insist 1.0.0
Using jar-dependencies 0.4.0
Using jrjackson 0.4.6 (java)
Using jruby-openssl 0.9.19 (java)
Using kramdown 1.14.0
Using openssl_pkcs8_pure 0.0.0.2
Using manticore 0.6.4 (java)
Using minitar 0.5.4
Using method_source 0.8.2
Using slop 3.6.0
Using spoon 0.0.6
Using pry 0.10.4 (java)
Using puma 2.16.0 (java)
Using rack 1.6.6
Using ruby-maven-libs 3.3.9
Using ruby-maven 3.3.12
Using rubyzip 1.1.7
Using rack-protection 1.5.5
Using tilt 2.0.8
Using sinatra 1.4.8
Using stud 0.0.23
Using thread_safe 0.3.6 (java)
Using polyglot 0.3.5
Using treetop 1.4.15
Using logstash-core 5.6.4 (java)
Using logstash-core-plugin-api 2.1.28 (java)
Using logstash-codec-plain 3.0.6
Using rspec-support 3.8.0
Using rspec-core 3.8.0
Using rspec-expectations 3.8.1
Using rspec-mocks 3.8.0
Using rspec 3.8.0
Using rspec-wait 0.0.9
Using logstash-devutils 1.3.6 (java)
Using logstash-mixin-http_client 6.0.1
Using logstash-output-seq 0.1.0 from source at `.`
Using webrick 1.4.2
Bundle complete! 9 Gemfile dependencies, 65 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
--- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.9.1 - omit version 2.9.5
--- jar coordinate com.fasterxml.jackson.core:jackson-core already loaded with version 2.9.1 - omit version 2.9.5
--- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.9.1 - omit version 2.9.5
--- jar coordinate com.fasterxml.jackson.module:jackson-module-afterburner already loaded with version 2.9.1 - omit version 2.9.5
Sending Logstash's logs to  which is now configured via log4j2.properties
Coverage may be inaccurate; set the "--debug" command line option, or do JRUBY_OPTS="--debug" or set the "debug.fullTrace=true" option in your .jrubyrc
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's default settings.
D:/Development/Tools/JRuby/v9.1.17.0/lib/ruby/gems/shared/gems/simplecov-0.16.1/lib/simplecov.rb:53: warning: tracing (e.g. set_trace_func) will not capture all events without --debug flag
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :couchdb=>true, :elasticsearch=>true, :elasticsearch_secure=>true, :export_cypher=>true, :integration=>true, :unix=>true}

Randomized with seed 893
D:/Development/Tools/JRuby/v9.1.17.0/lib/ruby/gems/shared/gems/webrick-1.4.2/lib/webrick/utils.rb:128: warning: toplevel constant Mutex referenced by Thread::Mutex
D:/Development/Tools/JRuby/v9.1.17.0/lib/ruby/gems/shared/gems/webrick-1.4.2/lib/webrick/server.rb:94: warning: toplevel constant SizedQueue referenced by Thread::SizedQueue
== Sinatra (v1.4.8) has taken the stage on 51089 for development with backup from WEBrick
D:/Development/Tools/JRuby/v9.1.17.0/lib/ruby/gems/shared/gems/webrick-1.4.2/lib/webrick/utils.rb:157: warning: toplevel constant Queue referenced by Thread::Queue
........

Finished in 1.52 seconds (files took 5.07 seconds to load)
8 examples, 0 failures

Randomized with seed 893

== Sinatra has ended his set (crowd applauds)
[Coveralls] Outside the CI environment, not sending data.

Can you do a pull and give it a try?

mjnorman · 2018-10-04T15:05:50Z

UGH--

PS C:\git\repos\ruby\logstash-output-seq> bundle install ; bundle exec rake vendor ; bundle exec rspec spec
Fetching gem metadata from https://rubygems.org/.........
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies.................................................................................................................
<...>
Bundle complete! 9 Gemfile dependencies, 65 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

An error occurred while loading ./spec/outputs/seq_spec.rb.
Failure/Error: # encoding: utf-8

RuntimeError:


        you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

  no such file to load -- org/apache/logging/log4j/log4j-core/compile/log4j-core-compile-2.6.2 (LoadError)
# ./spec/outputs/seq_spec.rb:1:in `(root)'
# ./spec/outputs/seq_spec.rb:3:in `(root)'
# c:\jruby-9.2.0.0\bin\rspec:23:in `<main>'
# ------------------
# --- Caused by: ---
# LoadError:
#   no such file to load -- org/apache/logging/log4j/log4j-core/compile/log4j-core-compile-2.6.2
#   ./spec/outputs/seq_spec.rb:1:in `(root)'
No examples found.


Finished in 0.01 seconds (files took 1.2 seconds to load)
0 examples, 0 failures, 1 error occurred outside of examples

tintoy · 2018-10-04T22:04:29Z

Bizarre!

Ok, perhaps stronger measures are called for:

Can you unzip spec.zip in the root of your repository and see if the unzipped files work? I'm thinking this might be a git configuration issue (or something along those lines)...

mjnorman · 2018-10-04T22:23:12Z

Yes I’ll do that. What is your dev environment?

tintoy · 2018-10-04T22:31:40Z

What is your dev environment?

Windows 10
PowerShell (cmder)
Java v1.8.0u151
jRuby v9.1.17.0

Output of git config --list:

core.symlinks=true
core.autocrlf=true
core.fscache=true
color.diff=auto
color.status=auto
color.branch=auto
color.interactive=true
help.format=html
rebase.autosquash=true
http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
http.sslbackend=schannel
diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
credential.helper=manager
core.longpaths=true
user.email=tintoy@tintoy.io
user.name=Adam Friedman
user.signingkey=C6AFE8C008E7F2B3
credential.https://source.developers.google.com.usehttppath=true
log.follow=true
gpg.program=C:\Program Files (x86)\GnuPG\bin\gpg.exe
core.repositoryformatversion=0
core.filemode=false
core.bare=false
core.logallrefupdates=true
core.ignorecase=true

mjnorman · 2018-10-04T23:31:28Z

I’m not in the office, will do this tonight. Wanted to get this down while I’m thinking about it though. I can create a new gem bundle gem newgem and it runs rspec just fine. However..uncomment this one line and it produces this same error. This is why I think it is more related to logstash.

https://github.com/mjnorman/newgem/blob/4e6aba363b750ac44df71bc99b9f8f65eb6c4dd4/spec/newgem_spec.rb#L2

mjnorman · 2018-10-05T01:32:48Z

Extracted that to the directory, same error. Even started with a fresh clone, git checkout feature branch, then extract to overwrite spec files. Same outcome.
no such file to load -- org/apache/logging/log4j/log4j-core/compile/log4j-core-compile-2.6.2 (LoadError)

tintoy · 2018-10-05T02:29:57Z

Interesting - are you using Logstash from source by any chance? I'm not; I'm just installing the gems...

tintoy · 2018-10-05T02:30:16Z

(I don't actually have Logstash installed locally)

mjnorman · 2018-10-05T02:58:04Z

I just downloaded the zip from logstash website and extracted it.

mjnorman · 2018-10-08T12:49:16Z

Ok so I have figured out it is something with my machine. I took a Windows 2012 server, installed jruby and git, cloned, and ran the rspec and it worked fine. I don't know what's up with my machine, but you can probably complete this pull request if you don't have any other changes.

tintoy · 2018-10-08T14:08:16Z

Ok, will get it merged and published :)

mjnorman · 2018-10-08T14:43:41Z

I finally managed to fix my build, bundle install --force, not sure why but this fixed it.

Thanks!

mjnorman · 2018-10-08T16:37:15Z

Believe we may have some issues here now that I'm actually able to use the plugin

[2018-10-08T12:35:06,809][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>31
, "name"=>"[mom-routing]<tcp", "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-input-tcp-5.1.0-java/lib/logstash/inputs/tcp.rb:183:in `
close'"}, {"thread_id"=>32, "name"=>"[mom-routing]<udp", "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/i
nputs/udp.rb:121:in `select'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["syslog_hostname", "syslog_message", "syslog_timestamp"], "add_field"
=>{"@level"=>"Informational"}, "id"=>"dd9b65adccf5b2b8d93860becd38979f75a5e0643b65d9f465e3b600f64f51fc"}]=>[{"thread_id"=>26, "name"=>nil, "current_ca
ll"=>"[...]/logstash-core/lib/logstash/pipeline.rb:372:in `multi_receive'"}, {"thread_id"=>27, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/l
ogstash/pipeline.rb:372:in `multi_receive'"}, {"thread_id"=>28, "name"=>nil, "current_call"=>"[...]/vendor/local_gems/13b86593/logstash-output-seq-0.2
.0/lib/logstash/outputs/seq.rb:70:in `pop'"}, {"thread_id"=>29, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:372:in `mul
ti_receive'"}]}}
[2018-10-08T12:35:06,871][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins.
 Check the logs for more information.
[2018-10-08T12:35:11,903][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>31
, "name"=>"[mom-routing]<tcp", "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-input-tcp-5.1.0-java/lib/logstash/inputs/tcp.rb:183:in `
close'"}, {"thread_id"=>32, "name"=>"[mom-routing]<udp", "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/i
nputs/udp.rb:121:in `select'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["syslog_hostname", "syslog_message", "syslog_timestamp"], "add_field"
=>{"@level"=>"Informational"}, "id"=>"dd9b65adccf5b2b8d93860becd38979f75a5e0643b65d9f465e3b600f64f51fc"}]=>[{"thread_id"=>26, "name"=>nil, "current_ca
ll"=>"[...]/logstash-core/lib/logstash/pipeline.rb:372:in `multi_receive'"}, {"thread_id"=>27, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/l
ogstash/pipeline.rb:372:in `multi_receive'"}, {"thread_id"=>28, "name"=>nil, "current_call"=>"[...]/vendor/local_gems/13b86593/logstash-output-seq-0.2
.0/lib/logstash/outputs/seq.rb:70:in `pop'"}, {"thread_id"=>29, "name"=>nil, "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:372:in `mul
ti_receive'"}]}}

tintoy · 2018-10-08T21:12:55Z

Ugh, that looks ugly. What version of logstash are you using? I'm going to try to set up a repro environment this morning.

mjnorman · 2018-10-08T21:15:44Z

So far I have not been able to reproduce it again, but it happened with 6.4.1. Right now I have had logstash streaming to seq using this branch's updates for a couple of hours on 6.4.1 and have not seen it since. Would like to see if you experience any issues with it.

tintoy · 2018-10-08T21:30:06Z

Ok, cheers! I'm wondering if there are any acceptance-test / integration-test suites as part of Logstash than can be used to test plugins for compliance and correctness of behaviour...?

Will see what I can come up with :)

tintoy added 4 commits September 29, 2018 16:23

Experimental fixes for build process (currently borked).

d38a4c3

This incorporates changes from #2 (thanks, @mjnorman!).

Use latest jruby.

05bcc8a

Fix up jRuby version for Travis.

071aeb3

Build against latest version of Logstash.

3c04d0f

#1

tintoy added the enhancement label Oct 2, 2018

tintoy self-assigned this Oct 2, 2018

Improve formatting of output from CI scripts.

34ffd35

#1

Force update of spec files (possibly a UTF8 preamble issue).

7cc4227

#3

bump version to 0.2.0

24708ab

tintoy merged commit 166d3dc into master Oct 8, 2018

tintoy deleted the feature/build-fixes branch October 8, 2018 22:40

Feature/build fixes #3

Feature/build fixes #3

Uh oh!

Conversation

tintoy commented Oct 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tintoy commented Oct 2, 2018

Uh oh!

coveralls commented Oct 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjnorman commented Oct 2, 2018

Uh oh!

tintoy commented Oct 2, 2018

Uh oh!

mjnorman commented Oct 2, 2018

Uh oh!

tintoy commented Oct 2, 2018

Uh oh!

mjnorman commented Oct 2, 2018

Uh oh!

tintoy commented Oct 2, 2018

Uh oh!

mjnorman commented Oct 2, 2018

Uh oh!

tintoy commented Oct 3, 2018

Uh oh!

mjnorman commented Oct 3, 2018

Uh oh!

tintoy commented Oct 3, 2018

Uh oh!

tintoy commented Oct 3, 2018

Uh oh!

tintoy commented Oct 3, 2018

Uh oh!

mjnorman commented Oct 4, 2018

Uh oh!

tintoy commented Oct 4, 2018

Uh oh!

mjnorman commented Oct 4, 2018

Uh oh!

tintoy commented Oct 4, 2018

Uh oh!

mjnorman commented Oct 4, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjnorman commented Oct 5, 2018

Uh oh!

tintoy commented Oct 5, 2018

Uh oh!

tintoy commented Oct 5, 2018

Uh oh!

mjnorman commented Oct 5, 2018

Uh oh!

mjnorman commented Oct 8, 2018

Uh oh!

tintoy commented Oct 8, 2018

Uh oh!

mjnorman commented Oct 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjnorman commented Oct 8, 2018

Uh oh!

tintoy commented Oct 8, 2018

Uh oh!

mjnorman commented Oct 8, 2018

Uh oh!

tintoy commented Oct 8, 2018

Uh oh!

Uh oh!

tintoy commented Oct 2, 2018 •

edited

Loading

coveralls commented Oct 2, 2018 •

edited

Loading

mjnorman commented Oct 4, 2018 •

edited

Loading

mjnorman commented Oct 8, 2018 •

edited

Loading