Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logstash 6.7 package from apt fails on Debian Buster #10593

Closed
rpetre opened this issue Mar 27, 2019 · 16 comments

Comments

Projects
None yet
@rpetre
Copy link

commented Mar 27, 2019

  • Version: 1:6.7.0-1
  • Operating System: Debian Buster
  • Steps to Reproduce: install the logstash-1:6.7.0 package from artifacts.elastic.co , the post-install script fails:
Setting up logstash (1:6.7.0-1) ...
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by jnr.posix.JavaLibCHelper$ReflectiveAccess to method sun.nio.ch.SelChImpl.getFD()
WARNING: Please consider reporting this to the maintainers of jnr.posix.JavaLibCHelper$ReflectiveAccess
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Errno::EBADF: Bad file descriptor - systemctl
            spawn at org/jruby/RubyProcess.java:1567
            spawn at org/jruby/RubyKernel.java:1632
        popen_run at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/open3.rb:206
           popen3 at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/open3.rb:102
          execute at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/detector.rb:74
   detect_systemd at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/detector.rb:29
  detect_platform at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/detector.rb:24
           detect at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/detector.rb:18
   setup_defaults at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/cli.rb:153
          execute at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/cli.rb:119
              run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67
              run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/cli.rb:114
              run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132
           <main> at /usr/share/logstash/lib/systeminstall/pleasewrap.rb:12
chmod: cannot access '/etc/default/logstash': No such file or directory
dpkg: error processing package logstash (--configure):
 installed logstash package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 logstash

I'm not yet 100% sure it's not something related to my global Java settings, if you can suggest additional debugging flags to pass to bin/system-install , I'd appreciate it.

@jsvd

This comment has been minimized.

Copy link
Member

commented Mar 27, 2019

hi @rpetre, can you confirm you have Java 11 in Debian Buster? any change you could check with Java 8?

@rpetre

This comment has been minimized.

Copy link
Author

commented Mar 27, 2019

Yes, if I export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 and I run /usr/share/logstash/bin/system-install /etc/logstash/startup.options it works (I also tested by removing the openjdk-11 packages and reinstalling the package via apt). If I try the same with an empty JAVA_HOME it switches to java 11 and I get the bad file descriptor error again. I can reliably switch between the two behaviours with this mechanism.

@lliknart

This comment has been minimized.

Copy link

commented Apr 1, 2019

Hi,
I have the exact same issue with Ubuntu 18.04

@THuizinga

This comment has been minimized.

Copy link

commented Apr 1, 2019

@lliknart I had the same problem. For me it was fixed after I uninstalled all versions of Java and installed Java 8 using the Oracle JDK (https://thishosting.rocks/install-java-ubuntu/). Before that I had OpenJDK and then I got the same error.

@pukhanov

This comment has been minimized.

Copy link

commented Apr 1, 2019

Looks like a similar issue appears when installing the RPM with Java 11. With Java 8, everything works.

Running transaction
  Installing : 1:logstash-6.7.0-1.noarch                                                                                                                                                            1/1
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.util.SecurityHelper to field java.lang.reflect.Field.modifiers
WARNING: Please consider reporting this to the maintainers of org.jruby.util.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
NoMethodError: undefined method `split' for nil:NilClass
   detect_systemd at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/detector.rb:33
  detect_platform at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/detector.rb:24
           detect at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/detector.rb:18
   setup_defaults at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/cli.rb:153
          execute at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/cli.rb:119
              run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67
              run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/cli.rb:114
              run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132
           <main> at /usr/share/logstash/lib/systeminstall/pleasewrap.rb:12
/usr/share/logstash/bin/system-install: line 88: systemd: command not found
Unable to install system startup script for Logstash.
chmod: cannot access '/etc/default/logstash': No such file or directory
warning: %post(logstash-1:6.7.0-1.noarch) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package 1:logstash-6.7.0-1.noarch
  Verifying  : 1:logstash-6.7.0-1.noarch                                                                                                                                                            1/1

Installed:
  logstash.noarch 1:6.7.0-1
@jsvd

This comment has been minimized.

Copy link
Member

commented Apr 2, 2019

of course, weirdly enough it doesn't seem to replicate with docker:

FROM debian:buster

RUN apt-get update && \
    apt-get install -y zlib1g-dev build-essential vim rake git curl libssl-dev libreadline-dev libyaml-dev  \
      libxml2-dev libxslt-dev openjdk-11-jdk-headless curl iputils-ping netcat && \
    apt-get clean

WORKDIR /root

RUN apt-get install wget sudo

RUN wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -

RUN apt-get install apt-transport-https

RUN echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list

RUN apt-get update && apt-get install logstash

RUN echo "input { tcp { port => 3333 } } output { stdout { codec => rubydebug }" >> /etc/logstash/conf.d/test.conf
% docker build --tag=test .
...
Preparing to unpack .../logstash_1%3a6.7.0-1_all.deb ...
Unpacking logstash (1:6.7.0-1) ...
Setting up logstash (1:6.7.0-1) ...
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.util.SecurityHelper to field java.lang.reflect.Field.modifiers
WARNING: Please consider reporting this to the maintainers of org.jruby.util.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash
...
Successfully tagged test:latest
% docker run -it test bash
root@010e0248cc07:~# service logstash start
logstash started
root@010e0248cc07:~# nc localhost 3333 
ewrewrwer
^C       
root@010e0248cc07:~# cat /tmp/data.txt 
{"@version":"1","port":53398,"message":"ewrewrwer","@timestamp":"2019-04-02T09:25:16.826Z","host":"localhost"}
root@010e0248cc07:~# curl localhost:9600?pretty
{
  "host" : "010e0248cc07",
  "version" : "6.7.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "94bd6498-41e0-44a1-b128-c856fe76770d",
  "name" : "010e0248cc07",
  "build_date" : "2019-03-21T16:26:15+00:00",
  "build_sha" : "5d01f605d9cc38b278aba0688c2545eaf3d9b843",
  "build_snapshot" : false
}
%
@jsvd

This comment has been minimized.

Copy link
Member

commented Apr 2, 2019

I'm now able to reproduce the bad file descriptor issue, it seems to be a JRuby issue with Open3#popen jruby/jruby#5642 (comment)

@chaami

This comment has been minimized.

Copy link

commented Apr 3, 2019

Hi,
For information, in my case, simply setting the JAVA_HOME to a freshly installed openjdk-8 didn't make the post-install script to work.
Setting up the alternative jdk to point the openjdk-8 didn't make it any better.
I had to completely remove the openjdk-11 for things to work as planned (didn't have to go for the Oracle proprietary version).
If you had created the /etc/default/logstash directory in a attempt to help the installation script to complete with the jdk-11, you'll need to remove it before trying again.
(Creating it makes the script go further, but it still fails, so don't bother none ;-) )
Hope this helps.

@jsvd jsvd closed this in #10658 Apr 5, 2019

jsvd added a commit that referenced this issue Apr 5, 2019

jsvd added a commit that referenced this issue Apr 5, 2019

jsvd added a commit that referenced this issue Apr 5, 2019

jsvd added a commit that referenced this issue Apr 5, 2019

@Dr-Steve

This comment has been minimized.

Copy link

commented Apr 10, 2019

This has been very frustrating. It took me a long time to find this page, but fortunately downgrading to java 8 as suggested above seems to have worked at long last.

@orykami

This comment has been minimized.

Copy link

commented Apr 10, 2019

Hi there !
Same problem here with JDK 11.
@jsvd Finally, is it possible to deal with this issue with JDK 11 & ES 6.7 ?

@jsvd

This comment has been minimized.

Copy link
Member

commented Apr 10, 2019

jruby 9.2.7.0 has been shipped yesterday and fixes this issue. We have a PR already up so that next releases of logstash will contain this version: #10674

For now the only workaround is using jdk8 during package installation and then flipping back to java 11

@orykami

This comment has been minimized.

Copy link

commented Apr 10, 2019

@jsvd Awesome thank's :)

@ynya

This comment has been minimized.

Copy link

commented Apr 25, 2019

Debian 9,
With a fresh installation of ELK, I installed the openjdk-11-jdk package for logstash. But he did not start with the error
Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (NameError) cannot load (ext) (org.jruby.ext.openssl.OpenSSL)

Solution : Install openjdk-11-jdk in unstable depot

Stable version
openjdk 11.0.3 2019-04-16 OpenJDK Runtime Environment (build 11.0.3+1-Debian-1bpo91) OpenJDK 64-Bit Server VM (build 11.0.3+1-Debian-1bpo91, mixed mode, sharing)
Unstable version
openjdk 11.0.3 2019-04-16 OpenJDK Runtime Environment (build 11.0.3+7-Debian-2) OpenJDK 64-Bit Server VM (build 11.0.3+7-Debian-2, mixed mode, sharing)

@kedare

This comment has been minimized.

Copy link

commented May 6, 2019

I'm getting the same issue with JDK 8

# java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

# ./bin/logstash-plugin update logstash-codec-netflow             
NameError: cannot load (ext) (org.jruby.ext.openssl.OpenSSL)                                   
  load_ext at org/jruby/ext/jruby/JRubyUtilLibrary.java:201                                    
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/jopenssl/load.rb:31             
   require at org/jruby/RubyKernel.java:987
    (root) at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
   require at org/jruby/RubyKernel.java:987
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/openssl.rb:1                    
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
   require at org/jruby/RubyKernel.java:987
   require at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/security.rb:12         
   require at org/jruby/RubyKernel.java:987
    (root) at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
   require at org/jruby/RubyKernel.java:987
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/package.rb:44          
    <main> at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1

On Debian 9

@ianwijaya

This comment has been minimized.

Copy link

commented May 11, 2019

+1

@antondollmaier

This comment has been minimized.

Copy link

commented Jul 4, 2019

@ianwijaya @kedare

I was stuck with the same issue you report.

After deleting the CA store, logstash is now working.

Source

I ran sudo rm /etc/ssl/certs/java/cacerts and then sudo update-ca-certificates -f and this fixed my issue in kubuntu 18.04.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.