payara not closing file handle on red hat linux

[engleth]

Description

---

When running Payara 4.1.2.173 or newer on a Red Hat Enterprise Linux 7.4 Server with kernel version

3.10.0-693.21.1.el7.x86_64 #1 SMP Fri Feb 23 18:54:16 UTC 2018

file handles are not closed correctly leading to breakdown of the server instance.

This is not the case for a suse linux or centos linux distribution.

Expected Outcome

Single domain of a payara server should only create about 7000 - 10000 file handles maximum at once when running (depending on deployed applications).

Current Outcome

A total of about 500000 file handles are open (lsof | wc -l), while having only a total of 3300 files open ( cat /proc/sys/fs/file-nr).

Thereof are 490000 for jar files connected to the payara java process (domain libs, server libs and osgi felix cache).

Steps to reproduce (Only for bug reports)

1 -** Start the domain

2 -** Check open file handles

Context (Optional)

This prevents running more than a single domain in parallel.

Environment

• Payara Version: 4.1.2.173
• Edition: Full
• JDK Version: 8
• Operating System: Red Hat Enterprise Linux 7.4
• Kernel: 3.10.0-693.21.1.el7.x86_64

[MattGill98]

Hi @engleth,

Are you able to reproduce this using any JDK on RedHat Linux? For example, Zulu JDK 172. I'd also encourage you to try with the Payara Server 5.183, as Payara 4.x is no longer in the community stream.

Kind regards,

Matt

[engleth]

Hello @MattGill98,

Thank you for the response.

We already tried running it with Payara 5.183 and 4.1.2.172, Red Hat Linux 7.5 (different server) and Open JDK 1.8 in each possible configuration and were able to reproduce the issue even without hosting any application on the Payara (in this case approximately 120000 file handles are open).

We also checked our other production system for this issue with the following environment:

Payara Version: 4.1.2.172
Edition: Full
JDK Version: 8
Operating System: SUSE Linux Enterprise Server 11
Kernel: 3.0.101-0.47.71-default

Here the issue is not present.

It seems like the only difference that could lead to the issue is with a different OS and Kernel version which seems to produce the problem.

Regards,

engleth

[svendiedrichsen]

@engleth Which exact JDK Version do you use? 'JDK 8' is somewhat generic.

[engleth]

Hello, Sorry for not being more specific. We used several different JDK Versions, including the most recent Releases 191 and 192 of Oracle JDK 8 and 192 of Open JDK 8. Regards, engleth

[svendiedrichsen]

JDK 8 191 as been built using gcc 7.3 (coming from gcc 4.3). Could you try 181 if you haven't already? Just to make sure.

https://www.oracle.com/technetwork/java/javase/8u191-relnotes-5032181.html

[engleth]

Tried build 181 this morning and got the same issue for build 181, 191 and 192 of Oracle JDK and even tried with your recommendation Zulu 172 and got the issue.

[svendiedrichsen]

@engleth Are there any specialties concerning mounted filesystems? Are there differences in filesystem types?

[engleth]

No specialties or differences between file systems as far as we know (all servers were ordered as mirrors). Any specific information that could help in this case?

[svendiedrichsen]

No specialties or differences between file systems as far as we know (all servers were ordered as mirrors). Any specific information that could help in this case?

No, I was just wondering if the Payara Server uses any remotely mounted filesystems to access its files. Or any rarely used ones. In the end this are just some wild guesses.

[smillidge]

I believe this is a bug (feature) of the way file handles are being reported in the RHEL version reported.