Our system encounter GRIZZLY0013

[glassfishrobot]

Hello I'm Toshikazu.
Please teach me. About GRIZZLY0013.

Overview

In my client & server of commercially operating of our system has came up with the message of [WARNING][Error] too much in Glassfish's server.log as problem.
In that case, Many sessions from client will be disconnected to Glassfish server.
Since if it did reconnection but couldn't be Push notification from Glassfish(process). And That became [Heart-beat]'s Error after client will be disconnected to Glassfish server.
About [Heart-beat]'s Error, I will describe after as [The overview of Application]

The follow message:
■A part of server.log

[YYYY-MM-DDT18:51:02.575+0900] [glassfish 4.1] [WARNING] [] [org.glassfish.grizzly.filterchain.DefaultFilterChain] [tid: _ThreadID=132 _ThreadName=BidRateTimer] [timeMillis: 1458553862575] [levelValue: 900] [[
GRIZZLY0013: Exception during FilterChain execution
java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:275)
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.fill(TCPNIOUtils.java:200)
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.writeCompositeBuffer(TCPNIOUtils.java:81)
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:112)
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:91)
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:261)
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:170)
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:70)
at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleWrite(TCPNIOTransportFilter.java:126)
at org.glassfish.grizzly.filterchain.TransportFilter.handleWrite(TransportFilter.java:191)
at org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:848)
at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:817)
at org.glassfish.grizzly.http.io.OutputBuffer.flushBuffer(OutputBuffer.java:1024)
at org.glassfish.grizzly.http.io.OutputBuffer.flushBinaryBuffers(OutputBuffer.java:1011)
at org.glassfish.grizzly.http.io.OutputBuffer.flushAllBuffers(OutputBuffer.java:982)
at org.glassfish.grizzly.http.io.OutputBuffer.flush(OutputBuffer.java:737)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:291)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:275)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:175)
at jp.co.my.do.publisher.handler.EventHandler.onEvent(EventHandler.java:88)
at org.glassfish.grizzly.comet.DefaultNotificationHandler.notify0(DefaultNotificationHandler.java:117)
at org.glassfish.grizzly.comet.DefaultNotificationHandler.notify(DefaultNotificationHandler.java:93)
at org.glassfish.grizzly.comet.DefaultNotificationHandler.notify(DefaultNotificationHandler.java:83)
at org.glassfish.grizzly.comet.CometContext.notify(CometContext.java:437)
at jp.co.my.do.publisher.management.CometManagement.notifyPrd(CometManagement.java:115)
at jp.co.my.do.publisher.management.timertask.DataPushTask.eventNotify(DataPushTask.java:131)
at jp.co.my.do.publisher.management.timertask.DtatPushTask.run(DataPushTask.java:99)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
]]

Otherwise, It was coming up other errors before or after. There ware coming up errors on end since a few months ago.

GRIZZLY0029: Error during Processor execution. java.lang.IllegalArgumentException: Position exceeds a limit: XXX>XXX
GRIZZLY0029: Error during Processor execution. java.lang.IllegalStateException: HeapBuffer has already been disposed
GRIZZLY0029: Error during Processor execution. java.lang.IndexOutOfBoundsException
GRIZZLY0013: Exception during FilterChain executio java.lang.IndexOutOfBoundsException

The overview of application.

• Using Push notification of Comet.
• After creating persistent session, It will be sent some data from process continually until disconnected by The client.
• There are several kinds of Push notification. Each thread has it that will be sent some data by event driven or timer.
  Otherwise The functions has a method for confirming normality from one server to another as [Heart-beat].
• There is sent [Heart-beat] from server to client interval of ten seconds.
• If The client side can't confirm [Heart-beat]'s signal from its communications three times (total 30 seconds),
  It will be disconnect the session that is not normal situations.
• Process is connected hundreds sessions always from any clients.
• Domain will be reboot weekly.
• OS will be reboot annual.(But not yet)

Situation

#1 The error has never occurred since the first construction(about half year ago)
#2 It was coming up this error, After OS was rebooting three days lately.(This rebooting is regularly and weekly)
#3 It was coming up this error only one side. Our system have the two nodes.
#4 Otherwise, It was coming up other errors before or after. There ware coming up errors on end since a few months ago.

GRIZZLY0029: Error during Processor execution.
java.lang.IllegalArgumentException: Position exceeds a limit: XXX>XXX

GRIZZLY0029: Error during Processor execution.
java.lang.IllegalStateException: HeapBuffer has already been disposed

GRIZZLY0029: Error during Processor execution.
java.lang.IndexOutOfBoundsException

GRIZZLY0013: Exception during FilterChain execution.
java.lang.IndexOutOfBoundsException

#5 After more than twenty minutes since these error was occured,
It has became only the error as follows.

GRIZZLY0013: Exception during FilterChain execution
java.lang.IllegalArgumentException

#6 As The coming timing of this error, it seems that was sent the first Push notification.
As follow:
（ Continuing damage for 17 hours, but we couldn't check all of errors ）

① Data is sent every a few minutes.
：
：
② It begin to connect from client.
Data is sent every a few minutes. → be output
Data is sent every a few minutes. → not be output
Data is sent every a few minutes. → not be output

③ It begin to connect from client.
Data is sent every a few minutes. → be output
Data is sent every a few minutes. → not be output

#7 State of All the services.

• When occur, Many sessions from rich client will be disconnected to Glassfish server.
• Since we will be unable to connect to server from rich clients every one.
• When we tried to reboot the domain(process) of Glassfish, But it was not fix.
  In fact the domain(process) couldn't be started by restart command.

Workaround

There is only one workaround.

Then we tried to do the below of methods, We can recover our system. Since the rich clients can connect to server as usual.
Stop its Domain -> Reboot OS -> Start its Domain

Note that if we reboot only process, Process was connected from client one-time-only.
But After about thirty minutes Process was rebooted.

■■■■■

Question 1

Do you know why come up this [WARNING][Error]?
What happen has this [WARNING][Error] our glassfish?

Question 2

Do you know how to workaround?

Question 3

Do you know how to fix?

Question 4

Do you has experienced improving this problem as case example?

Question 5

How should we investigate when we encounter this problem again?

Best Regards.

Environment

System Environment

OS Redhat EL 6
Glassfish 4.1
Grizzly 2.3.15
Oracle JDK 1.8.0_61

Affected Versions

[2.3.15]

[glassfishrobot]

Reported by t.shimada

[glassfishrobot]

oleksiys said:
Hi,

it's hard to say what exactly is going wrong. It's possible you read/update some thread-unsafe construction from multiple threads, that causes all sorts of issues.
Can you create a testcase to reproduce the problem?

Thank you.

[glassfishrobot]

t.shimada said:
Thank you for your reply.

I'm afraid to you.
we can't provide sample application as test-case.
Because that application is our customers's and It is complexity.

So if you know how to workaround, could you teach me it?
or please tell me if there are any examples.
I want to try these your knowledge.

Otherwise,
Will you please tell me?

#1 If our application don't use thread-unsafe, It will occur GRIZZLY0029?
#2 Is GRIZZLY0029 related to thread-unsafe?

Best regards.

[glassfishrobot]

oleksiys said:
Hi,

can you please try 2.3.25-SNAPSHOT version for the maven repo [1]?
If you still see the problem - I'll need a testcase, it doesn't have to be your customer's code, but just something I can use to reproduce the problem.

Thank you.

[1] https://maven.java.net/content/repositories/snapshots/

[glassfishrobot]

t.shimada said:
Hello I'm Toshikazu from Japan.
Thank you for your information.

Please tell me more.

We are going to try your advice.

can you please try 2.3.25-SNAPSHOT version for the maven repo [1]?

I want to know how to build and should include files for build.

1. Premise

To Compile with Java Version JDK1.7_025(64bit) and Maven Version 3.2.5

2. For Preparation

To get source

svn export https://svn.java.net/svn/glassfish~svn/tags/4.1

3. For update grizzly version

To edit nucleus\pom.xml

Below are the details:

Before：<grizzly.version>2.3.15-gfa</grizzly.version>
After：<grizzly.version>2.3.25-SNAPSHOT</grizzly.version>

4. To compile jar file

① We are goning to compile with mvn command for make jar files.
We think the file which it should update to be nucleus-grizzly-all.jar and glassfish-grizzly-extra-all.jar.

② We are going to locate to be made jar files.

・nucleus\grizzly\nucleus-grizzly-all\target\nucleus-grizzly-all.jar
-> glassfish4.1\glassfish\modules\nucleus-grizzly-all.jar

・appserver\grizzly\glassfish-grizzly-extra-all\target\glassfish-grizzly-extra-all.jar
-> glassfish4.1\glassfish\modules\glassfish-grizzly-extra-all.jar

Question 1:

About [ 1) ], Which are better JDK1.7_025(64bit) or JDK1.7_025(32bit) When I compile with JDK?

Question 2:

Our Does our procedure correct?

Question 3:

About [ 4) ], Do other jar files need for update?

Question 4:

We encountered a error of below when grizzly.version config「2.3.25-SNAPSHOT」in pom.xml.
But we could build without error after we config it「2.3.25」in pom.xml.
Which is correct?

Error:

[INFO] Scanning for projects...
[ERROR] The build could not read 2 projects -> [Help 1]
[ERROR]
[ERROR] The project org.glassfish.main:glassfish-nucleus-parent:4.1 (C:\XXXXXXXXXX\nucleus\pom.xml) has 1 error
[ERROR] Non-resolvable import POM: Could not find artifact org.glassfish.grizzly:grizzly-bom:pom:2.3.25-SNAPSHOT @ l
ine 659, column 25 -> [Help 2]
[ERROR]
[ERROR] The project org.glassfish.main:glassfish-parent:4.1 (C:\XXXXXXXXXX\appserver\pom.xml) has 1 error
[ERROR] Non-resolvable import POM: Could not find artifact org.glassfish.grizzly:grizzly-bom:pom:2.3.25-SNAPSHOT @ o
rg.glassfish.main:glassfish-nucleus-parent:4.1, C:\XXXXXXXXXX\nucleus\pom.xml, line 659, column 25 -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException

Question 5:

Which is better our method or to procedure to make individually according to GIZZLY-1734 and #1698")?

Best Regards.

[glassfishrobot]

This issue was imported from java.net JIRA GRIZZLY-1830