[Sprint: 49] XD-2817 - gemfire modules close client cache on undeploy

[dturanski]

Backport to 1.1.x?

[markpollack]

I have mixed feelings about this one. It is nice to remove the gemfire jar being in 4 places, but I don't like adding it to xd/lib so much since we generally want to minimize the size of xd/lib. It also means antlr goes onto xd/lib and that has the potential for some nasty CP conflicts in the future.

Depending on how you view it, having it in a single spot in xd/lib means that you can't easily have gemfire version specific modules. For example, the 7.0 series is a bit old now (waiting for spring data to upgrade to 8.x). So one either updates xd/lib with 8.0 jars or updates each module individually. I suspect people would generally want to have the same version of gemfire everywhere, but we should think about that a bit before basically enforcing it.

Does the comment in build.grade "The following loaded to prevent issues with gemfire modules" mean that something is broken OOTB now? If so, we must have some gap in the CI integration tests.

[dturanski]

The comment refers to this issue. Also Gemfire does require the same version everywhere in my experience. Another approach is to rely on the additional module classpath entries feature that Eric recently added and rely on $GEMFIRE_HOME to locate the jar.

[ericbottard]

I agree with Mark that moving stuff to the main classpath is starting to be the "default solution" to our classloading problems, but this creates other complications. At one point in time, we'll have to really look into those.

For the matter at hand though, please note that leveraging the "additional module classpath" feature I added recently would not change the current classloading scheme (if this is indeed the source of the problem this PR is trying to address). The gemfire jars would still be loaded by the module classloader. The "benefit", would be that they could be physically shared to a central filesystem location (what Dave denotes $GEMFIRE_HOME above)

[dturanski]

So my alternate suggestion won't work. Another related issue is the XD container is a client to gemfire and doesn't require a local GF installation. The root problem here is that Gemfire modules cannot close the client cache when undeployed because the client cache is a true singleton shared by all GF modules. If there are other streams deployed that contain GF modules, they will fail if one of them closes the cache. The CNFE happens because the thread that monitors the locator connection continues as long as the client cache and container is running. I have considered implementing a background task to detect when there are no GF modules deployed and shut down the client cache at that point. However this would also require gemfire.jar to be on the main classpath.

[markpollack]

Just finished reading the JIRA to better understand the root cause. Is it possible to have our gemfire modules never shutdown the cache? Alternatively, is it possible to use gemfire's REST API to do this or to expose a custom JMX endpoint that can be invoked 'out of band' to the stream destruction to shutdown the cache?

[ericbottard]

Just to recap in case this gets lost, I would -1 on this approach.

Making sure the (isolated) module shuts down evreything GemFire related seems like a better approach to me (and initial POC seemed promising)

[markfisher]

+1 to @ericbottard (so another -1 for adding GF to the main classpath)

[dturanski]

Agreed

On Tue, May 12, 2015 at 11:26 AM, Eric Bottard notifications@github.com
wrote:

Just to recap in case this gets lost, I would -1 on this approach.

Making sure the (isolated) module shuts down evreything GemFire related
seems like a better approach to me (and initial POC seemed promising)

—
Reply to this email directly or view it on GitHub
#1585 (comment)
.

[dturanski]

I manually testing with 2 streams, following the examples at http://docs.spring.io/spring-xd/docs/current-SNAPSHOT/reference/html/#gemfire-server

xd> stream create gf --definition "http  | gemfire-json-server --regionName=Stocks --keyExpression=payload.getField('symbol')" --deploy
xd> stream create gf2 --definition "http --port=9001 | gemfire-json-server --regionName=Stocks --useLocator=true --port=10334 --keyExpression=payload.getField('symbol')" --deploy

posting data and udeploying/destroying and redeploying/creating one or the other stream. Repeating, using a locator as in the reporter's scenario. This requires a full Gemfire 7.0.2 install and using gfsh, etc.

xd> stream create gf --definition "http --port | gemfire-json-server --regionName=Stocks --useLocator=true --port=10334 --keyExpression=payload.getField('symbol')" --deploy
xd> stream create gf2 --definition "http --port=9001 | gemfire-json-server --regionName=Stocks --useLocator=true --port=10334 --keyExpression=payload.getField('symbol')" --deploy

Everything works as expected.

[ericbottard]

Did you destroy the streams as well, as discussed in the JIRA comment?
I'll try to have a look at it myself, but this is what triggered an error (as reported in hipchat) at the time.
Also, I looked at possible leakages with yourkit and saw some (but the JMX bean was a retainer, need to have a closer look, e.g. with JMX disabled)

[ericbottard]

Rebased and merged as 3651677.

Tested with this stream definition:

stream create streamx --definition "trigger --fixedDelay=3 | http-client --url='''https://query.yahooapis.com/v1/public/yql?q=select * from yahoo.finance.quote where symbol in (\"MSFT\")&format=json&env=store://datatables.org/alltableswithkeys''' --httpMethod=GET | splitter --expression=#jsonPath(payload,'$.query.results.quote') | transform --expression=payload.toString() | gemfire-json-server --regionName=Stocks --keyExpression=payload.getField('Symbol')" --deploy

The GemFire error is gone.

There is still leakage though, but it is independent of GemFire.
One of the problems is from enabling JMX, and still after disabling that, there is leakage. I created https://jira.spring.io/browse/XD-3069 to track that.