NullPointerException in StandardTokenizerImpl in Solr 4.1.0

[nolanlawson]

This bug only seems to occur in Solr 4.1.0. Affects version 1.2.

Steps to reproduce: follow the "Getting Started" directions for Solr 4.1.0, and punch in the URL as directed. The response is:

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<response>
<lst name="responseHeader">
<int name="status">500</int>
<int name="QTime">6669</int>
<lst name="params">
<str name="debugQuery">on</str>
<str name="q">dog</str>
<str name="qf">text</str>
<str name="synonyms">true</str>
<str name="defType">synonym_edismax</str>
</lst>
</lst>
<lst name="error">
<str name="trace">
...
</str>
<int name="code">500</int>
</lst>
</response>

And the stacktrace is:

SEVERE: null:java.lang.NullPointerException
    at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:923)
    at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:1133)
    at org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:180)
    at org.apache.lucene.analysis.shingle.ShingleFilter.getNextToken(ShingleFilter.java:367)
    at org.apache.lucene.analysis.shingle.ShingleFilter.shiftInputWindow(ShingleFilter.java:421)
    at org.apache.lucene.analysis.shingle.ShingleFilter.incrementToken(ShingleFilter.java:286)
    at org.apache.lucene.analysis.synonym.SynonymFilter.parse(SynonymFilter.java:358)
    at org.apache.lucene.analysis.synonym.SynonymFilter.incrementToken(SynonymFilter.java:624)
    at org.apache.solr.search.SynonymExpandingExtendedDismaxQParser.generateSynonymQueries(SynonymExpandingExtendedDismaxQParserPlugin.java:359)
    at org.apache.solr.search.SynonymExpandingExtendedDismaxQParser.attemptToApplySynonymsToQuery(SynonymExpandingExtendedDismaxQParserPlugin.java:273)
    at org.apache.solr.search.SynonymExpandingExtendedDismaxQParser.parse(SynonymExpandingExtendedDismaxQParserPlugin.java:262)
    at org.apache.solr.search.QParser.getQuery(QParser.java:142)
    at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:117)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:194)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:365)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
    at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
    at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
    at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
    at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:680)

[sarowe]

Looks like zzReader wasn't initialized - here's the StandardTokenizerImpl statement where the NPE occurs:

923: int numRead = zzReader.read(zzBuffer, zzEndRead,
924: zzBuffer.length-zzEndRead);

If you look at the package javadocs for o.a.l.analysis: http://lucene.apache.org/core/4_1_0/core/org/apache/lucene/analysis/package-summary.html and the tokenstream workflow http://lucene.apache.org/core/4_0_0/core/org/apache/lucene/analysis/TokenStream.html (step 2), you can see the the reset() method on a tokenstream must be called before it's used.

You didn't do that.

Why is this suddenly a problem in Lucene 4.1? Calling reset() prior to consuming a tokenstream has been part of the contract since Lucene 2.9, but has not been strictly enforced. This commit, between the 4.0 and 4.1 releases, triggered the change: http://svn.apache.org/viewvc?view=revision&revision=1401449

[nolanlawson]

Thanks for the insight! That certainly fixed it; Solr 4.1.0 works fine now. Bad on me for not reading the documentation for TokenStream.