Caused by: java.lang.IllegalArgumentException: position increments (and gaps) must be >= 0 (got 65248) for field 'tf_attachments_field_library_attachments' [LUCENE-7467]

[asfimport]

I was try to indexing the large file like PDF,PPT,PPTX and XLXS,XLX,
actual token count are 65248 but this error is coming from the DefaultIndexingChain class while executing the public void invert(IndexableField field, boolean first) throws IOException, AbortingException in side this method we are checking the one condition like if (invertState.position < invertState.lastPosition) here invertState.position value become negative, when it increases int.MAX_VALUE,

org.apache.solr.common.SolrException: Exception writing document id dc65t0-marketing_site-141457 to the index; possible analysis error.
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:167)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:955)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1110)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:706)
at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:250)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:177)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:98)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2068)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:669)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:462)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:210)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1099)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: position increments (and gaps) must be >= 0 (got 65248) for field 'tf_attachments_field_library_attachments'
at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:631)
at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:344)
at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1475)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:163)
... 33 more

---

Migrated from LUCENE-7467 by adeppa, updated Jun 12 2019
Environment:

Mac 

Attachments: AutoPhrasingTokenFilter.java, AutoPhrasingTokenFilterFactory.java

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Hmm, just how large are your documents?

What analyzer are you using?

Do you have a very high position gap?

In normal usage it should be hard to overflow 2.1B positions.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

I'll fix the exception message ... it's ridiculous now since it claims 65248 is < 0.

[asfimport]

adeppa (migrated from JIRA)

Thanks for the reply ,
I am using the currently AutoPhrasingTokenFilter from third party i.e lucidworks ,
for the more details contains below link
http://lucidworks.com/blog/2014/07/12/solution-for-multi-term-synonyms-in-lucenesolr-using-the-auto-phrasing-tokenfilter/
source code
https://github.com/LucidWorks/auto-phrase-tokenfilter
please can you help me respective change for this

[asfimport]

ASF subversion and git services (migrated from JIRA)

Commit 01ecc06 in lucene-solr's branch refs/heads/master from Mike McCandless
https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=01ecc06

LUCENE-7467: improve exception message when token position overflows int

[asfimport]

ASF subversion and git services (migrated from JIRA)

Commit 6f6f73e in lucene-solr's branch refs/heads/branch_6x from Mike McCandless
https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6f6f73e

LUCENE-7467: improve exception message when token position overflows int

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Hi adeppa, I'm not sure what AutoPhrasingTokenFilter does; maybe you should open an issue with lucidworks? Are you sure the too-big position is caused by that token filter? Can you reproduce the issue with a small example?

[asfimport]

adeppa (migrated from JIRA)

@mikemccand Hi Thanks for the help form your end , small data no problem, if file size is more than 5 MB then only a problem is occurring, currently in progress of token code modification ,once finish i will update ticket with details ,please validate and let me know if you have chance

[asfimport]

Trejkaz (migrated from JIRA)

Saw the same thing on Lucene 5.5.2.

DefaultIndexingChain$PerField.invert: IllegalArgumentException: position increments (and gaps) must be >= 0 (got 536870912) for field 'content'

I guess what makes this confusing is that the value it's complaining about is >= 0, but the error message makes it sound like it isn't.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

I guess what makes this confusing is that the value it's complaining about is >= 0, but the error message makes it sound like it isn't.

We fixed the exception message to not be ridiculous like that :), at some point in the 6.x series ...

[asfimport]

longlei (migrated from JIRA)

https://stackoverflow.com

[asfimport]

longlei (migrated from JIRA)

change the field bellow:

@field(store = Store.YES, index = Index.YES, analyze = Analyze.YES)

--->

@field(store = Store.YES, index = Index.YES, analyze = Analyze.NO)