long text

[romanchyla]

sizes over 32766 chars cause solr to reject the document

incidentally, I'm seeing that the extraction is full of \u00a in place of spaces; perhaps that could be treated

example: 2009arXiv0906.5028K

this\\u00a0research\\u00a0project\\u00a0was\\u00a0conducted\\u00a0at\\u00a0St.\\u00a0Louis\\u00a0University\\u00a0School\\u00a0of\\u00a0Medicine.\\u00a0\\nTK\\u00a0received\\u00a0salary\\u00a0support\\u00a0from\\u00a0Sankyo\\u00a0Co,\\u00a0Ltd

[spacemansteve]

We still see long words in the body field, the code to prevent this is at https://github.com/adsabs/ADSfulltext/blob/master/adsft/utils.py#L246. The computed body field for bibcode 2018Tectp.722...69L has a very long toke that begins with
#[U] ppm[Th] ppm[Pb] ppmTh/U meas207Pb/206Pbs %207Pb/235Us %206Pb/238Us %r207Pb/206Pbs207Pb/235Us206Pb/238Usf206%54,550@114642786510.1900.128
Long tokens are identified with a regular expression r'\b\w{'+str(maxlength)+r',}. It doesn't work well because the tables that appear as a single token are full of special characters like /, %, @, etc. A very long string that repeated %123%123%123 would not be changed. Instead, I think \b\w should be replaced with \S which will match on all non-whitespace characters, including special characters which the tokens are full of.

[spacemansteve]

reopen until deployed

[spacemansteve]

Should not have been re-opened. Recent "token too large" were fixed by re-extracting fulltext. The code works.