Better handling for long text

[wesslen]

Currently use approx word counts by whitespace before sending to API; will truncate.

Also skips if OpenAI comes back with error.

Need a better solution to check with OpenAI's tokenizer.

[wesslen]

https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb

[wesslen]

Check if appendix is included for this example https://browse.arxiv.org/html/2401.00437v1

[wesslen]

Moved to 13,500 99add83

Still need to improve handling.

[wesslen]

Now skips when too long, but still unsatisfactory.

Here are some current ones:

2312.07392v1
2312.13107v1
2312.17164v1
2401.01149v1

[wesslen]

Found that many are due to equations. Removed ".ltx_equation" with this update: 6f65cbe

[wesslen]

Removed .ltx_Math and `.ltx_theroem" too

This seems to improve unexpected lengths

[wesslen]

Added LangChain plus MapReduce #23