payload in the development of lucene plugin pipelines

[ziqizhang]

Lucene supports attaching information to a token produced in the plugin pipeline. This information is attached in a payload attribute, which takes a BytesRef object that is constructed on a byte[] or string.

The best way to attach such information, based on experience so far, is to create a BytesRef object from a String (which can be a JSON string and using Gson library to convert between objects and strings), then convert the BytesRef object by calling BytesRef.utf8tostring.

DO NOT using serialisation or custom byte[] to construct BytesRef objects. This has caused many inconsistent write/read time data that are inexplicable.