[Java] Support re-emitting dictionaries in ArrowStreamWriter

[asfimport]

The ArrowStreamWriter currently takes a DictionaryProvider at construction time and emits the used dicts once.

However, the streaming format allows for the dictionaries to change between record batches. It would be useful to support this mechanism. It can be worked around in various ways (e.g. manually re-emitting DictionaryBatches between calling writeBatch), but this isn't very pleasant.

We'd somehow have to reconcile this with the abstract ArrowWriter parent and the ArrowFileWriter sibling. In the latter, for example, this mechanism is not supported.

An example solution (but perhaps we can do better) might be to add a virtual writeBatch(Provider provider) method, that is UnsupportedOperationException in ArrowFileWriter, and re-emits the used dicts in ArrowStreamWriter.

In the present context just looking at dictionary replacement, not dictionary delta's.

 

 

Reporter: Joris Peeters / @jmgpeeters

_{Note: This issue was originally created as ARROW-11869. Please see the migration documentation for further details.}

[asfimport]

Todd Farmer / @toddfarmer:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

[adamreeve]

We've run into this issue and I'd like to work on a fix. It appears that the C++ implementation handles this transparently by re-writing any dictionaries that have changed, so it should be possible to implement a similar solution in Java:

arrow/cpp/src/arrow/ipc/writer.cc

Line 1121 in 018e7d3

Status WriteDictionaries(const RecordBatch& batch) {

[adamreeve]

take