Dictionaries can only hold a maximum of 4096 indices

[asfimport]

Dictionaries seem to only be able to hold 4096 indices, meaning only vectors with 4096 values or less can be turned into dictionaries. The image attached is a stack trace of what happens when try to encode a dictionary with a vector containing 4097 strings, and a dictionary containing two distinct values.

Basically the error can be traced to line 95 of DictionaryEncoder.java (setter.invoke(mutator, i, encoded);). It seems that the indices array which hold the encoded values is allocated on line 84 as indices.allocateNew() and it seems that allocateNew() only allocates 4096 bytes of data initially. The code runs if there are 4096 rows of data or less. Anymore and the same error is given.

Reporter: Shayan Monshizadeh
Assignee: Li Jin / @icexelloss

Original Issue Attachments:

• Screen Shot 2017-08-22 at 7.14.07 PM.png

_{Note: This issue was originally created as ARROW-1407. Please see the migration documentation for further details.}

[asfimport]

Emilio Lahr-Vivaz / @elahrvivaz:
FYI, you don't have to use the DictionaryEncoder class. If you don't mind mapping your dictionary values yourself, you can do something like:

NullableIntVector vector = new FieldType(true, MinorType.INT.getType, dictionaryEncoding).createNewSingleVector(name, allocator, callBack);
vector.getMutator().setSafe(i, j);

[asfimport]

Wes McKinney / @wesm:
@icexelloss or @alphalfalfa any interest in taking a look at this?

[asfimport]

Li Jin / @icexelloss:
I'll take this one.

[asfimport]

Li Jin / @icexelloss:
PR: #1024

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 1024
#1024