MV encoding config not working

[ksnijjer]

This is related to #11993, while this config works

"fieldConfigList": [
      {
        "name": "col_name",
        "encodingType": "DICTIONARY",
        "compressionCodec": "MV_ENTRY_DICT"
      }]

this format doesn't seem to work

"fieldConfigList": [
{
  "name": "service_code",
  "encodingType": "DICTIONARY",
  "indexes": {
    "forward": {
      "dictIdCompressionType": "MV_ENTRY_DICT"
    }
  }
} ]

@Jackie-Jiang can we check on this one?

[Jackie-Jiang]

@gortiz Can you help take a look? Not sure if I missed something

[gortiz]

I wasn't able to reproduce the issue. This is what I did:

1. Open pinot-tools/src/main/resources/examples/batch/airlineStats/airlineStats_offline_table_config.json and modified the fieldConfigList to add

   {
      "name": "DivAirportIDs",
      "encodingType": "DICTIONARY",
      "compressionCodec": "MV_ENTRY_DICT"
    }

2. Started pinot-tools/src/main/java/org/apache/pinot/tools/GenericQuickstart.java
3. Checked that airlineStats.DivAirportIDs was using MV_ENTRY_DICT. I wasn't sure how to do this, so what I did was to add a breakpoint in both:
   • MultiValueFixedByteRawIndexCreator.java constructor
   • and FixedBitMVEntryDictForwardIndexReader.java constructor
4. Stopped the instance and delete temp data.
5. Modified airlineStats_offline_table_config.json again to change the fieldConfigList to:

    {
      "name": "DivAirportIDs",
      "encodingType": "DICTIONARY",
      "indexes": {
        "forward": {
          "dictIdCompressionType": "MV_ENTRY_DICT"
        }
      }
    }

6. Started GenericQuickstart again.
7. Checked that airlineStats.DivAirportIDs was using MV_ENTRY_DICT doing the same as before.

In both cases it looks like MV_ENTRY_DICT is being used. Am I missing something?

[ksnijjer]

Thanks Gonzalo, I was verifying it differently - by looking at index_map and forward index size for the column (without MV encoding it's really huge). Not sure why you can't repro perhaps difference could be quick start using latest version of master vs original env using earlier.
Anyways if we can't repro with latest code then I am good to close unless @Jackie-Jiang has any additional thoughts/thinks to check.

[gortiz]

Before closing it, let me try with your version. Do you know which commit or tag was used so I can use the same one?

[ksnijjer]

@gortiz will DM you the exact release, since it's StarTree release not sure how to map it to OSS.

[Jackie-Jiang]

@gortiz @ksnijjer Have you guys figured out the root cause?

[gortiz]

I was able to reproduce the issue with the older Pinot version @ksnijjer was using, but not with the latest one. I think we can close it