Add basic arrow schema conversion

[danielcweeks]

There are a number of things that still need to be addressed and we can create follow up issues for:

• Dictionaries are part of the Arrow Schema definition, but should be built based off the parquet dictionaries. This probably means that conversion needs to happen while processing the row group metadata to capture the parquet dictionaries.

• This converts Maps to lists<struct<key,value>>, but that is probably not what is ideal for columnar Spark representation (or other engines).

[mccheah]

For any new module we make, we should probably start with Baseline. For here, Baseline will complain about static imports - but these might make some sense to have exceptions for in the checkstyle.xml.

[mccheah]

I'd be more comfortable with importing org.apache.arrow.vector.types.pojo.ArrowType.ArrowTypeID, and then addressing by ArrowTypeID.Date, etc.

[danielcweeks]

This doesn't seem to be currently enforced. Was there a change to the checkstyle to relax this convention or do you still prefer the proposed import changes?

[mccheah]

Build is failing - is this ready for review as a standalone item or did you want to do more here? What's next for this feature moving forward?

[xhochy]

Dictionaries are part of the Arrow Schema definition, but should be built based off the parquet dictionaries.

@danielcweeks This actually changed in Arrow master as it lead to a lot of problems. With the changes in Arrow master, reading rowgroups with Dictionaries should be much simpler.

[danielcweeks]

@mccheah We're going to start investing in building out the arrow read path starting in a couple weeks, so I'll try to address the checkstyle issues and we'll build on this from there.

[mccheah]

Ping on this - are we continuing to make progress here?

[danielcweeks]

@mccheah thanks for the ping. I've rebased and addressed the checkstyle issues.

We are actively working toward an arrow read path, so despite this not being used, it would probably help to have this reviewed and included to keep the commits smaller.

[rdblue]

I'm not sure that it's a good idea to add Arrow to iceberg-core until there is more functionality there. Could we keep this as a PR until we know more about what the Arrow integration will look like? It would make sense as an add-on module like iceberg-parquet or iceberg-orc.

[danielcweeks]

@rdblue Keeping as a PR is fair and it does seem that arrow would make more sense as a separate module.

[mccheah]

Is there a world where we can keep this feature behind a feature flag so that development can proceed forward with reasonably sized diffs and merges?

[rdblue]

@mccheah, we do intend to move forward with reasonable size merges. For this, I'm not sure that it is worth updating it because we don't really know what the module structure will be for Arrow yet, and that will depend on the implementation. We can definitely update this to be in its own iceberg-arrow module and add the Arrow dependency to core, but since this PR is mostly to demonstrate the conversion I wasn't sure that was worth spending time on.

[mccheah]

I think it would be appropriate to clarify what the plan is for integrating Arrow with Iceberg and the data format integrations. I'll follow up on #9.

[rdblue]

I'm going to close this because it is in the vectorized read branch and will be merged with that work. For status, there is a vectorized read milestone.