Remove JsonEqual trait

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

All Array implementations must implement the JsonEqual trait. However:

• UnionArray will always panic
• DictionaryArray is incorrect (JsonEqual for DictionaryArray Only Compares Keys #2294)
• It is unclear to me why we would special case JSON
• It complicates making JSON an optional feature

Describe the solution you'd like

Rather than maintaining parallel logic to interpret serde_json::Value as arrow arrays, I wonder if we can just remove the JsonEqual functionality and encourage conversion to arrow arrays. I'm unclear of a use-case that would require highly-performant comparison of arrow and JSON data

Describe alternatives you've considered

Additional context

[tustvold]

This appears to be part of the integration test plumbing, which makes its inclusion in the public API even more unusual?

FYI @viirya

[viirya]

I think in the integration test, JSON data is read and converted to arrow array before comparing? Does it compare JSON with arrow arrays directly?

[tustvold]

It at least appears to be using the JsonEqual functionality, but I'm not very familiar with this part of the codebase

[viirya]

Oh, I will check it. I might miss it now.

[viirya]

At second glance, JsonEqual functionality is used by ArrowJson to compare with another RecordBatchReader. Underlying JSON record batch ArrowJsonBatch is compared with RecordBatch.

Although the stuffs like ArrowJson is under integration_util, and it is indeed used by integration test. But it seems to be only used as intermediate format to read JSON data and convert to arrow array. I don't see that JSON is compared directly with arrow arrays (I may miss it if it is deep).

Currently seems the JsonEqual functionality is only used in some IPC tests where JSON files are read and compare with arrow arrays without converting.

For that tests, I think we can follow integration test to convert JSON to arrow arrays before comparing.

I can take some time on this.

[jhorstmann]

For that tests, I think we can follow integration test to convert JSON to arrow arrays before comparing.

As another data point, in our integration tests we convert result RecordBatches to json to compare against the expected results. So removing the JsonEqual support should be fine.