Why no bulk Arrow→Parquet write API in Java? How to avoid row-by-row RecordConsumer + optimize?

[Fenil-v]

Describe the usage question you have. Please include as many useful details as possible.

I have ~ 20KB objects that I need to write to Parquet efficiently from Java.
In C++, C#, and Python there's a direct/bulk Arrow-Parquet write (e.g. WriteTable / write_table) that avoids row-by-row iteration, but in Java I only see row-by-row paths via RecordConsumer or internal/unstable column writers.
Questions:

1. Is there a supported bulk/columnar Arrow-Parquet write API in Java (e.g, VectorSchemaRoot
   → Parquet) that avoids row-by-row calls?
2. If not, why is Java limited to row-by-row writes today? Any roadmap for feature parity with C++/Python/C#?
3. For now, what's the recommended optimization path to write 20KB objects at high throughput from Java (without JNI), or is JNI/Dataset the recommended route?
4. Any best practices (batch sizing, encodings, writer settings) to mitigate the row-by-row overhead?

Component(s)

Java

[V-Fenil]

Describe the usage question you have. Please include as many useful details as possible.

I have ~ 20KB objects that I need to write to Parquet efficiently from Java. In C++, C#, and Python there's a direct/bulk Arrow-Parquet write (e.g. WriteTable / write_table) that avoids row-by-row iteration, but in Java I only see row-by-row paths via RecordConsumer or internal/unstable column writers. Questions:

1. Is there a supported bulk/columnar Arrow-Parquet write API in Java (e.g, VectorSchemaRoot
   → Parquet) that avoids row-by-row calls?
2. If not, why is Java limited to row-by-row writes today? Any roadmap for feature parity with C++/Python/C#?
3. For now, what's the recommended optimization path to write 20KB objects at high throughput from Java (without JNI), or is JNI/Dataset the recommended route?
4. Any best practices (batch sizing, encodings, writer settings) to mitigate the row-by-row overhead?

Component(s)

Java

Describe the usage question you have. Please include as many useful details as possible.

I have ~ 20KB objects that I need to write to Parquet efficiently from Java. In C++, C#, and Python there's a direct/bulk Arrow-Parquet write (e.g. WriteTable / write_table) that avoids row-by-row iteration, but in Java I only see row-by-row paths via RecordConsumer or internal/unstable column writers. Questions:

1. Is there a supported bulk/columnar Arrow-Parquet write API in Java (e.g, VectorSchemaRoot
   → Parquet) that avoids row-by-row calls?
2. If not, why is Java limited to row-by-row writes today? Any roadmap for feature parity with C++/Python/C#?
3. For now, what's the recommended optimization path to write 20KB objects at high throughput from Java (without JNI), or is JNI/Dataset the recommended route?
4. Any best practices (batch sizing, encodings, writer settings) to mitigate the row-by-row overhead?

Component(s)

Java

@pitrou any idea on this? Any help would be saver for me

[Fenil-v]

Describe the usage question you have. Please include as many useful details as possible.

I have ~ 20KB objects that I need to write to Parquet efficiently from Java. In C++, C#, and Python there's a direct/bulk Arrow-Parquet write (e.g. WriteTable / write_table) that avoids row-by-row iteration, but in Java I only see row-by-row paths via RecordConsumer or internal/unstable column writers. Questions:

1. Is there a supported bulk/columnar Arrow-Parquet write API in Java (e.g, VectorSchemaRoot
   → Parquet) that avoids row-by-row calls?
2. If not, why is Java limited to row-by-row writes today? Any roadmap for feature parity with C++/Python/C#?
3. For now, what's the recommended optimization path to write 20KB objects at high throughput from Java (without JNI), or is JNI/Dataset the recommended route?
4. Any best practices (batch sizing, encodings, writer settings) to mitigate the row-by-row overhead?

Component(s)

Java

Describe the usage question you have. Please include as many useful details as possible.

I have ~ 20KB objects that I need to write to Parquet efficiently from Java. In C++, C#, and Python there's a direct/bulk Arrow-Parquet write (e.g. WriteTable / write_table) that avoids row-by-row iteration, but in Java I only see row-by-row paths via RecordConsumer or internal/unstable column writers. Questions:

1. Is there a supported bulk/columnar Arrow-Parquet write API in Java (e.g, VectorSchemaRoot
   → Parquet) that avoids row-by-row calls?
2. If not, why is Java limited to row-by-row writes today? Any roadmap for feature parity with C++/Python/C#?
3. For now, what's the recommended optimization path to write 20KB objects at high throughput from Java (without JNI), or is JNI/Dataset the recommended route?
4. Any best practices (batch sizing, encodings, writer settings) to mitigate the row-by-row overhead?

Component(s)

Java

@pitrou any idea on this? Any help would be saver for me

@julienledem

[julienledem]

Hello @V-Fenil,
If a feature is not present, it is usually because it has not been contributed yet. If the internal column writers work for you, you are welcome to propose an API for them.
For your last question, I would suggest you experiment to see if changing batch size has an effect on what you're trying to achieve.

[lidavidm]

This doesn't even seem to be an Arrow question, given mentions of parquet-java APIs? parquet-java and arrow-java are unrelated projects.

I say this duplicates #735.

[V-Fenil]

Thanks for re-directing to #735, that was exactly I was looking for.