ARROW-16913: [Java] Implement ArrowArrayStream #13465

lidavidm · 2022-06-29T14:34:37Z

Implements ArrowArrayStream for Java. The equivalent Java-side interface chosen is ArrowReader.

Also:

Fixes a couple of JDK9 compatibility issues I ran into. I think these will not normally affect people except during development (I think because I was mixing IntelliJ and Maven).
Manually clang-format the C++ code. Clean up some things to match Arrow convention and remove some unused declarations.
Extends the DictionaryProvider interface. This is a potentially breaking change; we could make the method default (and raise an exception) instead.

github-actions · 2022-06-29T14:51:41Z

https://issues.apache.org/jira/browse/ARROW-16913

pitrou · 2022-06-29T15:17:18Z

java/c/src/main/cpp/jni_wrapper.cc

+    ThrowPendingException(message);
+  }
+  jclass global_class = (jclass)env->NewGlobalRef(local_class);
+  if (!local_class) {


Is this a mistake?

Suggested change

if (!local_class) {

if (!global_class) {

pitrou · 2022-06-29T15:20:32Z

java/c/src/main/cpp/jni_wrapper.cc

+  const int err_code = env->CallIntMethod(private_data->j_private_data_,
+                                          kPrivateDataGetSchemaMethod, out_addr);
+  if (env->ExceptionCheck()) {
+    env->ExceptionDescribe();


If there's an exception, should it perhaps participate in last_error_?

Normally the JNI side sets the last error, the check here is just a last-resort safeguard. I suppose this can be refactored though: copy the Java-side error to the C++ side after get_next/get_stream, and get_last_error only has to return the C++-side error; then get_next/get_stream can also update last_error_ if it ends up catching a stray error.

pitrou · 2022-06-29T15:23:41Z

java/c/src/main/cpp/jni_wrapper.cc

+  if (env->ExceptionCheck()) {
+    env->ExceptionDescribe();
+    env->ExceptionClear();
+    ThrowPendingException("Error calling close of private data");


Is this right? The release callback could be called from any context, such as a Python thread or R interpreter. In those contexts, a C++ exception would probably crash the process (or silently exit the thread)?

ah, you're right. The existing handler has this issue too. I'll remove the throw. (Actually here I suppose we should do our best to free resources in C++/Java regardless.)

pitrou · 2022-06-29T15:25:59Z

java/c/src/main/cpp/jni_wrapper.cc

+  JNIEnvGuard guard(private_data->vm_);
+  JNIEnv* env = guard.env();
+
+  const long out_addr = static_cast<long>(reinterpret_cast<uintptr_t>(out));


I suppose this doesn't work on 64-bit Windows? long is 32 bits there...

Also, according to the JNI spec, a jlong is always 64 bits, so perhaps we should use jlong or simply int64_t here?

Good catch, thanks.

pitrou · 2022-06-29T15:51:29Z

java/c/src/main/java/org/apache/arrow/c/Data.java

+   * @param stream C stream interface struct to import.
+   * @return Imported reader
+   */
+  public static ArrowReader importStream(BufferAllocator allocator, ArrowArrayStream stream) {


Is there a reason for the naming discrepancy (importStream vs. exportArrayStream)?

pitrou · 2022-06-29T15:53:46Z

java/c/src/main/java/org/apache/arrow/c/ArrayStreamExporter.java

+  static class ExportedArrayStreamPrivateData implements PrivateData {
+    final BufferAllocator allocator;
+    final ArrowReader reader;
+    int nextDictionary;


This member doesn't seem used anymore, or am I missing something?

Ah it's not used. I missed this when backing out a change.

pitrou

+1. For the record, did you try to use this to communicate with e.g. PyArrow?

pitrou · 2022-06-30T12:16:59Z

@lwhite1 Would you like to take a look?

lidavidm · 2022-06-30T12:24:12Z

For the record, did you try to use this to communicate with e.g. PyArrow?

I have not yet, I need to give this a try: https://arrow.apache.org/docs/dev/python/integration/python_java.html#java-to-python-communication-using-the-c-data-interface

and actually, I'll extend the doc page there as well.

lidavidm · 2022-06-30T12:24:49Z

…wow, whatever GitHub did to their UI is rather frustrating.

lidavidm · 2022-06-30T15:07:21Z

Hmm, there's a possible minor bug between PyArrow/C++/Java: Python can keep a reference to the reader until interpreter shutdown (at which point the JVM has been shut down), and then collects the reader. This frees the shared_ptr<RecordBatchReader>; the implementation is ArrayStreamBatchReader whose destructor unconditionally calls ArrowArrayStreamRelease. (It doesn't implement Close() since this was just added). This calls into the JNI code, which tries to attach the JVM, fails, throws an uncaught exception, and dies.

Changes needed:

Implement Close() properly so we can explicitly free
Don't unconditionally call ArrowArrayStreamRelease
In the JNI side, fail gracefully if we can't attach the JVM (warn instead of abort?)

pitrou · 2022-06-30T15:09:10Z

Python can keep a reference to the reader until interpreter shutdown (at which point the JVM has been shut down), and then collects the reader

Can Python perhaps release that reference once close() is called?

lidavidm · 2022-06-30T15:10:49Z

Well, the Python-side reference is the Python reader object itself. But close() should be wired up to call the new RecordBatchReader::Close() so we can at least explicitly call the release callback at a suitable time.

pitrou · 2022-06-30T15:10:53Z

Though the Java improvements are welcome as well. We should probably try to do both.

lidavidm · 2022-06-30T15:10:57Z

…

java/c/src/main/cpp/jni_wrapper.cc

pitrou

Just a question, this is great otherwise.

pitrou · 2022-06-30T15:48:14Z

@amol- Do you want to take a look at the doc additions?

Implements ArrowArrayStream for Java. The equivalent Java-side interface chosen is ArrowReader. Also: - Fixes a couple of JDK9 compatibility issues I ran into. I _think_ these will not normally affect people except during development (I think because I was mixing IntelliJ and Maven). - Manually clang-format the C++ code. Clean up some things to match Arrow convention and remove some unused declarations. - Extends the DictionaryProvider interface. This is a potentially breaking change; we could make the method default (and raise an exception) instead. Authored-by: David Li <li.davidm96@gmail.com> Signed-off-by: Alessandro Molina <amol@turbogears.org>

ARROW-16913: [Java] Implement ArrowArrayStream

22cc85f

github-actions bot added the Component: Java label Jun 29, 2022

pitrou reviewed Jun 29, 2022

View reviewed changes

Address feedback

8090dbf

pitrou reviewed Jun 30, 2022

View reviewed changes

lidavidm closed this Jun 30, 2022

lidavidm reopened this Jun 30, 2022

Extend Python/Java C Data integration tests

3e81c76

lidavidm closed this Jun 30, 2022

lidavidm reopened this Jun 30, 2022

pitrou reopened this Jun 30, 2022

lidavidm added 2 commits June 30, 2022 11:31

Safeguard against JVM shutdown

dad5d3c

Update docs with example

fec64cb

github-actions bot added Component: Documentation Component: C++ Component: Python labels Jun 30, 2022

pitrou reviewed Jun 30, 2022

View reviewed changes

java/c/src/main/cpp/jni_wrapper.cc Show resolved Hide resolved

pitrou approved these changes Jun 30, 2022

View reviewed changes

amol- merged commit cd3c6ea into apache:master Jul 5, 2022

lidavidm deleted the arrow-16913 branch July 5, 2022 14:00

jonathanswenson mentioned this pull request Jul 9, 2022

Add mechanism to ingest Arrow data via the JDBC driver. duckdb/duckdb#4071

Closed

6 tasks

lwhite1 mentioned this pull request Aug 2, 2022

Version 9.0.0 release blog post apache/arrow-site#227

Merged

asfimport mentioned this pull request Jul 5, 2022

[Java] Implement ArrowArrayStream/C Stream Interface #32233

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-16913: [Java] Implement ArrowArrayStream #13465

ARROW-16913: [Java] Implement ArrowArrayStream #13465

lidavidm commented Jun 29, 2022

github-actions bot commented Jun 29, 2022

pitrou Jun 29, 2022

pitrou Jun 29, 2022

lidavidm Jun 29, 2022

pitrou Jun 29, 2022

lidavidm Jun 29, 2022

pitrou Jun 29, 2022

pitrou Jun 29, 2022

lidavidm Jun 29, 2022

pitrou Jun 29, 2022

pitrou Jun 29, 2022

lidavidm Jun 29, 2022

pitrou left a comment

pitrou commented Jun 30, 2022

lidavidm commented Jun 30, 2022

lidavidm commented Jun 30, 2022

lidavidm commented Jun 30, 2022

pitrou commented Jun 30, 2022

lidavidm commented Jun 30, 2022

pitrou commented Jun 30, 2022

lidavidm commented Jun 30, 2022

pitrou left a comment

pitrou commented Jun 30, 2022

ARROW-16913: [Java] Implement ArrowArrayStream #13465

ARROW-16913: [Java] Implement ArrowArrayStream #13465

Conversation

lidavidm commented Jun 29, 2022

github-actions bot commented Jun 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pitrou left a comment

Choose a reason for hiding this comment

pitrou commented Jun 30, 2022

lidavidm commented Jun 30, 2022

lidavidm commented Jun 30, 2022

lidavidm commented Jun 30, 2022

pitrou commented Jun 30, 2022

lidavidm commented Jun 30, 2022

pitrou commented Jun 30, 2022

lidavidm commented Jun 30, 2022

pitrou left a comment

Choose a reason for hiding this comment

pitrou commented Jun 30, 2022