[Java] Error loading native libraries on Windows

[yspjack]

Describe the bug, including details regarding any error messages, version, and platform.

org.apache.arrow.dataset.jni.JniLoader cannot load "x86_64\arrow_dataset_jni.dll" from JAR file("arrow-dataset-11.0.0.jar" from Maven central repository) and throws an exception. However, this example works fine on Arrow 10.0.1.

Stack trace

java.lang.IllegalStateException: error loading native libraries: java.io.FileNotFoundException: x86_64\arrow_dataset_jni.dll
	at org.apache.arrow.dataset.jni.JniLoader.load(JniLoader.java:94)
	at org.apache.arrow.dataset.jni.JniLoader.loadRemaining(JniLoader.java:74)
	at org.apache.arrow.dataset.jni.JniLoader.ensureLoaded(JniLoader.java:61)
	at org.apache.arrow.dataset.jni.NativeMemoryPool.getDefault(NativeMemoryPool.java:34)
	at QuickStart.main(QuickStart.java:23)

QuickStart.java:

// Reading a Parquet file using FileSystemDatasetFactory
import org.apache.arrow.dataset.file.FileFormat;
import org.apache.arrow.dataset.file.FileSystemDatasetFactory;
import org.apache.arrow.dataset.jni.NativeMemoryPool;
import org.apache.arrow.dataset.scanner.ScanOptions;
import org.apache.arrow.dataset.scanner.Scanner;
import org.apache.arrow.dataset.source.Dataset;
import org.apache.arrow.dataset.source.DatasetFactory;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.vector.ipc.ArrowReader;

import java.nio.file.Path;

public class QuickStart {
    public static void main(String[] args) {
        String uri = Path.of("data1.parquet").toUri().toString();
        System.out.println(uri);
        ScanOptions options = new ScanOptions(32768);
        try (
                BufferAllocator allocator = new RootAllocator();
                DatasetFactory datasetFactory = new FileSystemDatasetFactory(allocator, NativeMemoryPool.getDefault(), FileFormat.PARQUET, uri);
                Dataset dataset = datasetFactory.finish();
                Scanner scanner = dataset.newScan(options);
                ArrowReader reader = scanner.scanBatches()
        ) {
            while (reader.loadNextBatch()) {
                try (VectorSchemaRoot root = reader.getVectorSchemaRoot()) {
                    System.out.print(root.contentToTSVString());
                }
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Version: arrow 11.0.0

Dependencies:

• org.apache.arrow:arrow-vector:11.0.0
• org.apache.arrow:arrow-memory:11.0.0
• org.apache.arrow:arrow-memory-netty:11.0.0
• org.apache.arrow:arrow-dataset:11.0.0

Platform:

• OS: Windows 11 x64
• JDK version: 17

Component(s)

Java

[kou]

Could you confirm that your arrow-dataset-11.0.0.jar includes x86_64\arrow_dataset_jni.dll?

@davisusanibar Could you try reproducing this problem?

[yspjack]

arrow-dataset-11.0.0.jar is downloaded from Maven central repository, and it includes x86_64/arrow_dataset_jni.dll instead of x86_64\arrow_dataset_jni.dll. I guess that the backslash in x86_64\arrow_dataset_jni.dll causes this problem.

[davisusanibar]

Could you confirm that your arrow-dataset-11.0.0.jar includes x86_64\arrow_dataset_jni.dll?

@davisusanibar Could you try reproducing this problem?

The Dataset jar contains DLL needed, main problem will be the way about how the libraryToLoad path is created: https://github.com/apache/arrow/blob/8776295ffb5dc635f2dfe016ce6f4e9847ae1131/java/dataset/src/main/java/org/apache/arrow/dataset/jni/JniLoader.java#L80:L81

final String libraryToLoad =
        getNormalizedArch() + File.separator + System.mapLibraryName(name);

Replace File.separator with /

Let me test and send the fix then.

[ssvendsen]

I wanted to try the support for writing parquet files via the DataSet API (as indicated in the v11 release notes), but ran into this problem. It would be great to see a fix.