[Java] while using s3 FileSystemDatasetFactory getting this exception

[febinct]

Describe the bug, including details regarding any error messages, version, and platform.

/Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/arrow/cpp/src/arrow/filesystem/s3fs.cc:2598: arrow::fs::FinalizeS3 was not called even though S3 was initialized. This could lead to a segmentation fault at exit

i think the java client is not closing the s3 client gracefully because of that memory leak happening

Component(s)

Java

[westonpace]

CC @davisusanibar @lidavidm (note that this warning was newly added to the S3 filesystem in the previous release so it is very possible the Java implementation has never been calling finalize)

[davisusanibar]

CC @davisusanibar @lidavidm (note that this warning was newly added to the S3 filesystem in the previous release so it is very possible the Java implementation has never been calling finalize)

Just able to reproduce this warning with:

import org.apache.arrow.dataset.file.FileFormat;
import org.apache.arrow.dataset.file.FileSystemDatasetFactory;
import org.apache.arrow.dataset.jni.NativeMemoryPool;
import org.apache.arrow.dataset.scanner.ScanOptions;
import org.apache.arrow.dataset.scanner.Scanner;
import org.apache.arrow.dataset.source.Dataset;
import org.apache.arrow.dataset.source.DatasetFactory;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.ipc.ArrowReader;
import org.apache.arrow.vector.types.pojo.Schema;

public class DatasetModule {
    public static void main(String[] args) {
        String uri = "s3://voltrondata-labs-datasets/nyc-taxi-tiny/year=2022/month=2/part-0.parquet"; // AWS S3
        // String uri = "hdfs://{hdfs_host}:{port}/nyc-taxi-tiny/year=2022/month=2/part-0.parquet"; // HDFS
        // String uri = "gs://voltrondata-labs-datasets/nyc-taxi-tiny/year=2022/month=2/part-0.parquet"; // Google Cloud Storage
        ScanOptions options = new ScanOptions(/*batchSize*/ 32768);
        try (
            BufferAllocator allocator = new RootAllocator();
            DatasetFactory datasetFactory = new FileSystemDatasetFactory(allocator, NativeMemoryPool.getDefault(), FileFormat.PARQUET, uri);
            Dataset dataset = datasetFactory.finish();
            Scanner scanner = dataset.newScan(options);
            ArrowReader reader = scanner.scanBatches()
        ) {
            Schema schema = scanner.schema();
            System.out.println(schema);
            while (reader.loadNextBatch()) {
                System.out.println("RowCount: " + reader.getVectorSchemaRoot().getRowCount());
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Output messages:

RowCount: 2979
/Users/runner/work/crossbow/crossbow/arrow/cpp/src/arrow/filesystem/s3fs.cc:2598:  arrow::fs::FinalizeS3 was not called even though S3 was initialized.  This could lead to a segmentation fault at exit

Next step:

1. Review reason of error messages
2. Add Arrow Java cookbook to cover S3 integration

[febinsathar]

while running s3 file reader using java after for sometime the machine is crash looping due to this AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 43, A libcurl function was given a bad argument
this generally happens when aws s3 connection pool are not closed AFAIK

[AcidChristLab]

A simple solution to fix this error:

pip install --upgrade --force-reinstall pyarrow==11.0.0

[febinct]

this is in java not python

[AcidChristLab]

this is in java not python

Are you not using the pyarrow package for java?

[mapleFU]

I guess the problem in introduced in #33858 . The description contains:

BREAKING CHANGE: S3 can only be initialized and finalized once.
BREAKING CHANGE: S3 (the AWS SDK) will not be finalized until after all CPU & I/O threads are finished.

A helpful way is calling FinalizeS3 when exit. This patch is in arrow-12.0

However, I guess this patch might fixed it: #36442 . Maybe you can confirm it latter?

[kou]

AFAIK, there is no pyarrow package for Java.

Java's dataset API uses JNI. It means that Java's dataset API calls C++ implementation directly. (It doesn't use Python.)

We need to implement a Java binding for arrow::fs::FinalizeS3() and users need to call it explicitly.
Or we may be able to do it implicitly if Java provides an atexit() like hook.

[lidavidm]

Java has Runtime#addShutdownHook

[lidavidm]

@davisusanibar @danepitkin any interest in this?

[danepitkin]

Yes! We can take this on. Thanks for the ping.