[Ruby] Add support for loading table by Arrow Dataset #18794

asfimport · 2021-08-20T20:46:38Z

Reporter: Kouhei Sutou / @kou
Assignee: Kouhei Sutou / @kou

Related issues:

[Ruby] Add support for S3FileSystem (duplicates)

PRs and other links:

GitHub Pull Request #10970

_{Note: This issue was originally created as ARROW-13687. Please see the migration documentation for further details.}

asfimport · 2021-08-21T20:50:45Z

Kouhei Sutou / @kou:
Issue resolved by pull request 10970
#10970

asfimport · 2021-11-03T21:42:45Z

Kanstantsin Ilchanka / @simpl1g:
@kou can you please provide example how to read file/folder from s3, I can't find anything in Ruby code related to S3, I would like to add it to examples

Also, doing this causes segfault on version 5.0.0

Arrow::S3FileSystem.new

asfimport · 2021-11-04T02:09:12Z

Kouhei Sutou / @kou:
https://slide.rabbit-shocker.org/authors/kou/rubykaigi-takeout-2021/?page=32


Arrow::Table.load("s3://bucket/path.arrow")

asfimport · 2021-11-04T02:14:54Z

Kouhei Sutou / @kou:
If you want to test this with MinIO, you need to add endpoint_override=http://127.0.0.1:... query string such as Arrow::Table.load("s3://bucket/path.arrow?endpoint_override=http://...").

asfimport · 2021-11-04T08:42:56Z

Kanstantsin Ilchanka / @simpl1g:
I tried to run it, but I have error.

ruby-2.7.4/gems/gobject-introspection-3.4.9/lib/gobject-introspection/loader.rb:616:in `invoke': [file-system-dataset-factory][set-file-system-uri]: NotImplemented: Got S3 URI but Arrow compiled without S3 support (Arrow::Error::NotImplemented)

asfimport · 2021-11-04T08:55:13Z

Kouhei Sutou / @kou:
You need S3 support enabled Apache Arrow C++ as the error message says.

asfimport · 2021-11-04T09:25:45Z

Kanstantsin Ilchanka / @simpl1g:
So I can't use HomeBrew version? and should build from sources? As far as I understood I need to have aws-sdk-cpp available and ARROW_S3 env variable set, is it correct? Or maybe there is some docs that I can read?

asfimport · 2021-11-04T09:53:32Z

Kouhei Sutou / @kou:
You can use Homebrew version once you send a pull request tohttps://github.com/homebrew/homebrew-core for enabling S3 support.

The following patch will work:


diff --git a/Formula/apache-arrow.rb b/Formula/apache-arrow.rb
index 4983e512380..14870af7b7f 100644
--- a/Formula/apache-arrow.rb
+++ b/Formula/apache-arrow.rb
@@ -15,6 +15,7 @@ class ApacheArrow < Formula
     sha256 cellar: :any_skip_relocation, x86_64_linux:  "ca1305cb5335250312a597d489717bb8066ef1ac5343ab9dfd59349b3eadfdf5"
   end
 
+  depends_on "aws-sdk-cpp"
   depends_on "boost" => :build
   depends_on "cmake" => :build
   depends_on "llvm" => :build
@@ -55,6 +56,7 @@ class ApacheArrow < Formula
       -DARROW_PLASMA=ON
       -DARROW_PROTOBUF_USE_SHARED=ON
       -DARROW_PYTHON=ON
+      -DARROW_S3=ON
       -DARROW_WITH_BZ2=ON
       -DARROW_WITH_ZLIB=ON
       -DARROW_WITH_ZSTD=ON

asfimport · 2021-11-04T09:54:08Z

Kouhei Sutou / @kou:
See also: https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request

asfimport · 2021-11-04T19:34:02Z

Kanstantsin Ilchanka / @simpl1g:
I build apache-arrow locally and it works with S3, thanks!. However is it possible to update brew formulae without updating it's version, is it safe to use --force?

Questions:

How can I pass access_token/secret_key so that I can access private files?
How can I read/write by partitions?

Also I did some testing, here are problems that I found:
Speed is very slow. It took almost 1 hour to download 400 Mb file through S3 compared to 15 seconds via usual Net::HTTP. For small files diff is not so huge. Here is benchmark with small file. Maybe it is somehow connected that I tested it with custom brew build?
{code:java}
require 'arrow-dataset'
require 'net/http'
require 'benchmark/ips'

s3_uri = URI("s3://simpl1g-example/correct.csv")
http_uri = URI("https://simpl1g-example.s3.eu-central-1.amazonaws.com/correct.csv")
Benchmark.ips do |x|
x.report('S3') { Arrow::Table.load(s3_uri) }
x.report('Http') { Arrow::Table.load(Arrow::Buffer.new(Net::HTTP.get(http_uri)), format: :csv) }
x.compare!
end

Comparison:
```
            Http:        9.6 i/s
```

              S3:        4.9 i/s - 1.97x  slower

{code}

Not sure if it is real problem, but I can't cancel downloading big objects, process stucks until download finished, now it takes hours, I guess because of slow read, I can only do kill -9 for process.
```
Arrow::Table.load(URI("s3://big-parquet-file.parquet"))
```
Doing S3 call doesn't work the same as for local file. I have TSV file with .csv extension. Parsing local file works fine. On S3 it fails
{code:java}

Works fine
Arrow::Table.load("file.csv", delimiter: "\t")
Arrow::Table.load("file.csv", format: :tsv)

{code:java}
Arrow::Table.load(URI("s3://simpl1g-example/file.csv"), delimiter: "\t")
gobject-introspection-3.4.9/lib/gobject-introspection/loader.rb:616:in `invoke': [file-system-dataset-factory][finish]: Invalid: Error creating dataset. Could not read schema from 'simpl1g-example/file.csv': Could not open CSV input source 'simpl1g-example/file.csv': Invalid: CSV parse error: Row #2: Expected 1 columns, got 2: 6	18	iPhone9,2	1635840547. Is this a 'csv' file? (Arrow::Error::Invalid)

Arrow::Table.load(URI("s3://simpl1g-example/file.csv"), format: :tsv)
Traceback (most recent call last):
	23: from bin/console:5:in `<main>'
	 6: from (irb):22:in `<main>'
	 5: from /Users/k.ilcenko/.rvm/gems/ruby-2.7.4/gems/red-arrow-6.0.0/lib/arrow/table.rb:29:in `load'
	 4: from /Users/k.ilcenko/.rvm/gems/ruby-2.7.4/gems/red-arrow-6.0.0/lib/arrow/table-loader.rb:24:in `load'
	 3: from /Users/k.ilcenko/.rvm/gems/ruby-2.7.4/gems/red-arrow-6.0.0/lib/arrow/table-loader.rb:56:in `load'
	 2: from /Users/k.ilcenko/.rvm/gems/ruby-2.7.4/gems/red-arrow-dataset-6.0.0/lib/arrow-dataset/arrow-table-loadable.rb:35:in `load_from_uri'
	 1: from /Users/k.ilcenko/.rvm/gems/ruby-2.7.4/gems/red-arrow-dataset-6.0.0/lib/arrow-dataset/arrow-table-loadable.rb:39:in `internal_load_from_uri'
/Users/k.ilcenko/.rvm/gems/ruby-2.7.4/gems/red-arrow-dataset-6.0.0/lib/arrow-dataset/file-format.rb:39:in `resolve': undefined method `[]' for nil:NilClass (NoMethodError)

asfimport · 2021-11-04T20:48:03Z

Kouhei Sutou / @kou:
Homebrew formula support revision for updating in the same version:


diff --git a/Formula/apache-arrow.rb b/Formula/apache-arrow.rb
index 4983e512380..901692d5a35 100644
--- a/Formula/apache-arrow.rb
+++ b/Formula/apache-arrow.rb
@@ -5,6 +5,7 @@ class ApacheArrow < Formula
   mirror "https://archive.apache.org/dist/arrow/arrow-6.0.0/apache-arrow-6.0.0.tar.gz"
   sha256 "69d268f9e82d3ebef595ad1bdc83d4cb02b20c181946a68631f6645d7c1f7a90"
   license "Apache-2.0"
+  revision 1
   head "https://github.com/apache/arrow.git", branch: "master"
 
   bottle do

Could you send the changes you tried to Homebrew?

asfimport · 2021-11-04T21:22:21Z

Kanstantsin Ilchanka / @simpl1g:
@kou done - Homebrew/homebrew-core#88797

asfimport · 2021-11-04T22:01:06Z

Kouhei Sutou / @kou:
Thanks.

How can I pass access_token/secret_key so that I can access private files?

You can use general user/password syntax for URI: s3://#\{USER\}:#\{PASSWORD\}\@backet/path

asfimport · 2021-11-04T22:11:15Z

Kouhei Sutou / @kou:

How can I read/write by partitions?

Convenient APIs (Arrow::Table#save/Arrow::Table.load) for them aren't implemented yet. Could you open a Jira issue for this?

asfimport · 2021-11-05T10:03:38Z

Kanstantsin Ilchanka / @simpl1g:

Convenient APIs (Arrow::Table#save/Arrow::Table.load) for them aren't implemented yet. Could you open a Jira issue for this?

https://issues.apache.org/jira/browse/ARROW-14604

asfimport · 2021-11-05T10:03:48Z

Kanstantsin Ilchanka / @simpl1g:

You can use general user/password syntax for URI: s3://#\{USER}:#\{PASSWORD}@backet/path

I'm not sure that this is valid for S3, they require different authentication

I expect something like this

arrow/python/pyarrow/_s3fs.pyx

Line 57 in 0dfe592

cdef class S3FileSystem(FileSystem):

s3_fs = Arrow::S3FileSystem.new(access_key: 'key', secret_key: 'key', region: 'region')
table = Arrow::Table.load(URI("s3://backet/path"), filesystem: s3_fs)

asfimport · 2021-11-05T22:08:25Z

Kouhei Sutou / @kou:
Did you try it?

S3FileSystem uses user/password information in URI if they exist:

arrow/cpp/src/arrow/filesystem/s3fs.cc

Lines 330 to 335 in 0dfe592

    
           const auto username = uri.username(); 
        
           if (!username.empty()) { 
        
             options.ConfigureAccessKey(username, uri.password()); 
        
           } else { 
        
             options.ConfigureDefaultCredentials(); 
        
           }

asfimport · 2021-11-06T10:04:33Z

Kanstantsin Ilchanka / @simpl1g:
Thanks, your code helped. It should be s3://#{ACCESS_KEY}:#{SECRET_KEY}@backet/path

However secret key can contain / for example and it will be an error, I have such case in one of production buckets

URI("s3://acces_key:secret/key@bucket/file.csv")
# rfc3986_parser.rb:67:in `split': bad URI(is not URI?)

asfimport · 2021-11-06T21:27:23Z

Kouhei Sutou / @kou:
You can use percent-encoding for it. You can use CGI.escape in cgi/util for it: URI("s3://#\{CGI.escape("access_key")\}:#\{CGI.escape("secret/key")\}\@bucket/file.csv")

asfimport · 2021-11-06T22:31:10Z

Kouhei Sutou / @kou:

Speed is very slow.

If you use "s3://..." on local, initialization process take a long time because of timeout. AWS_EC2_METADATA_DISABLED=true environment variable will help you.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ruby] Add support for loading table by Arrow Dataset #18794

[Ruby] Add support for loading table by Arrow Dataset #18794

asfimport commented Aug 20, 2021 •

edited

asfimport commented Aug 21, 2021

asfimport commented Nov 3, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 5, 2021

asfimport commented Nov 5, 2021

asfimport commented Nov 5, 2021

asfimport commented Nov 6, 2021

asfimport commented Nov 6, 2021

asfimport commented Nov 6, 2021

asfimport commented Nov 8, 2021

asfimport commented Nov 8, 2021

[Ruby] Add support for loading table by Arrow Dataset #18794

[Ruby] Add support for loading table by Arrow Dataset #18794

Comments

asfimport commented Aug 20, 2021 • edited

Related issues:

PRs and other links:

asfimport commented Aug 21, 2021

asfimport commented Nov 3, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 4, 2021

asfimport commented Nov 5, 2021

asfimport commented Nov 5, 2021

asfimport commented Nov 5, 2021

asfimport commented Nov 6, 2021

asfimport commented Nov 6, 2021

asfimport commented Nov 6, 2021

asfimport commented Nov 8, 2021

asfimport commented Nov 8, 2021

asfimport commented Aug 20, 2021 •

edited