Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-1325: [R] Initial R package that builds against the arrow C++ library #2489

Closed
wants to merge 6 commits into from

Conversation

@romainfrancois
Copy link
Contributor

commented Aug 28, 2018

This is very minimal in functionality, it just gives a simple R package that calls a function from the arrow C++ library.

@wesm wesm changed the title Initial R package that builds against the arrow C++ library ARROW-1325: [R] Initial R package that builds against the arrow C++ library Aug 28, 2018

@codecov-io

This comment has been minimized.

Copy link

commented Aug 28, 2018

Codecov Report

Merging #2489 into master will decrease coverage by 0.11%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2489      +/-   ##
==========================================
- Coverage   86.33%   86.22%   -0.12%     
==========================================
  Files         308      308              
  Lines       47120    47120              
==========================================
- Hits        40682    40629      -53     
- Misses       6366     6417      +51     
- Partials       72       74       +2
Impacted Files Coverage Δ
go/arrow/math/int64_avx2_amd64.go 0% <0%> (-100%) ⬇️
go/arrow/math/float64_avx2_amd64.go 0% <0%> (-100%) ⬇️
go/arrow/memory/memory_avx2_amd64.go 0% <0%> (-100%) ⬇️
go/arrow/math/uint64_avx2_amd64.go 0% <0%> (-100%) ⬇️
go/arrow/math/uint64_amd64.go 33.33% <0%> (-33.34%) ⬇️
go/arrow/math/int64_amd64.go 33.33% <0%> (-33.34%) ⬇️
go/arrow/math/float64_amd64.go 33.33% <0%> (-33.34%) ⬇️
go/arrow/math/math_amd64.go 31.57% <0%> (-31.58%) ⬇️
go/arrow/memory/memory_amd64.go 28.57% <0%> (-28.58%) ⬇️
rust/src/array.rs 84.03% <0%> (ø) ⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 21ef18b...9e3ffb4. Read the comment docs.

@romainfrancois romainfrancois force-pushed the romainfrancois:r-bootstrap branch from ca9d7f0 to a1a5e7c Sep 6, 2018

@wesm
Copy link
Member

left a comment

Left some comments. Let's try to merge this ASAP for the sake of development productivity

@@ -31,6 +31,7 @@ export ARROW_JS_DIR=${TRAVIS_BUILD_DIR}/js
export ARROW_INTEGRATION_DIR=$TRAVIS_BUILD_DIR/integration
export ARROW_RUBY_DIR=$TRAVIS_BUILD_DIR/ruby
export ARROW_RUST_DIR=${TRAVIS_BUILD_DIR}/rust
export ARROW_R_DIR=${TRAVIS_BUILD_DIR}/r

This comment has been minimized.

Copy link
@wesm

wesm Sep 6, 2018

Member

Any strong opinions about "R/" vs. "r/" I am OK with either

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

Lower case is fine, that's what other languages do

This comment has been minimized.

Copy link
@jimhester

jimhester Sep 7, 2018

I think lower case for the outer subdirectory is fine, but the R code has to be in an uppercase R. So as long as it doesn't look too weird to have r/R/*.

Title: Bindings to 'Apache' 'Arrow'
Version: 0.0.0.9000
Authors@R: c(
person("Romain", "François", email = "romain@rstudio.com", role = c("aut", "cre"))

This comment has been minimized.

Copy link
@wesm

wesm Sep 6, 2018

Member

Put "Apache Arrow" here and "dev@arrow.apache.org"

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

@@ -0,0 +1,20 @@
Package: arrow
Title: Bindings to 'Apache' 'Arrow'

This comment has been minimized.

Copy link
@wesm

wesm Sep 6, 2018

Member

This package will be a bit more than bindings. Perhaps "R integration with Apache Arrow"?

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

@@ -0,0 +1,194 @@
Apache License

This comment has been minimized.

Copy link
@wesm

wesm Sep 6, 2018

Member

Does this license file need to be duplicated? Can the one from the top level be used?

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

probably not, deleted it

the same “printed page” as the copyright notice for easier identification within
third-party archives.

Copyright 2018 Romain François

This comment has been minimized.

Copy link
@wesm

wesm Sep 6, 2018

Member

Use the project license

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

Deleted the license file for now.


## Installation

First install a release build of the C++ bindings to arrow.

This comment has been minimized.

Copy link
@wesm

wesm Sep 6, 2018

Member

Does it have to be a release build?

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

Not sure, I just stole this from @jimhester. I guess it could be other types of builds as long as pkg-config finds the relevant information in the ./configure script

r/README.md Outdated

# arrow

Low level bindings to the C++ Apache Arrow library

This comment has been minimized.

Copy link
@wesm

wesm Sep 6, 2018

Member

Maybe "R interface to the ..." or "R integration with...", since the extent of the work involved is a bit more than bindings

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

Sure. Made it R integration with Apache Arrow. for now, we can always revisit this later when we can better define what this is.

PKG_CPPFLAGS=@cflags@
PKG_CXXFLAGS=$(C_VISIBILITY)
CXX_STD=CXX11
PKG_LIBS=@libs@ -Wl,-rpath,/usr/local/lib

This comment has been minimized.

Copy link
@wesm

wesm Sep 6, 2018

Member

Hard-coded rpath?

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

That's another thing I borrowed from @jimhester who probably had a reason in this commit of rarrow: jimhester/rarrow@ac3aafc

Can maybe pkg-config give just /usr/local/lib somehow ?

This comment has been minimized.

Copy link
@jimhester

jimhester Sep 7, 2018

I don't recall exactly why the issue here did you try linking without this linker option entirely?

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

That works locally, I just sent a travis job to see what happens 🤞

This comment has been minimized.

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

🤔 maybe not, getting this now locally:

** testing if installed package can be loaded
Error: package or namespace load failed for ‘arrow’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/Library/Frameworks/R.framework/Versions/3.5/Resources/library/arrow/libs/arrow.so':
  dlopen(/Library/Frameworks/R.framework/Versions/3.5/Resources/library/arrow/libs/arrow.so, 6): Library not loaded: @rpath/libarrow.10.dylib
  Referenced from: /Library/Frameworks/R.framework/Versions/3.5/Resources/library/arrow/libs/arrow.so
  Reason: image not found
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/arrow’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/arrow’

Exited with status 1.

@jimhester is this what you had previously in rarrow perhaps ?

This comment has been minimized.

Copy link
@jimhester

jimhester Sep 7, 2018

What does otool -L arrow.so give you?

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor
romain@purrplex ~/git/apache/arrow/r/src $ otool -L arrow.so
arrow.so:
	arrow.so (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libarrow.10.dylib (compatibility version 10.0.0, current version 10.0.0)
	/Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libR.dylib (compatibility version 3.5.0, current version 3.5.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1454.90.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
// under the License.

#include <Rcpp.h>
#include "rrrow_types.h"

This comment has been minimized.

Copy link
@wesm

wesm Sep 6, 2018

Member

r_arrow_types.h?

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

#pragma once

#include <Rcpp.h>
#undef Free

This comment has been minimized.

Copy link
@wesm

wesm Sep 6, 2018

Member

Does this cause a conflict?

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

Yes, R defines Free here: https://github.com/wch/r-source/blob/trunk/src/include/R_ext/RS.h#L74

==> R CMD INSTALL --preclean --no-multiarch --with-keep.source r

* installing to library ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library’
* installing *source* package ‘arrow’ ...
Found pkg-config cflags and libs!
PKG_CFLAGS=-DNDEBUG -I/usr/local/include
PKG_LIBS=-L/usr/local/lib -larrow
** libs
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DNDEBUG -I/usr/local/include -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I/usr/local/include   -fPIC  -Wno-unused-result -O3 -c RcppExports.cpp -o RcppExports.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DNDEBUG -I/usr/local/include -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I/usr/local/include   -fPIC  -Wno-unused-result -O3 -c hello.cpp -o hello.o
In file included from hello.cpp:19:
In file included from ./r_arrow_types.h:22:
In file included from /usr/local/include/arrow/api.h:23:
In file included from /usr/local/include/arrow/array.h:28:
In file included from /usr/local/include/arrow/buffer.h:28:
/usr/local/include/arrow/memory_pool.h:55:38: error: too many arguments provided to function-like macro invocation
  virtual void Free(uint8_t* buffer, int64_t size) = 0;
                                     ^
/Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:74:9: note: macro 'Free' defined here
#define Free(p)        (R_chk_free( (void *)(p) ), (p) = NULL)
        ^
In file included from hello.cpp:19:
In file included from ./r_arrow_types.h:22:
In file included from /usr/local/include/arrow/api.h:23:
In file included from /usr/local/include/arrow/array.h:28:
In file included from /usr/local/include/arrow/buffer.h:28:
/usr/local/include/arrow/memory_pool.h:55:3: error: 'virtual' can only appear on non-static member functions
  virtual void Free(uint8_t* buffer, int64_t size) = 0;
  ^
/usr/local/include/arrow/memory_pool.h:55:16: error: field has incomplete type 'void'
  virtual void Free(uint8_t* buffer, int64_t size) = 0;
               ^
/usr/local/include/arrow/memory_pool.h:79:30: error: too many arguments provided to function-like macro invocation
  void Free(uint8_t* buffer, int64_t size) override;
                             ^
/Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:74:9: note: macro 'Free' defined here
#define Free(p)        (R_chk_free( (void *)(p) ), (p) = NULL)
        ^
In file included from hello.cpp:19:
In file included from ./r_arrow_types.h:22:
In file included from /usr/local/include/arrow/api.h:23:
In file included from /usr/local/include/arrow/array.h:28:
In file included from /usr/local/include/arrow/buffer.h:28:
/usr/local/include/arrow/memory_pool.h:79:8: error: field has incomplete type 'void'
  void Free(uint8_t* buffer, int64_t size) override;
       ^
/usr/local/include/arrow/memory_pool.h:101:30: error: too many arguments provided to function-like macro invocation
  void Free(uint8_t* buffer, int64_t size) override;
                             ^
/Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:74:9: note: macro 'Free' defined here
#define Free(p)        (R_chk_free( (void *)(p) ), (p) = NULL)
        ^
In file included from hello.cpp:19:
In file included from ./r_arrow_types.h:22:
In file included from /usr/local/include/arrow/api.h:23:
In file included from /usr/local/include/arrow/array.h:28:
In file included from /usr/local/include/arrow/buffer.h:28:
/usr/local/include/arrow/memory_pool.h:101:8: error: field has incomplete type 'void'
  void Free(uint8_t* buffer, int64_t size) override;
       ^
7 errors generated.
make: *** [hello.o] Error 1
ERROR: compilation failed for package ‘arrow’
* removing ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/arrow’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/arrow’

It looks like an alternative is to define STRICT_R_HEADERS so that it does not define Free as a macro.

#define STRICT_R_HEADERS
#include <Rcpp.h>
#include <arrow/api.h>
@romainfrancois
Copy link
Contributor Author

left a comment

Might need some help from @jimhester about rpath

@@ -31,6 +31,7 @@ export ARROW_JS_DIR=${TRAVIS_BUILD_DIR}/js
export ARROW_INTEGRATION_DIR=$TRAVIS_BUILD_DIR/integration
export ARROW_RUBY_DIR=$TRAVIS_BUILD_DIR/ruby
export ARROW_RUST_DIR=${TRAVIS_BUILD_DIR}/rust
export ARROW_R_DIR=${TRAVIS_BUILD_DIR}/r

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

Lower case is fine, that's what other languages do

@@ -0,0 +1,20 @@
Package: arrow
Title: Bindings to 'Apache' 'Arrow'

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

Title: Bindings to 'Apache' 'Arrow'
Version: 0.0.0.9000
Authors@R: c(
person("Romain", "François", email = "romain@rstudio.com", role = c("aut", "cre"))

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

@@ -0,0 +1,194 @@
Apache License

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

probably not, deleted it

the same “printed page” as the copyright notice for easier identification within
third-party archives.

Copyright 2018 Romain François

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

Deleted the license file for now.


## Installation

First install a release build of the C++ bindings to arrow.

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

Not sure, I just stole this from @jimhester. I guess it could be other types of builds as long as pkg-config finds the relevant information in the ./configure script

r/README.md Outdated

# arrow

Low level bindings to the C++ Apache Arrow library

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

Sure. Made it R integration with Apache Arrow. for now, we can always revisit this later when we can better define what this is.

PKG_CPPFLAGS=@cflags@
PKG_CXXFLAGS=$(C_VISIBILITY)
CXX_STD=CXX11
PKG_LIBS=@libs@ -Wl,-rpath,/usr/local/lib

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

That's another thing I borrowed from @jimhester who probably had a reason in this commit of rarrow: jimhester/rarrow@ac3aafc

Can maybe pkg-config give just /usr/local/lib somehow ?

// under the License.

#include <Rcpp.h>
#include "rrrow_types.h"

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

#pragma once

#include <Rcpp.h>
#undef Free

This comment has been minimized.

Copy link
@romainfrancois

romainfrancois Sep 7, 2018

Author Contributor

Yes, R defines Free here: https://github.com/wch/r-source/blob/trunk/src/include/R_ext/RS.h#L74

==> R CMD INSTALL --preclean --no-multiarch --with-keep.source r

* installing to library ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library’
* installing *source* package ‘arrow’ ...
Found pkg-config cflags and libs!
PKG_CFLAGS=-DNDEBUG -I/usr/local/include
PKG_LIBS=-L/usr/local/lib -larrow
** libs
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DNDEBUG -I/usr/local/include -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I/usr/local/include   -fPIC  -Wno-unused-result -O3 -c RcppExports.cpp -o RcppExports.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DNDEBUG -I/usr/local/include -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I/usr/local/include   -fPIC  -Wno-unused-result -O3 -c hello.cpp -o hello.o
In file included from hello.cpp:19:
In file included from ./r_arrow_types.h:22:
In file included from /usr/local/include/arrow/api.h:23:
In file included from /usr/local/include/arrow/array.h:28:
In file included from /usr/local/include/arrow/buffer.h:28:
/usr/local/include/arrow/memory_pool.h:55:38: error: too many arguments provided to function-like macro invocation
  virtual void Free(uint8_t* buffer, int64_t size) = 0;
                                     ^
/Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:74:9: note: macro 'Free' defined here
#define Free(p)        (R_chk_free( (void *)(p) ), (p) = NULL)
        ^
In file included from hello.cpp:19:
In file included from ./r_arrow_types.h:22:
In file included from /usr/local/include/arrow/api.h:23:
In file included from /usr/local/include/arrow/array.h:28:
In file included from /usr/local/include/arrow/buffer.h:28:
/usr/local/include/arrow/memory_pool.h:55:3: error: 'virtual' can only appear on non-static member functions
  virtual void Free(uint8_t* buffer, int64_t size) = 0;
  ^
/usr/local/include/arrow/memory_pool.h:55:16: error: field has incomplete type 'void'
  virtual void Free(uint8_t* buffer, int64_t size) = 0;
               ^
/usr/local/include/arrow/memory_pool.h:79:30: error: too many arguments provided to function-like macro invocation
  void Free(uint8_t* buffer, int64_t size) override;
                             ^
/Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:74:9: note: macro 'Free' defined here
#define Free(p)        (R_chk_free( (void *)(p) ), (p) = NULL)
        ^
In file included from hello.cpp:19:
In file included from ./r_arrow_types.h:22:
In file included from /usr/local/include/arrow/api.h:23:
In file included from /usr/local/include/arrow/array.h:28:
In file included from /usr/local/include/arrow/buffer.h:28:
/usr/local/include/arrow/memory_pool.h:79:8: error: field has incomplete type 'void'
  void Free(uint8_t* buffer, int64_t size) override;
       ^
/usr/local/include/arrow/memory_pool.h:101:30: error: too many arguments provided to function-like macro invocation
  void Free(uint8_t* buffer, int64_t size) override;
                             ^
/Library/Frameworks/R.framework/Resources/include/R_ext/RS.h:74:9: note: macro 'Free' defined here
#define Free(p)        (R_chk_free( (void *)(p) ), (p) = NULL)
        ^
In file included from hello.cpp:19:
In file included from ./r_arrow_types.h:22:
In file included from /usr/local/include/arrow/api.h:23:
In file included from /usr/local/include/arrow/array.h:28:
In file included from /usr/local/include/arrow/buffer.h:28:
/usr/local/include/arrow/memory_pool.h:101:8: error: field has incomplete type 'void'
  void Free(uint8_t* buffer, int64_t size) override;
       ^
7 errors generated.
make: *** [hello.o] Error 1
ERROR: compilation failed for package ‘arrow’
* removing ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/arrow’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/arrow’

It looks like an alternative is to define STRICT_R_HEADERS so that it does not define Free as a macro.

#define STRICT_R_HEADERS
#include <Rcpp.h>
#include <arrow/api.h>
romainfrancois added a commit to romainfrancois/arrow that referenced this pull request Sep 7, 2018
apply changes from apache#2489
Merge branch 'r-bootstrap' into r-dev

# Conflicts:
#	dev/release/rat_exclude_files.txt
#	r/.Rbuildignore
#	r/DESCRIPTION
#	r/NAMESPACE
#	r/R/RcppExports.R
#	r/README.Rmd
#	r/README.md
#	r/src/RcppExports.cpp
@wesm
Copy link
Member

left a comment

Looks good except for the licensing issue w/ anticonf cc @jeroen

# specific language governing permissions and limitations
# under the License.

# Anticonf (tm) script by Jeroen Ooms, Jim Hester (2017)

This comment has been minimized.

Copy link
@wesm

wesm Sep 7, 2018

Member

I missed this in my first pass. What is the license of this file? For future reference, any third party IP has to be carefully documented -- e.g. if you copy paste code from any other codebase the license must be noted and possibly added to https://github.com/apache/arrow/blob/master/LICENSE.txt. I'm especially wary of things from the R ecosystem because of GPL-ness

@jeroen

This comment has been minimized.

Copy link
Contributor

commented Sep 7, 2018

Not sure where exactly the script was adapted from, but you can use all my code under MIT license or anything compatible (i.e. pretty much any license).

@jeroen

This comment has been minimized.

Copy link
Contributor

commented Sep 7, 2018

However it looks like the information in that script is not accurate. The idea is to display which deb / rpm / brew package can be used to obtain the library. However there is no package called "arrow".

@jimhester

This comment has been minimized.

Copy link

commented Sep 7, 2018

@jeroen you are correct, in my prototype I changed the brew one, but not the rest. Looks like there are unofficial debian packages at https://arrow.apache.org/install/#c-and-glib-c-packages-for-debian-gnulinux-ubuntu-and-centos-unofficial, so these should be changed to libarrow-dev or similar and the error message with maybe a link to that page.

Add license addendums
Change-Id: I29dd0a8490bb545e68c15510b683a676fb3c611c
@wesm
wesm approved these changes Sep 7, 2018
Copy link
Member

left a comment

+1. I added a note and LICENSE addendum about the anticonf license. I will merge this shortly

@wesm wesm closed this in e7b269d Sep 7, 2018

@romainfrancois

This comment has been minimized.

Copy link
Contributor Author

commented Sep 8, 2018

Thanks 🎉

romainfrancois added a commit to romainfrancois/arrow that referenced this pull request Sep 10, 2018
romainfrancois added a commit to romainfrancois/arrow that referenced this pull request Sep 10, 2018
romainfrancois added a commit to romainfrancois/arrow that referenced this pull request Sep 20, 2018
romainfrancois added a commit to romainfrancois/arrow that referenced this pull request Sep 20, 2018

@romainfrancois romainfrancois deleted the romainfrancois:r-bootstrap branch Sep 21, 2018

romainfrancois added a commit to romainfrancois/arrow that referenced this pull request Sep 24, 2018
romainfrancois added a commit to romainfrancois/arrow that referenced this pull request Sep 24, 2018
wesm added a commit that referenced this pull request Sep 25, 2018
ARROW-3282: [R] initial R functionality
* Wrapping C++ pointers to arrow objects as R6 classes holding an R external pointer.
* Factory functions for the metadata types, int32(), ...
* Factory to create schemas and struct
* Create Array, RecordBatch, Table from R vectors and data frames. initially only support integer (int32), numeric (float64) and raw (int8) vectors.
* Reading and Writing record batches and Table to files.

Author: Romain Francois <romain@purrple.cat>

Closes #2596 from romainfrancois/r-dev-buffer and squashes the following commits:

9ab1882 <Romain Francois> mark Roxygen and Rcpp generated files
661f370 <Romain Francois> Using FirstTimeBitmapWriter instead of BitmapWriter.
e81b72b <Romain Francois> only set null_bitmap if null_count > 0
bfe853d <Romain Francois> using 0-based indices in the tests.
b391556 <Romain Francois> Also use arrow::internak::BitmapWriter
9e60555 <Romain Francois> name fixes. Using __ consistently
bf814bb <Romain Francois> Using arrow::internal::BitmapReader
c8aa703 <Romain Francois> Also use std::shared_ptr for MemoryPool.
2aa8a5f <Romain Francois> need dev version of `vctrs`
394bd33 <Romain Francois> 🐀 + RecordBatch$Slice
de93a4f <Romain Francois> RecordBatch tests
9d208a4 <Romain Francois> +Array$RangeEquals
f860063 <Romain Francois> Move each class to their own file
a89a9a8 <Romain Francois> Move RecordBatch impl to own file
a2f9f51 <Romain Francois> correctly handling offset()
8263c0d <Romain Francois> + tests for ChunkedArray
e02e24f <Romain Francois> +chunked_array and tests
b20e4b0 <Romain Francois> More tests
d11cda0 <Romain Francois> +R6 class ChunkedArray
29af2ea <Romain Francois> license headers
2f53ebf <Romain Francois> Additional tests for read_arrow / write_arrow
4237c32 <Romain Francois> Clear the bit for non NA.
ede8e44 <Romain Francois> Handle null buffer in R <-> Array conversions
a5b8190 <Romain Francois> update README with example of reading/writing arrow::Table
d951db8 <Romain Francois> "documentation" to quiet check()
908c2ac <Romain Francois> read_arrow and write_arrow now relate to arrow::Table.
110b00d <Romain Francois> resolving conflicts
ae55f8b <Romain Francois> ..
767e9d9 <Romain Francois> more generic print method
8d8cdd1 <Romain Francois> + read_arrow / write_arrow for now
c1385a0 <Romain Francois> export Array_as_vector, +Array$ToString
23fbd01 <Romain Francois> + column names
97659ff <Romain Francois> + as_tibble.arrow::RecordBatch
fa4ee22 <Romain Francois> + read_record_batch
f27eeba <Romain Francois> - MakeArray
4977bb2 <Romain Francois> no need to make ArrayData directly
ef7cda1 <Romain Francois> class constructors only take the external pointers, logic moved to factory functions
81e059a <Romain Francois> rebasing
421e471 <Romain Francois> +macro R_ERROR_NOT_OK similar to RETURN_NOT_OK but that Rcpp::stop()s
f5e3eff <Romain Francois> attempt RecordBatch$to_file
79205fb <Romain Francois> initial stab at arrow::table(data.frame)
f6f1775 <Romain Francois> s/data/.data/
b9c215b <Romain Francois> "document" array and record_batch
edf6098 <Romain Francois> Need to install `vctrs` from github for now
6aecdce <Romain Francois> skip using rpath linker option
b8dac54 <Romain Francois> +RecordBatch$schema
1fc3cc2 <Romain Francois> no longer need this
05da931 <Romain Francois> initial stab at record_batch
f4d0a34 <Romain Francois> must include arrow_types.h first
aee2d0a <Romain Francois> initial stab at arrow::array
a6ae2f3 <Romain Francois> cleanup
e14b546 <Romain Francois> follow up from @wesm comments on #2489
36e9801 <Romain Francois> + installation instructions
108caf9 <Romain Francois> not checking for headers on these files
b829bdf <Romain Francois> initial R 📦 with travis setup and testthat suite, that links to arrow c++ library and calls arrow::int32()
26e712d <Romain Francois> Initial work for type metadata, with tests.
e251299 <Romain Francois> + installation instructions
a9a8bbb <Romain Francois> not checking for headers on these files
e0a7eff <Romain Francois> initial R 📦 with travis setup and testthat suite, that links to arrow c++ library and calls arrow::int32()
b1c1109 <Romain Francois> finished rebasing after initial R patch merged
887df48 <Romain Francois> skip using rpath linker option
a6de975 <Romain Francois> cleanup
8526e51 <Romain Francois> follow up from @wesm comments on #2489
f03a277 <Romain Francois> + installation instructions
0995ca4 <Romain Francois> not checking for headers on these files
1cb547e <Romain Francois> initial R 📦 with travis setup and testthat suite, that links to arrow c++ library and calls arrow::int32()
705c125 <Romain Francois> exclude Rd files 🐀
605e302 <Romain Francois> time32 only handles second and millisecond time64 only handles microsecond and nanosecond
afdbae6 <Romain Francois> + licence header for R6.R file
65563f5 <Romain Francois> minimal documentation for check()
b7135c7 <Romain Francois> stop exporting everything
6aaf192 <Romain Francois> ignoring the .clang-format file
d854f2f <Romain Francois> + license headers for R files 🙊
d992b26 <Romain Francois> Initial work for type metadata, with tests.
614dd07 <Romain Francois> + installation instructions
afce06a <Romain Francois> initial R 📦 with travis setup and testthat suite, that links to arrow c++ library and calls arrow::int32()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.