Skip to content

Commit

Permalink
ARROW-7444: [GLib] Add LocalFileSystem support
Browse files Browse the repository at this point in the history 
TODO:
- [x] Implementation
- [x] Make GArrowFileType an GEnum type
- [x] Make setters and getters of GArrowFileStats properties
- [x] Write tests

Closes #6105 from mrkn/ARROW-7444 and squashes the following commits:

3726abd <Sutou Kouhei> Use method
9f4d0f7 <Sutou Kouhei> Simplify API
a4ed9a2 <Sutou Kouhei> Adjust style
a042e54 <Sutou Kouhei> Add tests GBytes version reads
19eb216 <Sutou Kouhei> Implement garrow_file_system_open_input_file()
1c1a504 <Sutou Kouhei> Add garrow_seekable_input_stream_read_at_bytes()
0013736 <Sutou Kouhei> Add garrow_seekable_input_stream_new_raw()
2944a19 <Sutou Kouhei> Remove needless "" tag
4d2c328 <Sutou Kouhei> Use fs::FileInfo
a380282 <Sutou Kouhei> Fix document
6129632 <Sutou Kouhei> Use "dir" instead of "directory" for API
22cfe38 <Sutou Kouhei> Use nonexistent because not-found isn't suitable for file selector
2e20ad6 <Sutou Kouhei> Add garrow_file_stats_to_string()
d62ef8e <Sutou Kouhei> Document missing garrow_file_stats_is_file() and is_directory()
9764716 <Sutou Kouhei> Use "directory" instead of "dir"
06dfda6 <Sutou Kouhei> Use arrow::fs::kNoSize and kNoTime to get the minimum value
7fa2a03 <Sutou Kouhei> Fold long lines
ca3d350 <Sutou Kouhei> Remove wrong const
a3939c5 <Sutou Kouhei> Use G_DECLARE_INTERFACE
acf4035 <Sutou Kouhei> Use not_found instead of non_existent
cfbe3e6 <Sutou Kouhei> Keep reference to base file system
cc434d7 <Sutou Kouhei> Don't put tests into helper/
8ff0af3 <Sutou Kouhei> Add garrow_readable_read_bytes() and use it in test
d924772 <Sutou Kouhei> Ensure destroy a read buffer for stream
9a88c17 <Sutou Kouhei> Add missing include
44d0ef5 <Sutou Kouhei> Document missing parameter
d91335e <Sutou Kouhei> Improve readable method name for building GArrowBuffer
9b3284f <Sutou Kouhei> Don't use duplicated ID
3b9404a <Sutou Kouhei> Improve test case names
990684e <Sutou Kouhei> Don't access to a buffer read from a closed stream
566f2dc <Kenta Murata> Rename garrow_file_stats_list_new
59724ea <Kenta Murata> Use g_strndup to return a temporary string
fe26c83 <Kenta Murata> Use garrow_file_stats_get_raw
24ec232 <Kenta Murata> Use placement new to initialize a shared_ptr of FileSystem member
d15b190 <Kenta Murata> Get the default property values from a arrow::fs::FileStats object
0431b40 <Kenta Murata> Fix parameter long names
6dd6878 <Kenta Murata> Remove trailing periods from parameter descriptions
994f5d7 <Kenta Murata> Fix an error tag
0b61376 <Kenta Murata> Replace "file_system" to "file-system" in a param name
042f75c <Kenta Murata> Replace "filesystem" to "file system" or "file-system" appropriately
34bc4b4 <Kenta Murata> Remove needless TODO mark
76a049b <Kenta Murata> Add tests of GArrowSlowFileSystem
8262060 <Kenta Murata> Rewrite test-local-file-system.rb
110ffd9 <Sutou Kouhei> Add component availability check
8f4dfa4 <Kenta Murata> Fix function names
8806015 <Kenta Murata> Fix coding style
3b19ea3 <Kenta Murata> Include file-system and local-file-system in documentation
09ef670 <Kenta Murata> Add a section for local file system in documentation
6c831f7 <Kenta Murata> Use "file-system" instead of "file_system"
3c86869 <Kenta Murata> Update arrow-glib/Makefile.am
6a62eb6 <Kenta Murata> Fix gi annotations and tests
79046e1 <Kenta Murata> Add tests of GArrowLocalFileSystem
1d62528 <Kenta Murata> Make attributes of LocalFileSystemOptions properties
68c5d1c <Kenta Murata> Add tests of GArrowFileSelector
463703d <Kenta Murata> Remove needless prototype declarations
10762e9 <Kenta Murata> Add tests of GArrowFileStats
dcc16b0 <Kenta Murata> Fix the default value of GArrowFileStats:size and GArrowFileStats:mtime
d9c3da0 <Kenta Murata> Add the missing prototype declaration of garrow_file_stats_equal
e0ce704 <Kenta Murata> Fix the default value of GArrowFileStats:type
1a5e963 <Kenta Murata> Remove accessors of FileStats
58e1601 <Kenta Murata> Make attributes of FileStats properties
1b4d8af <Kenta Murata> Stop using unique_ptr to keep FileStats and FileSelector in a GObject
4a205b6 <Kenta Murata> Return the result of garrow::check
edd9d89 <Kenta Murata> Replace _ with - in error messages
24c5d6b <Kenta Murata> Fix suffixes of function names
0aa9976 <Kenta Murata> Make anonymous functions static ones
64249da <Kenta Murata> Use unique_ptr to keep a arrow::fs::FileStats instance
b9ef9d0 <Kenta Murata> Make attributes of GArrowFileSelector properties
88157d8 <Kenta Murata> Aline the trailing backslashes
0ae6b97 <Kenta Murata> Use static_cast
71083f0 <Kenta Murata> Remove empty get_property functions
cb5288f <Kenta Murata> Add GArrowLocalFileSystem
418525c <Kenta Murata> Add GArrowFileSystem

Lead-authored-by: Kenta Murata <mrkn@mrkn.jp>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
  • Loading branch information
@mrkn @kou
mrkn and kou committed Mar 7, 2020
1 parent 88e3267 commit fe71388
Show file tree
Hide file tree
Showing 26 changed files with 2,874 additions and 36 deletions.
14 changes: 7 additions & 7 deletions c_glib/arrow-cuda-glib/cuda.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -653,12 +653,12 @@ garrow_cuda_ipc_memory_handle_serialize(GArrowCUDAIPCMemoryHandle *handle,
}

GArrowBuffer *
garrow_cuda_buffer_input_stream_new_raw_readable_interface(std::shared_ptr<arrow::Buffer> *arrow_buffer)
garrow_cuda_buffer_input_stream_buffer_new_raw_readable_interface(std::shared_ptr<arrow::Buffer> *arrow_buffer)
{
auto buffer = GARROW_BUFFER(g_object_new(GARROW_CUDA_TYPE_BUFFER,
"buffer", arrow_buffer,
NULL));
return buffer;
auto arrow_cuda_buffer =
reinterpret_cast<std::shared_ptr<arrow::cuda::CudaBuffer> *>(arrow_buffer);
auto cuda_buffer = garrow_cuda_buffer_new_raw(arrow_cuda_buffer);
return GARROW_BUFFER(cuda_buffer);
}

static std::shared_ptr<arrow::io::Readable>
Expand All @@ -672,8 +672,8 @@ garrow_cuda_buffer_input_stream_get_raw_readable_interface(GArrowReadable *reada
static void
garrow_cuda_buffer_input_stream_readable_interface_init(GArrowReadableInterface *iface)
{
iface->new_raw =
garrow_cuda_buffer_input_stream_new_raw_readable_interface;
iface->buffer_new_raw =
garrow_cuda_buffer_input_stream_buffer_new_raw_readable_interface;
iface->get_raw =
garrow_cuda_buffer_input_stream_get_raw_readable_interface;
}
Expand Down
12 changes: 12 additions & 0 deletions c_glib/arrow-glib/Makefile.am
View file
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,10 @@ libarrow_glib_la_headers += \
libarrow_glib_la_headers += \
compute.h

libarrow_glib_la_headers += \
file-system.h \
local-file-system.h

if HAVE_ARROW_ORC
libarrow_glib_la_headers += \
orc-file-reader.h
Expand Down Expand Up @@ -137,6 +141,10 @@ libarrow_glib_la_sources += \
libarrow_glib_la_sources += \
compute.cpp

libarrow_glib_la_sources += \
file-system.cpp \
local-file-system.cpp

if HAVE_ARROW_ORC
libarrow_glib_la_sources += \
orc-file-reader.cpp
Expand Down Expand Up @@ -179,6 +187,10 @@ libarrow_glib_la_cpp_headers += \
libarrow_glib_la_cpp_headers += \
compute.hpp

libarrow_glib_la_cpp_headers += \
file-system.hpp \
local-system.hpp

if HAVE_ARROW_ORC
libarrow_glib_la_cpp_headers += \
orc-file-reader.hpp
Expand Down
3 changes: 3 additions & 0 deletions c_glib/arrow-glib/arrow-glib.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,6 @@
#include <arrow-glib/metadata-version.h>
#include <arrow-glib/reader.h>
#include <arrow-glib/writer.h>

#include <arrow-glib/file-system.h>
#include <arrow-glib/local-file-system.h>
Loading

0 comments on commit fe71388

Please sign in to comment.