PostgreSQL Table to Apache Arrow file converter
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
LICENSE Initial commit Dec 29, 2018
Makefile composite type looks working Jan 19, 2019
README.md fix bool type support Jan 19, 2019
arrow_defs.h a workaround commit Jan 16, 2019
arrow_dump.c add Enum type support Jan 22, 2019
arrow_read.c a few bug fixes Jan 23, 2019
arrow_types.c
arrow_write.c code cleanup for default value of Date/Time Jan 24, 2019
pg2arrow.c file creation permission was explicitly specified to 0644 Jan 22, 2019
pg2arrow.h add Enum type support Jan 22, 2019
query.c add Enum type support Jan 22, 2019

README.md

What is pg2arrow?

pg2arrow is a simple and lightweight utility to query PostgreSQL relational database, and to dump the query results in Apache Arrow format.

Apache Arrow is a kind of file format that can save massive amount of structured data using columnar layout, thus, optimized to analytic workloads for insert-only dataset.

Advantages

  • Binary data transfer
    • It uses binary transfer mode of libpq. It eliminates parse/deparse operations of text encoded query results. Binary operations are fundamentally lightweight, and has less problems to handle cstring literals.
  • Exact data type mapping
    • It fetches exact data-type information from PostgreSQL, and keeps data types as is. For example, user defined composite data types are written as Struct type (it can have nested child types), however, some implementation encodes these unknown types to string representation anyway.
  • Less memory consumption
    • It writes out RecordBatch (that is a certain amount of rows) to the result file per specified size (default: 512MB). Since it does not load entire dataset on memory prior to writing out, we can dump billion rows even if it is larger than physical memory.

For more details, see our wikipage: https://github.com/heterodb/pg2arrow/wiki