A semantically unified SQL and NoSQL query and update system
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
QueryArrow-cli
QueryArrow-common
QueryArrow-db-cache
QueryArrow-db-cypher
QueryArrow-db-elastic
QueryArrow-db-filesystem
QueryArrow-db-include
QueryArrow-db-inmemory
QueryArrow-db-remote
QueryArrow-db-sql-common
QueryArrow-db-sql-hdbc-cockroachdb
QueryArrow-db-sql-hdbc-common
QueryArrow-db-sql-hdbc-postgresql
QueryArrow-db-sql-hdbc-sqlite3
QueryArrow-db-sql-libpq-postgresql
QueryArrow-db-sum
QueryArrow-db-trans
QueryArrow-ffi-c-genquery
QueryArrow-ffi-c-local
QueryArrow-ffi-c-tcp
QueryArrow-ffi-c-unixdomainsocket
QueryArrow-ffi-c
QueryArrow-ffi-genquery
QueryArrow-ffi-service-handle-common
QueryArrow-ffi-service-local
QueryArrow-ffi-service-tcp
QueryArrow-ffi-service-unixdomainsocket
QueryArrow-ffi-service
QueryArrow-gen-tool
QueryArrow-gen
QueryArrow-plugins
QueryArrow-rpc-common
QueryArrow-rpc-http
QueryArrow-rpc-socket
data-some
docker
examples/data
heterogeneous-list
log-adapter
test-QueryArrow
.gitignore
CPackConfig.cmake
LICENSE.txt
README.md
find_dependencies.sh
stack.yaml

README.md

QueryArrow

Resources

Hao Xu, Ben Keller, Antoine de Torcy, Jason Coposky (2016) QueryArrow: Bidirectional Integration of Multiple Metadata Sources. 8th iRODS User Group Meeting, University of North Carolina at Chapel Hill. June 2016.

Slides: https://irods.org/uploads/2016/06/queryarrow2_Hao-Xu_DICE_iRODS-UGM-2016.pdf https://irods.org/uploads/2017/Xu-RENCI-QueryArrow-slides.pdf

Technical Report: https://irods.org/uploads/2015/01/xu-queryarrow-2016.pdf

Paper: https://irods.org/uploads/2017/Xu-RENCI-QueryArrow-paper.pdf

Specification: https://github.com/xu-hao/CertifiedQueryArrow

Introduction

QueryArrow is motivated by the following applications: bidirectional metadata integration from different metadata sources, metadata policies, metadata migration from different databases, metadata indexing.

QueryArrow provides a systematic solution to shared namespace and unshared namespace federation of metadata. In particular, QueryArrow allows querying multiple multiple data sources including NoSQL databases, and updating data sources. For data sources that support two-phase commit, QueryArrow also supports distributed transactions. QueryArrow also enables poly-fill for features that the underlying database does not support.

A QueryArrow instance includes a QueryArrow Service and QueryArrow plugins (QAPs). Each plugin provides interface with one data store. The queries are issued from the client in the QueryArrow Language (QAL). QAL is a unified query and update language for SQL and NoSQL databases.

Build docker image

build bases

cd docker/build
docker build -t queryarrow-stack-build:0.1.0 .
cd docker/run
docker build -t queryarrow-stack-run:0.1.0 .

If you use customized bases from Internet, then

stack docker pull

build

stack build

create image

stack image container

How to build (deprecated)

Ubuntu 16.04 and CentOS 7

Install GHC 8.0.2

From source:

https://www.haskell.org/ghc/download_ghc_8_0_2#sources

Find out where ghc is installed.

If built from source, it is <prefix>/lib/ghc-8.0.2/. Default <prefix> is /usr/local.

Install Stack

Follow the instructions on this page:

https://docs.haskellstack.org

If you install from system repo, make sure you run

stack upgrade

Install Packages

On Ubuntu

apt-get install postgresql-server-dev-all libsqlite3-dev -y

On CentOS

yum install postgresql-devel sqlite3-devel -y

Build QueryArrow

git clone http://github.com/xu-hao/QueryArrow

cd QueryArrow

stack build

Create QueryArrow Package

cd ..

Make a new directory

mkdir build

cd build

../QueryArrow/find_dependencies.sh ../QueryArrow

cpack --config CPackConfig.cmake

Install QueryArrow Package

On Ubuntu

dpkg -i queryarrow-0.2-Linux-amd64.deb

On CentOS

yum install queryarrow-0.2-Linux-amd64.rpm

QueryArrow Configuration

By default QueryArrow Configuration files are stored in the /etc/QueryArrow/tdb-plugin-gen-abs.yaml file.

An example is

db_plugin:
  qap_name: cache
  catalog_database_type: Cache
  db_config:
    max_cc: 1024
    cache_db_plugin:
      qap_name: trans
      catalog_database_type: Translation
      db_config:
        rewriting_file_path: "../QueryArrow-gen/rewriting-plugin-gen.rules"
        include_file_path:
        - "../QueryArrow-gen"
        trans_db_plugin:
          qap_name: sum
          catalog_database_type: Sum
          db_config:
            summands:
            - qap_name: ICAT
              catalog_database_type: SQL/HDBC/PostgreSQL
              db_config:
                db_port: 5432
                db_name: ICAT
                db_password: testpassword
                db_host: localhost
                db_username: irods
                db_predicates: "../QueryArrow-gen/gen/ICATGen"
                db_namespace: ICAT
                db_sql_mapping: "../QueryArrow-gen/gen/SQL/ICATGen"
            - qap_name: ''
              catalog_database_type: FileSystem
              db_config:
                fs_port: 0
                db_namespace: FileSystem
                fs_host: ''
                fs_root: "/tmp"
                fs_hostmap:
                - - ''
                  - 0
                  - "/tmp"
            - qap_name: ''
              catalog_database_type: InMemory/BuiltIn
              db_config:
                db_namespace: BuiltIn
servers:
- server_protocol: service/tcp
  server_config:
    tcp_server_addr: "*"
    tcp_server_port: 12345

QueryArrow Plugin

Currently, the implemented QAPs include

Name Description db_config
Sum aggregation summands, db_namespace
Translation policy support rewriting_file_path, include_file_path,trans_db_plugin
Cache caching max_cc, cache_db_plugin
Remote/TCP remoting db_host, db_port
FileSystem interfacing with file system fs_host, fs_port, fs_root, fs_hostmap, db_namespace
ElasticSearch/ElasticSearch interfacing with ElasticSearch db_name, db_namespace, db_predicates, db_sql_mapping, db_host, db_port, db_username, db_password
Cypher/Neo4j interfacing with Neo4j db_namespace, db_predicates, db_sql_mapping, db_host, db_port, db_username, db_password
SQL/HDBC/PostgreSQL interfacing with Postgres db_name, db_namespace, db_predicates, db_sql_mapping, db_host, db_port, db_username, db_password
SQL/HDBC/SQLite3 interfacing with SQLite3 db_file_path, db_namespace, db_predicates, db_sql_mapping
SQL/HDBC/CockroachDB interfacing with CockroachDB db_name, db_namespace, db_predicates, db_sql_mapping, db_host, db_port, db_username, db_password
Include include other config files include
InMemory/StateMap in-memory mutable map db_namespace,predicate_name,db_map
InMemory/Map in-memory immutable map db_namespace,predicate_name,db_map
InMemory/BuiltIn built-in predicates: like_regex, not_like_regex, eq, ne, le, ge, lt, gt, concat, substr, strlen, add, sub, mul, div, mod, exp, like, not_like, in, replace, regex_replace, sleep, encode db_namespace

QueryArrow CLI

QueryArrow provides a CLI command QueryArrow.

QueryArrow Server

QueryArrow provides server protocols. The service protocol is used for clients to communicate with QueryArrow. The remote protocol is used by the Remote QAP. The file system protocol is used by the FileSystem QAP.

unix domain socket tcp socket http
service implemented implemented implemented
remote implemented implemented
file system implemented
Service server_config
service/tcp, remote/tcp tcp_server_addr, tcp_server_port
service/http http_server_port
service/unix domain socket, remote/unix domain socket uds_server_addr
file system/tcp fs_server_port, fs_server_root, fs_server_addr

The command for starting the server is QueryArrowServer.

iRODS Database Plugin

In addition to running QueryArrow as a standalone service, QueryArrow provides an iRODS database plugin. To build it, follow instructions for building iRODS database plugins.