dbcrossbar
moves large data sets between different databases and storage formats.
Some examples:
# Copy from a CSV file to a PostgreSQL table.
dbcrossbar cp \
--if-exists=overwrite \
--schema=postgres-sql:my_table.sql \
csv:my_table.csv \
'postgres://postgres@127.0.0.1:5432/postgres#my_table'
# Upsert from a PostgreSQL table to BigQuery.
dbcrossbar cp \
--if-exists=upsert-on:id \
--temporary=gs://$GS_TEMP_BUCKET \
--temporary=bigquery:$GCLOUD_PROJECT:temp_dataset \
'postgres://postgres@127.0.0.1:5432/postgres#my_table' \
bigquery:$GCLOUD_PROJECT:my_dataset.my_table
It can also convert between table schema formats, including PostgreSQL CREATE TABLE
statements and BigQuery JSON schemas:
# Convert a PostgreSQL `CREATE TABLE` statement to a BigQuery JSON schema.
dbcrossbar schema conv postgres-sql:my_table.sql bigquery-schema:my_table.json
# Extract a schema from a CSV file and convert to Postgres `CREATE TABLE`.
dbcrossbar schema conv csv:data.csv postgres-sql:schema.sql
For more information, see the documentation.
For more instructions about building dbcrossbar
, running tests, and contributing code, see CONTRIBUTING.md.
We require nightly Rust. We lock a specific version of nightly Rust using the rust-toolchain
file. If you want to update this, take a look at Rustup components history and choose the newest version with support for rls
, clippy
and rustfmt
.
You can run the regular test suite with cargo test
, but to run the full integration tests, you'll need to do the following:
# Run a local PostgreSQL on port 5432.
docker run --name postgres -e POSTGRES_PASSWORD= -p 5432:5432 -d mdillon/postgis
createdb -h localhost -U postgres -w dbcrossbar_test
export POSTGRES_TEST_URL=postgres://postgres:@localhost:5432/dbcrossbar_test
echo "create extension if not exists postgis;" | psql $POSTGRES_TEST_URL
echo "create extension if not exists citext;" | psql $POSTGRES_TEST_URL
echo "create schema if not exists testme1;" | psql $POSTGRES_TEST_URL
# Point to test databases and test buckets.
export GS_TEST_URL=gs://$MY_GS_TEST_BUCKET/dbcrossbar/
export BQ_TEST_DATASET=$MY_BQ_ROOT:test
export S3_TEST_URL=s3://$MT_S3_TEST_BUCKET/dbcrossbar/
# This helps to ensure that we're not depending on our users to have set
# a default gcloud project anywhere.
gcloud config unset project
# These can be omitted if you don't want to test Redshift.
export REDSHIFT_TEST_URL=redshift://user:pass@server:port/db
export REDSHIFT_TEST_IAM_ROLE=$MY_IAM_ROLE
export REDSHIFT_TEST_REGIION=$MY_AWS_REGION
# Needed for BigML. Does not work with AWS_SESSION_TOKEN.
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...
export BIGML_USERNAME=... BIGML_API_KEY=...
# Run the integration tests.
env RUST_BACKTRACE=1 RUST_LOG=warn,dbcrossbar=debug \
cargo test --all -- --ignored --nocapture
Licensed under either of:
- Apache License, Version 2.0, (LICENSE-APACHE.txt or on the web)
- MIT license (LICENSE-MIT.txt or on the web)
...at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above without any additional terms or conditions.
dbcrossbar
depends on a variety of third-party libraries, each with their own copyright and license. We have configured a deny.toml
file that currently attempts to restrict our dependencies to MIT, Apache-2.0, BSD-3-Clause, BSD-2-Clause, CC0-1.0, ISC, OpenSSL and Zlib licenses, with a single MPL-2.0 dependency. But please verify the deny.toml
file (and individual dependencies) to be certain, because details may change in the future. Each of these licenses imposes certain obligations on redistribution.