Skip to content
Migrate to PostgreSQL in a single command!
Common Lisp Makefile Other
Branch: master
Clone or download
dimitri Implement support for DBF deleted records.
Skip over deleted records rather than blindly importing them. Requires an
update of the underlying DBF library.

Fixes #985.
Latest commit 26cc9ca Jun 10, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
build Add a bundle distribution. Jan 31, 2016
bundle Fix building for 3.6.1. Jan 21, 2019
conf Update the VM setup, use sid to build the debian package. Sep 10, 2014
debian New upstream version. Jan 21, 2019
docs Allow casting rules to guard on signed data types. Jun 4, 2019
src Implement support for DBF deleted records. Jun 10, 2019
test Allow casting rules to guard on signed data types. Jun 4, 2019
.dockerignore Improved Dockerfiles/docker image size (#821) Aug 10, 2018
.gitattributes Fix github language statistics. Mar 26, 2017
.gitignore Docs cleanup. Jan 24, 2018
.travis.sh Fix the SBCL 1.5.2 checksum. May 8, 2019
.travis.yml Optimize Travis-CI workload. May 9, 2019
Dockerfile Change the main Dockerfile for sbcl to use `make save` too. Apr 17, 2019
Dockerfile.ccl Fix ccl docker images not having libssl (#968) May 17, 2019
INSTALL.md Update INSTALL.md (#585) Jun 21, 2017
ISSUE_TEMPLATE.md Some improvements on the GitHub issue template. Feb 16, 2018
LICENSE Let's be more specific about the license. Jun 17, 2017
Makefile Optimize Travis-CI workload. May 9, 2019
README.md Remove the old mention about building the docs in the README. May 8, 2019
TODO.md Fixed typos Feb 18, 2015
Vagrantfile Allow pgloader to work on windows. Nov 6, 2014
bootstrap-centos.sh Make sure EPEL is enabled when installing SBCL (#494) Dec 17, 2016
bootstrap-centos7.sh Update scripts to reference freetds-devel. Nov 1, 2016
bootstrap-debian.sh Update scripts to reference freetds-devel. Nov 1, 2016
pgloader.asd Generic Function API for Materialized Views support. (#970) May 20, 2019
pgloader.lisp Cleanup. Aug 28, 2014
pgloader.spec Update pgloader.spec (#537) Apr 17, 2017

README.md

PGLoader

Build Status Join the chat at https://gitter.im/dimitri/pgloader Docker Build Status Read The Docs Status

pgloader is a data loading tool for PostgreSQL, using the COPY command.

Its main advantage over just using COPY or \copy, and over using a Foreign Data Wrapper, is its transaction behaviour, where pgloader will keep a separate file of rejected data, but continue trying to copy good data in your database.

The default PostgreSQL behaviour is transactional, which means that any erroneous line in the input data (file or remote database) will stop the entire bulk load for the table.

pgloader also implements data reformatting, a typical example of that being the transformation of MySQL datestamps 0000-00-00 and 0000-00-00 00:00:00 to PostgreSQL NULL value (because our calendar never had a year zero).

Versioning

pgloader version 1.x is quite old and was developed in TCL. When faced with maintaining that code, the new emerging development team (hi!) picked python instead because that made sense at the time. So pgloader version 2.x was written in python.

The current version of pgloader is the 3.x series, which is written in Common Lisp for better development flexibility, runtime performance, and support of real threading.

The versioning is now following the Emacs model, where any X.0 release number means you're using a development version (alpha, beta, or release candidate). The next stable versions are going to be 3.1 then 3.2 etc.

When using a development snapshot rather than a released version the version number includes the git hash (in its abbreviated form):

  • pgloader version "3.0.99"

    Release candidate 9 for pgloader version 3.1, with a git tag named v3.0.99 so that it's easy to checkout the same sources as the released code.

  • pgloader version "3.0.fecae2c"

    Development snapshot again git hash fecae2c. It's possible to have the same sources on another setup with using the git command git checkout fecae2c.

  • pgloader version "3.1.0"

    Stable release.

LICENCE

pgloader is available under The PostgreSQL Licence.

INSTALL

You can install pgloader directly from apt.postgresql.org and from official debian repositories, see packages.debian.org/pgloader.

$ apt-get install pgloader

You can also use a docker image for pgloader at https://hub.docker.com/r/dimitri/pgloader/:

$ docker pull dimitri/pgloader
$ docker run --rm --name pgloader dimitri/pgloader:latest pgloader --version
$ docker run --rm --name pgloader dimitri/pgloader:latest pgloader --help

Build from sources

pgloader is now a Common Lisp program, tested using the SBCL (>= 1.2.5) and Clozure CL implementations with Quicklisp.

When building from sources, you should always build from the current git HEAD as it's basically the only source that is managed in a way to ensure it builds aginst current set of dependencies versions.

Building from sources on debian

$ apt-get install sbcl unzip libsqlite3-dev make curl gawk freetds-dev libzip-dev
$ cd /path/to/pgloader
$ make pgloader
$ ./build/bin/pgloader --help

Building from sources on macOS

When using brew, it should be a simple brew install --HEAD pgloader.

When using macports, then we have a situation to deal with with shared objects pgloader depends on, as reported in issue #161 at https://github.com/dimitri/pgloader/issues/161#issuecomment-201162647:

I was able to get a clean build without having to disable compression after symlinking /usr/local/lib to /opt/local/lib. Note that I did not have anything installed to /usr/local/lib so I didn't lose anything here.

Building from sources on Windows

Building pgloader on Windows is supported, thanks to Common Lisp implementations being available on that platform, and to the Common Lisp Standard for making it easy to write actually portable code.

It is recommended to have a look at the issues labelled with Windows support if you run into trouble when building pgloader:

https://github.com/dimitri/pgloader/issues?utf8=✓&q=label%3A%22Windows%20support%22%20

Building Docker image from sources

You can build a Docker image from source using SBCL by default:

$ docker build .

Or Clozure CL (CCL):

$ docker build -f Dockerfile.ccl .

More options when building from source

The Makefile target pgloader knows how to produce a Self Contained Binary file for pgloader, found at ./build/bin/pgloader:

$ make pgloader

By default, the Makefile uses SBCL to compile your binary image, though it's possible to build using CCL.

$ make CL=ccl pgloader

If using SBCL and it supports core compression, the make process will use it to generate a smaller binary. To force disabling core compression, you may use:

$ make COMPRESS_CORE=no pgloader

The --compress-core is unique to SBCL, so not used when CC is different from the sbcl value.

You can also tweak the default amount of memory that the pgloader image will allow itself using when running through your data (don't ask for more than your current RAM tho):

$ make DYNSIZE=8192 pgloader

The make pgloader command when successful outputs a ./build/bin/pgloader file for you to use.

Usage

You can either give a command file to pgloader or run it all from the command line, see the pgloader quick start on https://pgloader.readthedocs.io for more details.

$ ./build/bin/pgloader --help
$ ./build/bin/pgloader <file.load>

For example, for a full migration from SQLite:

$ createdb newdb
$ pgloader ./test/sqlite/sqlite.db postgresql:///newdb

Or for a full migration from MySQL, including schema definition (tables, indexes, foreign keys, comments) and parallel loading of the corrected data:

$ createdb pagila
$ pgloader mysql://user@localhost/sakila postgresql:///pagila
You can’t perform that action at this time.