Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Commits on Oct 5, 2015
  1. Rationalize common generic API implementation.

    When devising the common API, the first step has been to implement
    specific methods for each generic function of the protocol. It now
    appears that in some cases we don't need the extra level of flexibility:
    each change of the API has been systematically reported to all the
    specific methods, so just use a single generic definition where possible.
    In particular, introduce new intermediate class for COPY subclasses
    allowing to share more common code in the methods implementation, rather
    than having to copy/paste and maintain several versions of the same
    It would be good to be able to centralize more code for the database
    sources and how they are organized around metadata/import-data/complete
    schema, but it doesn't look obvious how to do it just now.
  2. Send one update-stats message per batch.

    Update the stats used to be a quite simple incf and doing it once per
    read row was good enough, but now that it involves sending a message to
    the monitor thread let's only send a message per batch, reducing the
    communication load here.
  3. Implement a TimeZone option for IXF loading.

    The local-time:encode-timestamp function takes a default timezone and it
    is necessary to have control over it when loading from pgloader. Hence,
    add a timezone option to the IXF option list, that is now explicit and
    local to the IXF parser rather than shared with the DBF option list.
  4. Move sexp parsing into its own file.

    After all, it's shared between the CSV command parsing and the Cast
    Rules parsing. src/parsers/command-csv.lisp still contains lots of
    facilities shared between the file based sources, will need another
    series of splits.
  5. Forgot a new file, thanks Travis.

    Someday I will learn not to code that late at night.
  6. Fix user defined casting rules.

    Commit 598c860 broke user defined
    casting rules by interning "precision" and "scale" in the
    pgloader.user-symbols package: those symbols need to be found in the
    pgloader.transforms package instead.
    Luckily enough the infrastructure to do that was already in place for
Commits on Oct 4, 2015
  1. Review the stats and reporting code organisation.

    In order to later be able to have more worker threads sharing the
    load (multiple readers and/or writers, maybe more specialized threads
    too), have all the stats be managed centrally by a single thread. We
    already have a "monitor" thread that get passed log messages so that the
    output buffer is not subject to race conditions, extend its use to also
    deal with statistics messages.
    In the current code, we send a message each time we read a row. In some
    future commits we should probably reduce the messaging here to something
    like one message per batch in the common case.
    Also, as a nice side effect of the code simplification and refactoring
    this fixes #283 wherein the before/after sections of individual CSV
    files within an ARCHIVE command where not counted in the reporting.
  2. Monitor events are now structures.

    This allows to use typecase to dispatch events in the main loop and
    avoid using destructuring-bind, as we now have properly type events.
Commits on Sep 24, 2015
  1. Add support for IXF blobs to bytea.

    A quick test shows it should work, so push that too.
Commits on Sep 23, 2015
  1. Add support for IXF CLOB data type.

    In passing, fix the condition message to read the unknown IXF data types
    as decimal numbers (rather than hexadecimal) as they are documented that
    way in the IBM reference documentation.
Commits on Sep 21, 2015
  1. Improve user code parsing, fix #297.

    To be able to use "t" (or "nil") as a column name, pgloader needs to be
    able to generate lisp code where those symbols are available. It's
    simple enough in that a Common Lisp package that doesn't :use :cl
    fullfills the condition, so intern user symbols in a specially crafted
    package that doesn't :use :cl.
    Now, we still need to be able to run transformation code that is using
    the :cl package symbols and the pgloader.transforms functions too. In
    this commit we introduce a heuristic to pick symbols either as functions
    from pgloader.transforms or anything else in pgloader.user-symbols.
    And so that user code may use NIL too, we provide an override mechanism
    to the intern-symbol heuristic and use it only when parsing user code,
    not when producing Common Lisp code from the parsed load command.
Commits on Sep 17, 2015
  1. Update README file for build instructions, fix #296.

    When building from source you should really build from current's HEAD in
    git master branch...
    In passing, comment out the --self-update paragraph as it's know to be
    broken unless you still have all the source dependencies at the right
    place for ASDF to find them... making the feature developer only.
Commits on Sep 16, 2015
Commits on Sep 11, 2015
  1. Fix the build again.

    Once more did I change a test file data and forgot to commit the changes
    to the expected file of the regression test.
  2. Improve CSV date format, fix #293.

    The date format wouldn't allow using colon (:) in the noise parts of it,
    and would also insist that milliseconds should be on 4 digits and micro
    seconds on 6 digits. Allow for "ragged" input and take however many
    digits we actually find in the input.
  3. Adapt to the new cl-ixf API.

    This allows fixing bugs in processing the IXF files, which pgloader
    directly benefits from.
  4. Fix schema qualified table names usage again.

    When the list of columns of the PostgreSQL target table isn't given in
    the load command, pgloader will happily query the system catalogs to get
    that information. The list-columns query didn't get the memo about the
    qualified table name format and the with-schema macro... fix #288.
Commits on Sep 7, 2015
  1. Travis: set PGTZ in

    The TimeZone parameter should be set both for input and for output in
    order to match our expected result file. Let's try to set PGTZ in the
    shell environment...
  2. Travis: let's try to force client timezone.

    The cvs-parse-date test is failing on Travis because the server up there
    in the Cloud isn't using the same timezone as my local machine. Let's
    just force the timezone in the SET clause...
  3. Travis: update expected output file.

    Forgot to update the expected output file in the previous commit, which
    Travis is rightfully complaining about...
  4. Allow date format parsing to support time.

    A useful use case for date parsing at tine input level is to parse
    time (hour, minutes, seconds) rather than a full date (timestamp).
    Improve the code so that it's possible to use the date format facility
    even when the data field lacks the year/month/day information.
    Fix #288.
Commits on Sep 6, 2015
  1. Fix MS SQL int conversion, fix #282.

    This is a blind fix that I hope is the one needed, wherein -1 get sent
    to unsigned-to-sign which fails to handle it as expected. The protocol
    and driver are then sending unsigned ints and that's what we now handle
    in the CFFI layer.
    This allows to push the attempt to ease testing of it, and allows me
    also to cut a release without modified files handling around.
  2. Update website, download, sponsors, license.

    The download section now explains why binaries are not to be found here
    anymore. Also we add Redpill Linpro as a sponsor, and we add a pgloader
    Moral License page in the spirit of the Varnish License. Let's see what
    happens with that.
Commits on Sep 4, 2015
  1. Default to source table name in IXF loading.

    The cleanup introduced by eabfbb9 did
    break the case when we lack a target table name for the IXF load. Just
    default to using the source name in that case.
  2. Handle default values particularities for IXF.

    It turns out that IXF format might embed particulars of the systems the
    data comes from (Informix or DB2) and we need to process some strings so
    that they are compatible with PostgreSQL ("CURRENT TIMESTAMP" here).
    See #272 that should be fixed here, but being in the blind, don't close
    the issue just yet.
  3. Error early when failing to match IXF type.

    Our IXF type matching appears to be incomplete, and pgloader would then
    happily try to create a table column of type NIL in PostgreSQL: prevent
    that from happening by raising an error condition early. Fix #289.
Commits on Sep 3, 2015
  1. Fix schema qualified table names usage (more).

    When parsing table names in the target URI, we are careful of splitting
    the table and schema name and store them into a cons in that case. Not
    all sources methods got the memo, clean that up.
    See #182 and #186, a pull request I am now going to be able to accept.
    Also see #287 that should be helped by being able to apply #186.
  2. Allow quote/downcase identifiers option for MS SQL.

    As seen in #287 the previous decision to force quoting to :none is
    wrong, because index names in MS SQL source database might contain
    spaces, and then need to be quoted.
    Let's see what happens if we do it the usual way for MS SQL too, and
    allow users to control the quoting behaviour of pgloader here.
Commits on Sep 1, 2015
  1. Allow quoted field names to contain spaces, fix #285.

    Given a fully quoted field name, there should be no restriction about
    using spaces in between the quotes, but the parser used to choke on that
Commits on Aug 31, 2015
  1. Support String Constants with Escapes in SQL files.

    pgloader has to parse external SQL files because of the driver we use,
    Postmodern, only know how to deal with sending one query at a time. So
    SQL parsing we do, and split the queries, and send them one after the
    other to the server.
    PostgreSQL allows String Constants with C-style Escapes to be used in
    some situations, and the SQL parsing done in pgloader failed to support
    This fixes #284.
Commits on Aug 29, 2015
  1. Simplify how to compute total load time, see #283.

    In some cases pgloader total time computing is quite off: in the archive
    case because it fails to take into account per-file before and after
    sections, and in the general case when there's parallel work done.
    This patch helps by storing the start time explicitely and using it at
    the moment when the summary is displayed: no guessing any more. This is
    only used in the archive case for now because I want some feedback.
    On my machine with the usual test cases I have, the difference with and
    without this patch is line-noise, something more serious has to be done:
    let's push testing to the user by committing this early version of the
Commits on Aug 24, 2015
  1. Quote table names when migrating from SQLite, fix #281.

    Apparently I just forgot to apply any smartness whatsoever to SQLite
    identifiers and just copied them as they are to PostgreSQL. Change that
    by calling apply-identifier-case.
Something went wrong with that request. Please try again.