Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Commits on Jul 26, 2015
  1. Fix the drop indexes option again, fix #251.

    The index and constraint names given by PostgreSQL catalogs should not
    be second guessed, we need to just quote them. The identifier down
    casing is interesting when we get identifiers from other system for a
    migration, but are wrong when dropping existing indexes in PostgreSQL.
    Also, the :null special value from Postmodern was routing the code
    badly, just transform it manually to nil when fetching the index list,
  2. Fix parsing dates with less-than 4 digits, fix #266.

    The previous coding decided to add 2000 to the year as an integer if it
    was below 2000, which parses 1999 as 3999. Oops. Trigger the correction
    only when the date is given on 2 digits only, parsing 04 as 2004.
    Dates given on 3 digits are kept as-is.
    Playing with the *century* special parameter allows to cancel this
    behavior, that maybe should be made entirely optional. It's just too
    common to find current years on 2 digits only, sadly.
Commits on Jul 24, 2015
  1. Improve SQL blocks support, fix #265.

    It's now possible to use several files in a BEFORE LOAD EXECUTE section,
    and to mix DO and EXECUTE parts, bringing lots of flexibility in the
    commands. Also it actually simplifies the parser.
Commits on Jul 22, 2015
  1. Process MS SQL smallint datatypes as unsigned, fix #262.

    The freetds protocol apparently sends unsigned versions of the values on
    the wire, so that we have to convert them to signed numbers upon
Commits on Jul 18, 2015
  1. Fix the preserve index names option.

    MySQL names its primary keys "PRIMARY" and we need to always uniquify
    this name even when the used asked pgloader to preserve index names.
    Also, the create-indexes-again function now needs to ask for index names
    to be preserved specifically.
  2. Fix index creation reporting, see #251.

    The new option 'drop indexes' reuses the existing code to build all the
    indexes in parallel but failed to properly account for that fact in the
    summary report with timings.
    While fixing this, also fix the SQL used to re-establish the indexes and
    associated constraints to allow for parallel execution, the ALTER TABLE
    statements would block in ACCESS EXCLUSIVE MODE otherwise and make our
    efforts vain.
Commits on Jul 17, 2015
  1. Generalize index support to handle constraints, fix #251.

    PostgreSQL rightfully forbifs DROP INDEX when the index is used to
    enforce a constraint, the proper SQL to issue is then ALTER TABLE DROP
    CONSTRAINT. Also, in such a case pg_dump issues a single ALTER TABLE ADD
    CONSTRAINT statement to restore the situation.
    Have pgloader do the same with indexes that are used to back a constraint.
  2. Cast MySQL bigint(20) into numeric, fix #253.

    In MySQL it's possible to have a bigint of 20 digits when using the
    "unsigned" variant of the data type, whereas in PostgreSQL there's no
    such variant and bigints are "limited" to the range -9223372036854775808
    to +9223372036854775807 (19 digits numbers).
    Fix the default casting rule to switch to PostgreSQL numeric in such cases.
Commits on Jul 16, 2015
  1. Implement drop indexes option for copy and fixed.

    The option doesn't seem relevant to the db3 source type which contains a
    table definition: pgloader will create the table from scratch and no
    indexes are going to be found.
  2. Refactor code from previous commit.

    The goal is to make it easy to add support for the 'drop indexes' option
    in other source types (fixed, ixf, db3, file-based sources).
  3. Implement a "drop indexes" option in CSV mode, fix #251.

    When loading against a table that already has index definitions, the
    load can be quite slow. Previous commit introduced a warning in such a
    case. This commit introduces the option "drop indexes" that is not used
    by default.
    When this option is used, pgloader drops the indexes before loading the
    data then create the indexes again with the same definitions as before.
    All the indexes are created again in parallel to optimize performances.
    Only primary key indexes can't be created in parallel, so those are
    created in two steps (create unique index then alter table).
  4. Warn against pre-existing indexes.

    Pre-existing indexes will reduce data loading performances and it's
    generally better to DROP the index prior to the load and CREATE them
    again once the load is done. See #251 for an example of that.
    In that patch we just add a WARNING against the situation, the next
    patch will also add support for a new WITH clause option allowing to
    have pgloader take care of the DROP/CREATE dance around the data
Commits on Jul 10, 2015
  1. Merge pull request #257 from Jamim/master

    Default cast rules for MySQL's datatime types fixed
Commits on Jul 4, 2015
  1. Default to a static list of PostgreSQL keywords.

    In some cases (such as when using a very old PostgreSQL instance or an
    Amazon Redshift service, as in #255), the function pg_get_keywords()
    does not exists but we assume that pgloader might still be able to
    complete its job.
    We're better off with a static list of keywords than with a unhandled
    error here, so let's see what happens next with Redshift.
Commits on Jun 27, 2015
  1. Fix error counts when transformation functions fail.

    Related to #249, stop reporting 0 errors on sources where we failed to
    handle some data transformation.
  2. Fix float-to-string to accept integers, fix #249.

    The problem in #249 is that SQLite is happy processing floats in an
    integer field, so pgloader needs to be instructing via the CAST
    mechanism to cast to float at migration time.
    But then the transformation function would choke on integers, because of
    its optimisation "declare" statement. Of course the integer
    representation expected by PostgreSQL is float-compatible, so just
    instruct the function that integers are welcome to the party.
Commits on Jun 25, 2015
  1. Add test coverage for #80.

  2. Expose cl-csv escape mode option, fix #80.

    Some CSV files are using the CSV escape character internally in their
    fields. In that case we enter a parsing bug in cl-csv where backtracking
    from parsing the escape string isn't possible (or at least
    To handle the case, change the quote parameter from \" to just \ and let
    cl-csv use its escape-quote mechanism to decide if we're escaping only
    separators or just any data.
    See AccelerationNet/cl-csv#17 where the escape
    mode feature was introduced for pgloader issue #80 already.
  3. Fix CSV parsing to log errors when trying to continue.

    The error handling was good enough to continue parsing the CSV data
    after a recoverable parser error, but not good enough to actually report
    its misfortunes to the user.
    See #250 for a report where this is misleading.
Commits on Jun 16, 2015
Commits on Jun 11, 2015
Commits on May 28, 2015
  1. Merge pull request #241 from alexbaretta/master

    Fix DROP TABLE statements on tables with foreign keys
Commits on May 26, 2015
  1. Merge pull request #240 from alexbaretta/master

    Two bugfixes
Commits on May 22, 2015
  1. Fix type declaration to include null values, fix #238.

    In passing, add a test case for NIL datetime values in our SQLite sample
  2. Allow more PostgreSQL URI options, fix #199.

    As per PostgreSQL documentation on connection strings, allow overriding
    of main URI components in the options parts, with a percent-encoded
    syntax for parameters. It allows to bypass the main URI parser
    limitations as seen in #199 (how to have a password start with a
  3. Allow for more options in the CSV escape character, fix #38.

    To allow for importing JSON one-liners as-is in the database it can be
    interesting to leverage the CSV parser in a compatible setup. That setup
    requires being able to use any separator character as the escape
  4. Fix unknown source type error, fix #237.

    In passing also recognize the ".sqlite3" file type as being a SQLite
    database file.
Commits on May 21, 2015
  1. Implement CSV headers support.

    Some CSV files are given with an header line containing the list of
    their column names, use that when given the option "csv header".
    Note that when both "skip header" and "csv header" options are used,
    pgloader first skip as many required lines and then uses the next one as
    the csv header.
    Because of temporary failure to install the `ronn` documentation tool,
    this patch only commits the changes to the source docs and omits to
    update the man page (pgloader.1). A following patch is intended to be
    pushed that fixed that.
    See #236 which is using shell tricks to retrieve the field list from the
    CSV file itself and motivated this patch to finally get written.
Commits on May 19, 2015
  1. Review upgrade config code, fix #235.

    The database connection code needed to switch to the "new" connection
    facilities, and there was a bug in the processing of template sections
    wherein the template user would inherit the template property.
Something went wrong with that request. Please try again.