Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Commits on Jul 4, 2015
  1. Default to a static list of PostgreSQL keywords.

    In some cases (such as when using a very old PostgreSQL instance or an
    Amazon Redshift service, as in #255), the function pg_get_keywords()
    does not exists but we assume that pgloader might still be able to
    complete its job.
    We're better off with a static list of keywords than with a unhandled
    error here, so let's see what happens next with Redshift.
Commits on Jun 27, 2015
  1. Fix error counts when transformation functions fail.

    Related to #249, stop reporting 0 errors on sources where we failed to
    handle some data transformation.
  2. Fix float-to-string to accept integers, fix #249.

    The problem in #249 is that SQLite is happy processing floats in an
    integer field, so pgloader needs to be instructing via the CAST
    mechanism to cast to float at migration time.
    But then the transformation function would choke on integers, because of
    its optimisation "declare" statement. Of course the integer
    representation expected by PostgreSQL is float-compatible, so just
    instruct the function that integers are welcome to the party.
Commits on Jun 25, 2015
  1. Add test coverage for #80.

  2. Expose cl-csv escape mode option, fix #80.

    Some CSV files are using the CSV escape character internally in their
    fields. In that case we enter a parsing bug in cl-csv where backtracking
    from parsing the escape string isn't possible (or at least
    To handle the case, change the quote parameter from \" to just \ and let
    cl-csv use its escape-quote mechanism to decide if we're escaping only
    separators or just any data.
    See AccelerationNet/cl-csv#17 where the escape
    mode feature was introduced for pgloader issue #80 already.
  3. Fix CSV parsing to log errors when trying to continue.

    The error handling was good enough to continue parsing the CSV data
    after a recoverable parser error, but not good enough to actually report
    its misfortunes to the user.
    See #250 for a report where this is misleading.
Commits on Jun 16, 2015
Commits on Jun 11, 2015
Commits on May 28, 2015
  1. Merge pull request #241 from alexbaretta/master

    Fix DROP TABLE statements on tables with foreign keys
  2. @alexbaretta
Commits on May 26, 2015
  1. Merge pull request #240 from alexbaretta/master

    Two bugfixes
  2. @alexbaretta
  3. @alexbaretta
Commits on May 22, 2015
  1. Fix type declaration to include null values, fix #238.

    In passing, add a test case for NIL datetime values in our SQLite sample
  2. Allow more PostgreSQL URI options, fix #199.

    As per PostgreSQL documentation on connection strings, allow overriding
    of main URI components in the options parts, with a percent-encoded
    syntax for parameters. It allows to bypass the main URI parser
    limitations as seen in #199 (how to have a password start with a
  3. Allow for more options in the CSV escape character, fix #38.

    To allow for importing JSON one-liners as-is in the database it can be
    interesting to leverage the CSV parser in a compatible setup. That setup
    requires being able to use any separator character as the escape
  4. Fix unknown source type error, fix #237.

    In passing also recognize the ".sqlite3" file type as being a SQLite
    database file.
Commits on May 21, 2015
  1. Implement CSV headers support.

    Some CSV files are given with an header line containing the list of
    their column names, use that when given the option "csv header".
    Note that when both "skip header" and "csv header" options are used,
    pgloader first skip as many required lines and then uses the next one as
    the csv header.
    Because of temporary failure to install the `ronn` documentation tool,
    this patch only commits the changes to the source docs and omits to
    update the man page (pgloader.1). A following patch is intended to be
    pushed that fixed that.
    See #236 which is using shell tricks to retrieve the field list from the
    CSV file itself and motivated this patch to finally get written.
Commits on May 19, 2015
  1. Review upgrade config code, fix #235.

    The database connection code needed to switch to the "new" connection
    facilities, and there was a bug in the processing of template sections
    wherein the template user would inherit the template property.
Commits on May 15, 2015
  1. Merge pull request #233 from malept/accept-periods-in-csv-field-names

    Accept periods in CSV field names
  2. @malept

    Accept periods in CSV field names

    malept authored
    Periods are allowed in PG column names as well.
Commits on May 14, 2015
  1. Improve SQLite values parsing, fix #231.

    It turns out that SQLite3 data type handling is back to kick us wherever
    it hurts, this time by the driver deciding to return blob data (a vector
    of unsigned bytes) when we expect properly encoded text data.
    In the wikipedia data test case used to reproduce the bug, we're lucky
    enough that the byte vectors actually map to properly encoded strings.
    Of course doing the proper thing costs some performances.
    I'd like to be able to decide if I should blame the SQLite driver or the
    whole product on this one. The per-value data type handling still is a
    disaster in my book, tho, which means it's crucially important for
    pgloader to get it right and allow users to seemlessly migrate away from
    using such a system.
Commits on May 12, 2015
  1. Accept MySQL dbname beginning with digits, fix #230.

    pgloader used to have a single database name parsing rule that is
    supposed to be compliant with PostgreSQL identifier rules. Of course it
    turns out that MySQL naming rules are different, so adjust the parser so
    that the following connection string is accepted:
Commits on May 1, 2015
  1. Be smarter about MSSQL column_default values, fix #207.

    MS SQL default values can be quite... sophisticated, so get around with
    using a more complex expression in the SQL query that retrieve the
    default values.
    The query and implementation has been largely provided by luqelinux and
    jstans github users, and I finally merged manually their cumulated
    efforts on this front.
Commits on Apr 30, 2015
  1. Implement more COPY options, fix #218.

    The COPY format now supports user defined delimiter and null options,
    and we don't require the column names anymore as it's useless in that
  2. Fix handling of COPY data, fix #222.

    When given a file in the COPY format, we should expect that its content
    is already properly escaped as expected by PostgreSQL. Rather than
    unescape the data then escape it again, add a new more of operation to
    format-vector-row in which it won't even try to reformat the data.
    In passing, fix an off-by-one bug in dealing with non-ascii characters.
Commits on Apr 28, 2015
Commits on Apr 26, 2015
  1. Merge pull request #213 from weepee-org/centos7

    Adds bootstrap for CentOS 7
Commits on Apr 23, 2015
  1. @gvangool

    Adds bootstrap for CentOS 7

    gvangool authored
Something went wrong with that request. Please try again.