Skip to content

Support decimal.Decimal type #110

Closed
Closed

Description

@dvhthomas

Decimal types in Postgres cause a failure in db.py data type selection

I have a Django app using a MoneyField, which uses a numeric(14,0) data type in Postgres (https://www.postgresql.org/docs/9.3/datatype-numeric.html). When attempting to export that table I get the following error:

$ db-to-sqlite --table isaweb_proposal "postgres://connection" test.db
....
    column_type=COLUMN_TYPE_MAPPING[column_type],
KeyError: <class 'decimal.Decimal'>

Looking at sql_utils.db.py at 292-ish it's clear that there is no matching type for what I assume SQLAlchemy interprets as Python decimal.Decimal.

From the SQLite docs it looks like DECIMAL in other DBs are considered numeric.

I'm not quite sure if it's as simple as adding a data type to that list or if there are repercussions beyond it.

Thanks for a great tool!

Activity

  1. changed the title [-]Export failed when Postgres table has numeric(X,0) column[/-] [+]Support decimal.Decimal type[/+] on May 10, 2020
  2. simonw commented on May 10, 2020

    @simonw
    Owner

    It definitely makes sense to me that this library should support decimal.Decimal.

    Here are the current supported types:

    COLUMN_TYPE_MAPPING = {
    float: "FLOAT",
    int: "INTEGER",
    bool: "INTEGER",
    str: "TEXT",
    bytes.__class__: "BLOB",
    bytes: "BLOB",
    datetime.datetime: "TEXT",
    datetime.date: "TEXT",
    datetime.time: "TEXT",
    None.__class__: "TEXT",
    # SQLite explicit types
    "TEXT": "TEXT",
    "INTEGER": "INTEGER",
    "FLOAT": "FLOAT",
    "BLOB": "BLOB",
    "text": "TEXT",
    "integer": "INTEGER",
    "float": "FLOAT",
    "blob": "BLOB",
    }

    Should decimal.Decimal be stored as a text or float in SQLite I wonder?

  3. simonw commented on May 10, 2020

    @simonw
    Owner

    https://www.sqlite.org/datatype3.html#affinity_name_examples suggests that DECIMAL(10,5) should be mapped to the SQLite affinity of NUMERIC - which I've not worked with before.

  4. simonw commented on May 10, 2020

    @simonw
    Owner

    From https://www.sqlite.org/datatype3.html#type_affinity :

    A column with NUMERIC affinity may contain values using all five storage classes. When text data is inserted into a NUMERIC column, the storage class of the text is converted to INTEGER or REAL (in order of preference) if the text is a well-formed integer or real literal, respectively. If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL. For conversions between TEXT and REAL storage classes, only the first 15 significant decimal digits of the number are preserved. If the TEXT value is not a well-formed integer or real literal, then the value is stored as TEXT.

  5. simonw commented on May 10, 2020

    @simonw
    Owner

    sqlite-utils currently treats the SQLite NUMERIC concept as a float:

    # Numeric, treated as float:
    ("NUMERIC", float),
    ("DECIMAL(10,5)", float),

  6. simonw commented on May 10, 2020

    @simonw
    Owner

    I'm going to set it up so that Python decimal.Decimal is treated in a FLOAT column, until someone convinces me otherwise!

  7. simonw commented on May 11, 2020

    @simonw
    Owner
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions