Skip to content

Casting with varchar[] (array) data types from PGSQL source #1371

@kingwilly71

Description

@kingwilly71
  • pgloader --version

    3.6.a94a0a3
    
  • did you test a fresh compile from the source tree?
    yes

  • did you search for other similar issues?
    yes

  • how can I reproduce the bug?


load database
  from pgsql://source/database?sslmode=prefer
  into pgsql://target/database?sslmode=prefer
  CAST type int to int drop typemod,
       type bigint to bigint drop typemod

  • pgloader output you obtain
2022-04-12T09:24:43.015000Z LOG pgloader version "3.6.a94a0a3"
2022-04-12T09:24:43.155000Z LOG Migrating from #<PGSQL-CONNECTION pgsql://source/database {1008CE5C23}>
2022-04-12T09:24:43.155000Z LOG Migrating into #<PGSQL-CONNECTION pgsql://target/database {1008CE7413}>
2022-04-12T09:24:49.131000Z ERROR Database error 42601: syntax error at or near "("
QUERY: CREATE TABLE "public"."fun_chats"
(
  "chat_id"         bigint default NULL,
  "level"           integer default 1,
  "coordinates"     geography default NULL,
  "badges"          jsonb default NULL,
  "location"        text default NULL,
  "apply_questions" boolean default false,
  "questions"       character varying[](200) default NULL,
  "admin_approval"  boolean default false,
  "restricted"      boolean default false,
  "service_id"      text default NULL
);
2022-04-12T09:24:49.145000Z FATAL Failed to create the schema, see above.
2022-04-12T09:24:49.148000Z LOG report summary reset
       table name     errors       rows      bytes      total time
-----------------  ---------  ---------  ---------  --------------
  fetch meta data          0        112                     4.577s
   Create Schemas          0          0                     0.002s
 Create SQL Types          0          0                     1.059s
    Create tables          0          0                     0.000s
-----------------  ---------  ---------  ---------  --------------
-----------------  ---------  ---------  ---------  --------------

If I run that query directly on the database it points out that error occurs at position "questions" character varying[](200) default NULL. It is complaining about the typemod maybe.

By removing the (200) part it is possible to execute query directly and the table is succesfully created.

  • data that is being loaded, if relevant

Use this query as sample data:

INSERT INTO "public"."fun_chats" ("chat_id", "level", "coordinates", "badges", "location", "apply_questions", "questions", "admin_approval", "restricted", "service_id") VALUES
(274194652951740416, 1, 'SRID=4326;POINT(121.564558 25.03746)', NULL, NULL, 't', '{why,what}', 'f', 't', NULL),
(271622218557947904, 1, 'SRID=4326;POINT(1.4 1.4)', NULL, NULL, 't', '{Hello}', 't', 'f', NULL)
  • How the data is different from what you expected, if relevant

varchar[] data type is casted to text with or without casting specified.
First I thought that I forgot some casting rules for array data types, but reading the pgloader manual did not help.
I found a possible relative part set-to-enum-array that can convert MySQL enum to PGSQL arrays,
but this seems not applicable here.
Technically array types are identified by PG with an underline prefix (Ex. for varchar[] it is represented as _varchar),
while pgloader seems to complain about that underline. Is array data types supported?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions