PostgresToGoogleCloudStorageOperator - Custom schema mapping

[TheSuperAgent]

Version : 1.10.12

I used PostgresToGoogleCloudStorageOperator to export the data and the schema file as well. But I saw a column on Postgres was TIMESTAMP without time zone but in BigQuery the auto-create table (via GoogleCloudStorageToBigQueryOperator) used the JSON schema file and created the table. When I checked the BQ table the data type was TIMESTAMP.

For without timezone data, DATETIME would be the right choice. So can we manually MAP the data types during the schema file export?

[TheSuperAgent]

I customized the original operator and used my own mapping there.

type_map = {
        1114: 'DATETIME',
        1184: 'TIMESTAMP',
        1082: 'DATE',
        1083: 'TIME',
        1005: 'INTEGER',
        1007: 'INTEGER',
        1016: 'INTEGER',
        20: 'INTEGER',
        21: 'INTEGER',
        23: 'INTEGER',
        16: 'BOOLEAN',
        700: 'FLOAT',
        701: 'FLOAT',
        1700: 'FLOAT',
    }

[pierrejeambrun]

Hello,

Thank you @eladkal for pointing me to this issue. I'd be glad to work on it and implement the suggested modifications.

Best,