Add option of columns to copied in Postgres exporter

[acirtep]

Is your feature request related to a problem? Please describe.
There are cases in which the target table of a Postgres data exporter step might have more columns than in the source file (for example audit columns).
It would be useful to have an option to add columns in the Postgres data exporter:
https://github.com/mage-ai/mage-ai/blob/master/mage_ai/io/postgres.py#L191

Postgres doc: https://www.postgresql.org/docs/current/sql-copy.html

Describe the solution you'd like
I would like to be able to configure which columns to export data to in Postgres.

Describe alternatives you've considered
Either using pandas to_sql, either the copy command.

Additional context
N/A

Thank you!

[tommydangerous]

Sorry for the delay, will take a look 👀

[tommydangerous]

In the data integration pipeline, new columns from the source are automatically added to the destination table, even on successive syncs.

As for writing out code to export to Postgres in a standard batch pipeline, one possible stop gap in the meantime is to write the code to perform the alter table command to add the column, then execute the export method to save the new data with the new column.

[acirtep]

The request is for the data exporter, not for the data integration pipeline. In the data exporter we can have export data to a Postgres table, which will call the copy command. atm, we need to make sure that the input has exactly the same columns as the postgres table. The postgres copy command has the possibility to add the list of columns which should exist in the source of the load.

[tommydangerous]

Thanks for the clarification. Great idea! Will add to our roadmap.

[thomaschung408]

Hi @acirtep ! We've added your request on our roadmap and currently planned to be release by the end of the quarter. You can see our current roadmap here: https://airtable.com/shrJctaGt4tvU27uF/tbl5bIlFkk2KhCpL4

Thank you so much for your patience and we'll keep you updated and let you know when the feature is available!

[acirtep]

Nice, thank you for the update! the prio is low from my side :)

[thomaschung408]

Thank you! I'll make sure to reach out to you when it is available. :)

[wangxiaoyou1993]

this feature is supported now.https://github.com/mage-ai/mage-ai/blob/master/mage_ai/io/postgres.py#L360