Data source refactor#438

Merged

mildbyte merged 16 commits intomasterfrom

feature/data-sources-refactor

Apr 14, 2021

Commits on Apr 12, 2021

Rename some parameters for mongo_fdw and mysql_fdw to match the actual fdw_params (breaking change):
mildbyte
committed
Bunch of breaking changes (from the Py API POV -- commandline sgr mount invocations are unchanged) to the data source class:
mildbyte
committed
Fix the Snowflake adapter (extract table options out as well)
mildbyte
committed
Fix CSV / Singer ingestion tests
mildbyte
committed
Fix Socrata data source
mildbyte
committed
Refactor the table inversion code for Socrata a bit.
mildbyte
committed
Change `tableschema_to_dict` to `table_schema_params_to_dict` (utility conversion function)
mildbyte
committed
Allow the Postgres data source to receive arbitrary table params (e.g. to override the table name).
mildbyte
committed
Use semver ordering when figuring out the latest migration to apply (instead of lexicographical in which 0.0.9 > 0.0.10)
mildbyte
committed
Get introspection to suggest foreign table options (by scraping them from the engine after a mount operation).
mildbyte
committed
Make import_foreign_schema public and allow passing a dict of table options.
mildbyte
committed
Bump Multicorn to pick up the fix for table params overriding server params.
mildbyte
committed
CSV inference: instead of passing the dialect into the `CSVOptions` struct, unpack it and set the correct CSV parsing options (delimiter, quotechar)
mildbyte
committed
CSV data source: allow passing a partially initialized list of table options without a table schema (just the options like the S3 object key, not the table schema). This will make it introspect jus…
mildbyte
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data source refactor#438

Data source refactor#438
mildbyte merged 16 commits intomasterfrom
feature/data-sources-refactor

Commits on Apr 12, 2021

Rename some parameters for mongo_fdw and mysql_fdw to match the actual fdw_params (breaking change):

Bunch of breaking changes (from the Py API POV -- commandline sgr mount invocations are unchanged) to the data source class:

Fix the Snowflake adapter (extract table options out as well)

Fix CSV / Singer ingestion tests

Fix Socrata data source

Refactor the table inversion code for Socrata a bit.

Change `tableschema_to_dict` to `table_schema_params_to_dict` (utility conversion function)

Allow the Postgres data source to receive arbitrary table params (e.g. to override the table name).

Use semver ordering when figuring out the latest migration to apply (instead of lexicographical in which 0.0.9 > 0.0.10)

Get introspection to suggest foreign table options (by scraping them from the engine after a mount operation).

Make import_foreign_schema public and allow passing a dict of table options.

Bump Multicorn to pick up the fix for table params overriding server params.

CSV inference: instead of passing the dialect into the `CSVOptions` struct, unpack it and set the correct CSV parsing options (delimiter, quotechar)

CSV data source: allow passing a partially initialized list of table options without a table schema (just the options like the S3 object key, not the table schema). This will make it introspect jus…

Commits on Apr 13, 2021

Record PG notices for last run_sql inside of the Engine class instead of discarding them

Improve data source error handling: