Skip to content

FDW JSONSchema fixes#545

Merged
mildbyte merged 6 commits intomasterfrom
bugfix/fdw-jsonschema-fixes
Oct 8, 2021
Merged

FDW JSONSchema fixes#545
mildbyte merged 6 commits intomasterfrom
bugfix/fdw-jsonschema-fixes

Conversation

@mildbyte
Copy link
Copy Markdown
Contributor

@mildbyte mildbyte commented Oct 8, 2021

Fixes to JSONSchemas for Socrata, Snowflake and CSV in S3.

CSV and Snowflake now have a changed JSONSchema (some credentials are pushed down into a separate object for better oneOf ergonomics), so added a compatibility layer to their __init__ that lets them accept both old and new JSONSchemas. For example, the CSV plugin will accept both:

            {
                "s3_endpoint": "objectstorage:9000",
                "s3_secure": false,
                "s3_bucket": "test_csv",
                "delimiter": ","
            }

and

        {
            "delimiter": ",",
            "connection": {
                "connection_type": "s3",
                "s3_endpoint": "objectstorage:9000",
                "s3_secure": false,
                "s3_bucket": "test_csv",
            },
        }

This will let us kick down the road the problem of figuring out migrations for these configs.

Also, some minor stylistic fixes to other JSONSchemas (delete useless null type, add minimum/maximum).

CU-1gh1c5k

Push things like auth methods / CSV locations down into sub-objects and accept
either set of fields with `oneOf`.

Note this isn't complete yet, as we need to figure out how to migrate existing
configuration or how to get the data sources to accept/store multiple versions
of the config schema.

CU-1mz7fdd
Instead of writing a config migration interface for data sources, punt on it
and get the CSV data source to attempt to migrate the parameters when it's
initialized (see if they have `url` or `s3_object, ...` at toplevel and push
them down into a separate `connection` object).

We still flatten these back when initializing the FDW, as PostgreSQL only lets us
pass a key-string value map (`OPTIONS ...`).
Use a similar hack to the CSV plugin (if the `secret` object doesn't exist,
convert it before the JSONSchema validation and unpack it back when we're passing
FDW params).

#1mz7fdd
@mildbyte mildbyte merged commit d6c1fed into master Oct 8, 2021
@mildbyte mildbyte deleted the bugfix/fdw-jsonschema-fixes branch October 8, 2021 18:01
mildbyte added a commit that referenced this pull request Oct 14, 2021
  * Code refactor / optimizations (#531)
  * Support for pluggable authorization logic (#542, #549)
  * FDW JSONSchema fixes (#545)
  * Upgrade pglast to 3.4 to fix issues with Splitfile validation (#534)
  * Speed up Splitfile builds (#550)

Full set of changes: [`v0.2.16...v0.2.17`](v0.2.16...v0.2.17)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant