Skip to content

Fix mount for data source with empty credentials schema#515

Merged
mildbyte merged 1 commit intosplitgraph:masterfrom
naskoro:grisha--fix-hn-mount-empty-credentials-schema
Aug 5, 2021
Merged

Fix mount for data source with empty credentials schema#515
mildbyte merged 1 commit intosplitgraph:masterfrom
naskoro:grisha--fix-hn-mount-empty-credentials-schema

Conversation

@naspeh
Copy link
Copy Markdown
Contributor

@naspeh naspeh commented Aug 5, 2021

When I was checking the hn example I got an error

root@b809fd2dba0c:/# sgr mount hackernews hackernews
error: KeyError: 'properties'

root@b809fd2dba0c:/# sgr -v DEBUG mount hackernews hackernews
error: Traceback (most recent call last):
error:   File "/usr/local/lib/python3.8/site-packages/splitgraph/commandline/__init__.py", line 115, in invoke
error:     result = super(click.Group, self).invoke(ctx)
error:   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
error:     return _process_result(sub_ctx.command.invoke(sub_ctx))
error:   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
error:     return _process_result(sub_ctx.command.invoke(sub_ctx))
error:   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
error:     return ctx.invoke(self.callback, **ctx.params)
error:   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
error:     return callback(*args, **kwargs)
error:   File "/usr/local/lib/python3.8/site-packages/splitgraph/commandline/mount.py", line 69, in _callback
error:     mount(schema, mount_handler=handler_name, handler_kwargs=handler_options)
error:   File "/usr/local/lib/python3.8/site-packages/splitgraph/hooks/mount_handlers.py", line 68, in mount
error:     source = data_source.from_commandline(engine, handler_kwargs)
error:   File "/hn_fdw/hn_fdw/mount.py", line 42, in from_commandline
error:     self = super().from_commandline(*args, **kwargs)
error:   File "/usr/local/lib/python3.8/site-packages/splitgraph/hooks/data_source/fdw.py", line 71, in from_commandline
error:     for k in cast(Dict[str, Any], cls.credentials_schema["properties"]).keys():
error: KeyError: 'properties'

I was thinking about just fixing HackerNewsDataSource like this:

--- a/examples/custom_fdw/src/hn_fdw/mount.py
+++ b/examples/custom_fdw/src/hn_fdw/mount.py
@@ -34,7 +34,7 @@ _all_endpoints = [
 
 
 class HackerNewsDataSource(ForeignDataWrapperDataSource):
-    credentials_schema = {"type": "object"}
+    credentials_schema = {"type": "object", "properties": {}}
 
     @classmethod
     def from_commandline(cls, *args, **kwargs):

But decided to go a little bit further and create a test reproducing error and more general fix.

@mildbyte
Copy link
Copy Markdown
Contributor

mildbyte commented Aug 5, 2021

Good catch, thanks! I'll merge this so that CI/typechecks can run on our side.

@mildbyte mildbyte merged commit a52eea3 into splitgraph:master Aug 5, 2021
@naspeh naspeh deleted the grisha--fix-hn-mount-empty-credentials-schema branch August 5, 2021 13:46
mildbyte added a commit that referenced this pull request Aug 18, 2021
  * Various Airbyte ingestion improvements and support for different normalization modes, including a custom dbt model (#510, #513, #514)
  * Fix mount for data source with empty credentials schema (#515)
  * Fix `sgr cloud load`/`dump` (#520)

Full set of changes: [`v0.2.15...v0.2.16`](v0.2.15...v0.2.16)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants