Connecting to presto through https

[yaxxie]

I'm trying to connect to presto which is reachable only via SSL. How is this achieved with omniduct? I note that pyhive have support for https, but I didn't see any way in documentation for me to specify this is available over HTTPS only.

Advice appreciated.

[danfrankj]

Hi @yaxxie

When creating an omniduct PrestoClient, you can pass additional connection_options as keyword arguments (https://github.com/airbnb/omniduct/blob/master/omniduct/databases/presto.py#L38). These then get passed to pyhive's presto client. In this case I think you want protocol='https'. (https://github.com/dropbox/PyHive/blob/19d7c2b8fb8b19dadeb2339d4d3f42c8f09204c1/pyhive/presto.py#L93)

I'll close this but feel free to re-open if that doesn't work for you.

[yaxxie]

Hi, it did not work for me. I can see from the source code that while the kwargs get passed to the connect call, they're not being passed to the SQLAlchemy engine: https://github.com/airbnb/omniduct/blob/master/omniduct/databases/presto.py#L76

[yaxxie]

@danfrankj are you able to re-open? I'm not able to reopen this.

[danfrankj]

@yaxxie Ah I see, the sql_alchemy engine is only used for schema browsing and saving dataframes ("push"ing). Are you able to query using connection_options (https://github.com/airbnb/omniduct/blob/master/omniduct/databases/presto.py#L75) ? This should only use the pyhive client and does not rely on the sqlalchemy engine.

[yaxxie]

@danfrankj sorry for wasting your time -- I just realised that this is to do with a known issue with presto returning http URLs when using HTTPs terminated at a load balancer.
The sqlalchemy engine would still need the protocol passed through if SSL were enabled and we wanted to use the dataframe retrieval function, right? So perhaps that needs to be an enhancement request/PR?

[danfrankj]

@yaxxie that is an annoying issue and hopefully the presto team is working on it.

for omniduct the connection we use to issue queries is pyhive's presto client (https://github.com/airbnb/omniduct/blob/master/omniduct/databases/presto.py#L73) which you can already configure to use https and retrieve dataframes.