Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[hue4.5]presto use sqlalchemy problem #1020

Open
ganborui opened this issue Dec 17, 2019 · 1 comment
Open

[hue4.5]presto use sqlalchemy problem #1020

ganborui opened this issue Dec 17, 2019 · 1 comment
Assignees

Comments

@ganborui
Copy link

@ganborui ganborui commented Dec 17, 2019

When connecting presto through sqlalchemy, each query will generate a new uuid and save all previous results, which is not reasonable.

def execute(self, notebook, snippet):
    guid = uuid.uuid4().hex
    LOG.info("FDD_sql_alchemy guid is %s" % guid)
    engine = self._create_engine()
    connection = engine.connect()
    result = connection.execution_options(stream_results=True).execute(snippet['statement'])
    cache = {
      'connection': connection,
      'result': result,
      'meta': [{
          'name': col[0] if (type(col) is tuple or type(col) is dict) else col.name if hasattr(col, 'name') else col,
          'type': 'STRING_TYPE',
          'comment': ''
        } for col in result.cursor.description]
    }
    CONNECTION_CACHE[guid] = cache

change guid = self.user.username Is it better?

@ganborui ganborui changed the title [hue4.5] [hue4.5]presto use sqlalchemy problem Dec 17, 2019
@romainr

This comment has been minimized.

Copy link
Member

@romainr romainr commented Dec 17, 2019

For sure if the close query fails, the cache stays there forever.

Those should fix that but it is not ready yet:

  • Task Server workers HUE-8738
  • Result caching and storage HUE-8787

@jdesjean maybe there is a way to pickle the ids into Django cache that automatically support expirations or make the cache a FIFO? Or just keep the previous query like suggested above?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.