Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Upstream breaking changes to SQLConnector - please pin sdk #16

Closed
qbatten opened this issue Feb 7, 2023 · 0 comments · Fixed by #17
Closed

chore: Upstream breaking changes to SQLConnector - please pin sdk #16

qbatten opened this issue Feb 7, 2023 · 0 comments · Fixed by #17

Comments

@qbatten
Copy link
Contributor

qbatten commented Feb 7, 2023

This PR meltano/sdk#1394 makes some small breaking changes to SQLConnector.

Recommendation: Pin the SDK <=0.19.0 in your pyproject.toml. Looks like you're currently pulling in from main, which will allow updates to happen automatically past 0.19.0; I'd recommend altering that spec to <=0.19.0.

Please take a look at the PR description for a deeper discussion. Here is a summary of what's changed.

Overview:

  • The SQLAlchemy connection on the SQLConnector is now protected, and belongs only to the SQLConnector or it's subclasses. Streams and other objects/functions should stop accessing the SQLConnector's engine or connection.
  • Within the SQLConnector, DB connections are opened in only one way: by using the _connect() method as a context manager.
  • The SQLConnector uses a single engine throughout its lifetime. Connections are opened as needed and closed after each operation.

Specific code changes:

  • _connection is gone altogether. You cannot pass it to SQLConnector and you cannot access it off of the connector. We are not caching or passing around an open DB connection anywhere anymore.
  • create_sqlalchemy_engine and create_sqlalchemy_connection have been altered to use the single cached engine, and are also deprecated. Ideally, subclasses & other objects would stop using those methods.

Suggestions:

  • If you have any database-interacting logic that happens outside of the SQLConnector (eg if your SQLStream subclass has a get_records method that gets the SQLConnector.connection and does something with it), please move that logic onto a method on your subclass of SQLConnector and call that method from the Stream/other object.
  • Wherever you're accessing the connection on SQLConnector/a subclass, please do so by using the _connect() method as a context manager. It's directly equivalent to SQLAlchemy.Engine.connect(). Here's an example of its use.
qbatten added a commit to qbatten/tap-snowflake that referenced this issue Feb 7, 2023
qbatten added a commit to qbatten/tap-snowflake that referenced this issue Feb 7, 2023
kgpayne pushed a commit that referenced this issue Mar 30, 2023
Co-authored-by: Ken Payne <ken@meltano.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant