Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dlt init should check if the local dlt version support the pipeline #241

Closed
rudolfix opened this issue Apr 9, 2023 · 0 comments
Closed
Assignees

Comments

@rudolfix
Copy link
Collaborator

rudolfix commented Apr 9, 2023

Background
When pipeline code is upgraded, it often requires a more recent dlt version that the one that user has. We should at least warn the user before updating the code.

Tasks

    • all pipelines have a requirements.txt. use it to get the dlt dependency version
    • there are helper functions to get the local dlt version. use standard python means (distutils??) to check if local version is matching the version string in requirements
    • perform that check during dlt init. if version does not match ask user if to continue
    • when listing pipelines check for the version and show it in the listing. warn about incompatibilities.
    • when adding/updating pipeline we generate requirements.txt, now it assumes that dlt dependency is never there. it must be smarter and actually use the dlt provided one. just remember to add extra to it with destination name
    • keep the required dlt version in TVerifiedSourceFileIndex which is dumped to .sources folder after successful dlt init

You may prepare pipelines for this upgrade first here: dlt-hub/verified-sources#97

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Ready for Deployment
Development

No branches or pull requests

2 participants