Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

validate_df implementation to ADLSToAzureSQL #834

Merged
merged 5 commits into from
Jan 17, 2024

Conversation

burzekj
Copy link
Contributor

@burzekj burzekj commented Dec 15, 2023

Summary

Added validate_df func into ADLSToAzureSQL flow for potential test on level ADLS-to-AzureSQL

Importance

Genesys connector needs validation on this level of data flow

Checklist

This PR:

  • follows the guidelines laid out in CONTRIBUTING.md
  • links relevant issue(s)
  • adds/updates tests (if appropriate)
  • adds/updates docstrings (if appropriate)
  • adds an entry in CHANGELOG.md

Copy link
Collaborator

@Rafalz13 Rafalz13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this function here? It was created to prevent uploading incorrect data (files) to ADLS. We have this function 'active' for each viadot source. What is the reason for putting this function here? It is the next step ADLS->DB

Please add a new parameter to tests.

@Rafalz13 Rafalz13 changed the title Validate df implementation 'validate_df implementation to ADLSToAzureSQL` Dec 19, 2023
@Rafalz13 Rafalz13 changed the title 'validate_df implementation to ADLSToAzureSQL` validate_df implementation to ADLSToAzureSQL Dec 19, 2023
@Rafalz13
Copy link
Collaborator

Rafalz13 commented Jan 4, 2024

Please resolve conflict

@burzekj
Copy link
Contributor Author

burzekj commented Jan 5, 2024

Please resolve conflict

Done

@burzekj
Copy link
Contributor Author

burzekj commented Jan 16, 2024

Do we need this function here? It was created to prevent uploading incorrect data (files) to ADLS. We have this function 'active' for each viadot source. What is the reason for putting this function here? It is the next step ADLS->DB

Please add a new parameter to tests.

Added tes for validate_df param ✅ Added tests for validate_df param

@Rafalz13
Copy link
Collaborator

Looks good, please add information to the Changelog that a new parameter was added to ADLSToAzureSQL

@burzekj
Copy link
Contributor Author

burzekj commented Jan 17, 2024

Looks good, please add information to the Changelog that a new parameter was added to ADLSToAzureSQL

Changed CHANGELOG.md file
Change CHANGELOG.md

@Rafalz13 Rafalz13 merged commit 57d9535 into dyvenia:dev Jan 17, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants