Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor validation modules #542

Merged
merged 4 commits into from
Apr 14, 2023
Merged

Refactor validation modules #542

merged 4 commits into from
Apr 14, 2023

Conversation

gabegma
Copy link
Contributor

@gabegma gabegma commented Apr 12, 2023 •
edited
Loading

Description:

  • The validation modules will now leverage the caching.
    • This will improve the speed when updating the config with fields that would not impact the validation module.
    • It will also improve speed when developing.
    • The only caveat is that if users modify their pipelines/data without changing the config itself, the validation modules won't catch the error. I think that we can live with it.

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

  • ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
  • PRE-COMMIT. You ran pre-commit on all commits, or else, you
    ran pre-commit run --all-files at the end.
  • USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact
    our users.
  • DEV CHANGES.
    • Update the documentation if this PR changes how to develop/launch on the app.
    • Update the README files and our wiki for any big design decisions, if relevant.
    • Add unit tests, docstrings, typing and comments for complex sections.

@gabegma gabegma self-assigned this Apr 12, 2023
@gabegma gabegma changed the base branch from main to ggm/clean-task-in-memory April 12, 2023 21:56
@gabegma gabegma force-pushed the ggm/refactor-validation-modules branch from bca91cb to 0ae40bd Compare April 12, 2023 21:57
@gabegma gabegma marked this pull request as ready for review April 13, 2023 20:27
@gabegma gabegma force-pushed the ggm/refactor-validation-modules branch from 5876ff3 to 601875c Compare April 13, 2023 20:37
Base automatically changed from ggm/clean-task-in-memory to main April 14, 2023 00:30
JosephMarinier
JosephMarinier approved these changes Apr 14, 2023
View reviewed changes
Copy link
Contributor

@JosephMarinier JosephMarinier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's gooo! 👍

@gabegma gabegma force-pushed the ggm/refactor-validation-modules branch from 601875c to b87c32e Compare April 14, 2023 17:31
@gabegma gabegma enabled auto-merge April 14, 2023 17:32
@gabegma gabegma disabled auto-merge April 14, 2023 17:32
@gabegma gabegma enabled auto-merge (squash) April 14, 2023 17:32
@gabegma gabegma merged commit 6eb80b1 into main Apr 14, 2023
@gabegma gabegma deleted the ggm/refactor-validation-modules branch April 14, 2023 17:54
gabegma added a commit that referenced this pull request Apr 19, 2023
@gabegma
Refactor validation modules (#542)
5db81cb 
* Modify call to validation module so it uses the task manager and leverage the cache

* Reduce batch size in validation module

* Rename validation folder

* Wait for validation to be done before continuing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JosephMarinier JosephMarinier JosephMarinier approved these changes

Assignees

@gabegma gabegma

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@gabegma @JosephMarinier