Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: parallelize dataset add for performance #3338

Merged
merged 2 commits into from Mar 1, 2023
Merged

fix: parallelize dataset add for performance #3338

merged 2 commits into from Mar 1, 2023

Conversation

Panaetius
Copy link
Member

@Panaetius Panaetius commented Feb 28, 2023 •
edited

closes #3332

I tested it with the mscoco dataset the user mentioned and it took 55 seconds to add.

@Panaetius Panaetius force-pushed the bugfix/3332-parallel-dataset-add branch from c15d8e1 to 685495e Compare February 28, 2023 12:03
@coveralls
Copy link
Collaborator

coveralls commented Feb 28, 2023 •
edited

Pull Request Test Coverage Report for Build 4294347284

  • 23 of 23 (100.0%) changed or added relevant lines in 4 files are covered.
  • 18 unchanged lines in 11 files lost coverage.
  • Overall coverage increased (+0.2%) to 86.845%
Files with Coverage Reduction New Missed Lines %
renku/core/dataset/dataset.py 1 91.24%
renku/core/workflow/activity.py 1 93.89%
renku/domain_model/project_context.py 1 96.37%
renku/domain_model/workflow/composite_plan.py 1 81.55%
renku/ui/service/controllers/datasets_add_file.py 1 95.24%
renku/ui/service/errors.py 1 82.18%
renku/ui/service/jobs/contexts.py 1 73.33%
renku/version.py 2 75.0%
renku/domain_model/provenance/activity.py 3 95.81%
renku/ui/cli/exception_handler.py 3 55.79%
Totals Coverage Status
Change from base Build 4293975116: 0.2%
Covered Lines: 25443
Relevant Lines: 29297
💛 - Coveralls

@Panaetius Panaetius force-pushed the bugfix/3332-parallel-dataset-add branch from 685495e to 0f8886e Compare February 28, 2023 13:09
@Panaetius Panaetius force-pushed the bugfix/3332-parallel-dataset-add branch from 0f8886e to ebf0797 Compare February 28, 2023 13:28
@Panaetius Panaetius marked this pull request as ready for review February 28, 2023 14:18
@Panaetius Panaetius requested a review from a team as a code owner February 28, 2023 14:18
m-alisafaee
m-alisafaee approved these changes Mar 1, 2023
View reviewed changes
Copy link
Contributor

@m-alisafaee m-alisafaee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Thank you!

@Panaetius Panaetius merged commit 85ab141 into develop Mar 1, 2023
@Panaetius Panaetius deleted the bugfix/3332-parallel-dataset-add branch March 1, 2023 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@m-alisafaee m-alisafaee m-alisafaee approved these changes

Assignees
No one assigned
Labels
None yet
Projects
renku-python
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Enhance dataset add
3 participants
@Panaetius @coveralls @m-alisafaee