Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix forced measurement parquet files write #287

Merged
merged 2 commits into from
Aug 26, 2020
Merged

Fix forced measurement parquet files write #287

merged 2 commits into from
Aug 26, 2020

Conversation

srggrs
Copy link
Contributor

@srggrs srggrs commented Aug 26, 2020

Fix #209

@srggrs srggrs requested a review from ajstewart August 26, 2020 01:17
@github-actions github-actions bot added this to In progress in Pipeline Backlog Aug 26, 2020
@srggrs srggrs requested review from marxide and nikzadb August 26, 2020 01:17
@srggrs srggrs changed the title Fix forced maesurement parquet files write Fix forced measurement parquet files write Aug 26, 2020
Pipeline Backlog automation moved this from In progress to Reviewer approved Aug 26, 2020
ajstewart
ajstewart approved these changes Aug 26, 2020
View reviewed changes
Copy link
Contributor

@ajstewart ajstewart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me! Have you been able to test it? I was never able to reproduce large memory usage at the parquet writing step.

@srggrs
Copy link
Contributor Author

srggrs commented Aug 26, 2020

This looks good to me! Have you been able to test it? I was never able to reproduce large memory usage at the parquet writing step.

Nope I haven't. I'm busying closing all this high priority issues. But I know that should make the memory management better as the parallelisation works on different elements rather than the same dataframe with groups

@srggrs srggrs merged commit a3c5f76 into master Aug 26, 2020
Pipeline Backlog automation moved this from Reviewer approved to Done Aug 26, 2020
@srggrs srggrs deleted the fix-forced-parquet-write branch August 26, 2020 02:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ajstewart ajstewart ajstewart approved these changes

@marxide marxide Awaiting requested review from marxide

@nikzadb nikzadb Awaiting requested review from nikzadb

Assignees
No one assigned
Labels
None yet
Projects
Pipeline Backlog
  
Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Bug in parallel writing forced measurement parquet files
2 participants
@srggrs @ajstewart