Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory Footprint Resolved #523

Merged
merged 2 commits into from
Sep 25, 2020
Merged

Memory Footprint Resolved #523

merged 2 commits into from
Sep 25, 2020

Conversation

fahadsiddiqui
Copy link
Contributor

@fahadsiddiqui fahadsiddiqui commented Sep 24, 2020 •
edited
Loading

Resolves #522

Changes

  1. Memory footprint reduced

Checklist

  • Code
    • Includes unit tests
    • New functions have docstrings with examples that can be run with doctest
    • New functions are included in API docs
    • Docstrings include notes for any changes to API or behaviour
    • All changes documented in docs/changes.rst
  • Testing
    • (Optional) Tested local against remote servers
    • Travis CI passes (unit tests run under Linux)
    • AppVeyor CI passes (unit tests run under Windows)
    • Unit test coverage has not decreased (see Coveralls)
  • Changes
    • (Optional) Just a proof of concept
    • (Optional) Work in progress
    • Ready to review
    • Ready to merge

@coveralls
Copy link

coveralls commented Sep 24, 2020 •
edited
Loading

Coverage Status

Coverage increased (+0.005%) to 91.981% when pulling 365f4e4 on fahadsiddiqui:f/jline-memory-improvement into 704cbe4 on petl-developers:master.

@fahadsiddiqui
Copy link
Contributor Author

fahadsiddiqui commented Sep 24, 2020 •
edited
Loading

@juarezr @alimanfoo please read #522 on why I did this enhancement. It is a huge improvement to my previous implementation. Processing a 577M file took around 2.87G of RAM (may be keeping dicts appended where the dicts were huge). After making this change, the memory footprint is decreased to just 26M for a 577M of JSON lines file. Huge improvement!

If possible, let's merge this - I will use it finally tomorrow.

juarezr
juarezr approved these changes Sep 25, 2020
View reviewed changes
Copy link
Member

@juarezr juarezr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

@juarezr juarezr merged commit bed3bfb into petl-developers:master Sep 25, 2020
@juarezr
Copy link
Member

juarezr commented Sep 25, 2020 •
edited
Loading

@fahadsiddiqui ,

Do you plan more improvements or fixes before the next release?

@juarezr juarezr mentioned this pull request Feb 8, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@juarezr juarezr juarezr approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Minimizing Memory Footprint for JSON Lines Files
3 participants
@fahadsiddiqui @coveralls @juarezr