Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Double LMDB map_size on MapFullError #209

Merged
merged 1 commit into from
Aug 12, 2015

Conversation

lukeyeager
Copy link
Member

Closes #206.

Changed

  • Don't set the initial map_size to 1TB like before
    • It seems to start with 10MB by default
  • When LMDB throws a MapFullError, double the map size

TODO before merge

  • Add this change to digits/dataset/images/generic/test_lmdb_creator.py as well
  • Write some tests
  • Update requirements.txt to require newest version of LMDB

@lukeyeager lukeyeager mentioned this pull request Aug 11, 2015
curr_limit = self.db.info()['map_size']
new_limit = curr_limit*2
logger.debug('Doubling LMDB map size to %sMB ...' % (new_limit>>20,))
self.db.set_mapsize(curr_limit*2) # double it
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since this is a new API in py-lmdb, is it worth adding a check that version is >0.86? I notice the version has not been updated yet in https://github.com/dw/py-lmdb/blob/master/lmdb/__init__.py but I guess this will come soon.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @gheinrich, I see the problem now. Somehow I committed the version bump to the release branch, not master. I've merged the single missing commit back to master. Thanks for catching!

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to be clear, the release branch and all released wheels, tarballs, git tags and GitHub tarballs will include the correct __init__.py, the problem only affected master branch.

@lukeyeager
Copy link
Member Author

Ok, I've checked off all the TODOs. @crohkohl, could you run the test suite on Windows when you get a chance?

@gheinrich
Copy link
Contributor

This looks good to me.

@crohkohl
Copy link

@lukeyeager Windows tests finally pass, see #199 (comment)

@lukeyeager
Copy link
Member Author

Squashed to a single commit and rebased to include TravisCI fix in #214.

lukeyeager added a commit that referenced this pull request Aug 12, 2015
Double LMDB map_size on MapFullError
@lukeyeager lukeyeager merged commit 7eb15d4 into NVIDIA:master Aug 12, 2015
@lukeyeager
Copy link
Member Author

@gheinrich and @crohkohl, thanks for the review help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants