MRG: expanding, rethinking mmap'ing #267
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Check that BytesIO raises UnsupportedOperation, and other types have non-zero fileno.
Test that ``array_from_file`` works correctly with ``Opener`` instances.
matthew-brett
changed the title
bz2 file does not have fileno in Python 3.2 and 2.7 on travis at least.
Test for gzip / bz2 file object, so we can in due course avoid memory-mapping these guys in ``array_from_file``.
If we can change strings loaded from files without messing up the result of a subsequent load, then we can use the string memory as modifiable memory backing the array returned from `array_from_file`.
Check that reading an array, modifying the array, then rereading the array, returns the same result for both reads.
Use test for compressed files so we don't mmap them. Numpy was trying to mmap bz2 files. Following suggestion from Nathaniel, use `readinto` method of most file-like objects to read data into given memory location, so we can be sure that memory location is not used by some other object. If we can't use `readinto`, use heuristic to guess whether we can re-use memory from file bytes read for array memory store. Use bytearray to read data into, to avoid complications with numpy array.data buffers and Python 3 / "non-native" datatypes.
|
Benchmarks ( This is the second to last PR before the release - we are in the home stretch - any takers? |
Add tests that unscaled image returns memory map by default.
| @@ -73,6 +76,14 @@ def test_Opener_various(): | |||
| with Opener(input, 'rb') as fobj: | |||
| message_back = fobj.read() | |||
| assert_equal(message, message_back) | |||
| if input is sobj: | |||
There was a problem hiding this comment.
It's a bit weird to have if input == sobj above and is sobj here, even though they probably have the same truth conditions here
There was a problem hiding this comment.
Thanks - good point changed.
Anyone have a look at this, the next-to-last PR before release?
|
This looks reasonable to me, but someone else more experienced with the code should probably take a look. |
From comment by Eric L.
|
Last call for review. I will merge later today unless I hear otherwise. |
matthew-brett
added a commit
that referenced
this pull request
Nov 11, 2014
MRG: expanding, rethinking mmap'ing Expanding the tests for the bugfix suggested by @MichielCottaar Work out the previous logic for file mapping broken for a long time by using Openers.
grlee77
pushed a commit
to grlee77/nibabel
that referenced
this pull request
Mar 15, 2016
MRG: expanding, rethinking mmap'ing Expanding the tests for the bugfix suggested by @MichielCottaar Work out the previous logic for file mapping broken for a long time by using Openers.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Expanding the tests for the bugfix suggested by @MichielCottaar
Trying to work out the previous logic broken for a long time by using Openers.