Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use mmap directly for memory-mapped FITS files #7597

Merged
merged 3 commits into from
Jul 2, 2018
Merged

Use mmap directly for memory-mapped FITS files #7597

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0947f5b
Use mmap directly for memory-mapped files
mwcraig Jun 25, 2018
813bf35
Workaround to fix windows failure
mwcraig Jun 26, 2018
6faa7c0
Remove unnecessary numpy.memmap modes
mwcraig Jul 2, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
34 changes: 19 additions & 15 deletions astropy/io/fits/file.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
from functools import reduce

import numpy as np
from numpy import memmap as Memmap

from .util import (isreadable, iswritable, isfile, fileobj_open, fileobj_name,
fileobj_closed, fileobj_mode, _array_from_file,
Expand Down Expand Up @@ -63,6 +62,11 @@
MEMMAP_MODES = {'readonly': 'c', 'copyonwrite': 'c', 'update': 'r+',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you still need these now that np.memmap is not used any more?

'append': 'c', 'denywrite': 'r'}

# Translate the numpy.memmap modes 'c', 'r', 'r+' to the appropriate
# mmap modes.
MMAP_MODES = {'r': mmap.ACCESS_READ, 'r+': mmap.ACCESS_WRITE,
'c': mmap.ACCESS_COPY}

# TODO: Eventually raise a warning, and maybe even later disable the use of
# 'copyonwrite' and 'denywrite' modes unless memmap=True. For now, however,
# that would generate too many warnings for too many users. If nothing else,
Expand Down Expand Up @@ -291,20 +295,20 @@ def readarray(self, size=None, offset=0, dtype=np.uint8, shape=None):
# Instantiate Memmap array of the file offset at 0 (so we
# can return slices of it to offset anywhere else into the
# file)
memmap = Memmap(self._file, mode=MEMMAP_MODES[self.mode],
dtype=np.uint8)

# Now we immediately discard the memmap array; we are
# really just using it as a factory function to instantiate
# the mmap object in a convenient way (may later do away
# with this usage)
self._mmap = memmap.base

# Prevent dorking with self._memmap._mmap by memmap.__del__
# in Numpy 1.6 (see
# https://github.com/numpy/numpy/commit/dcc355a0b179387eeba10c95baf2e1eb21d417c7)
memmap._mmap = None
del memmap
access_mode = MMAP_MODES[MEMMAP_MODES[self.mode]]

# For reasons unknown the file needs to point to (near)
# the beginning or end of the file. No idea how close to
# the beginning or end.
# If I had to guess there is some bug in the mmap module
# of CPython or perhaps in microsoft's underlying code
# for generating the mmap.
self._file.seek(0, 0)
# This would also work:
# self._file.seek(0, 2) # moves to the end
self._mmap = mmap.mmap(self._file.fileno(), 0,
access=access_mode,
offset=0)

return np.ndarray(shape=shape, dtype=dtype, offset=offset,
buffer=self._mmap)
Expand Down