Skip to content

Commit

Permalink
Don't seek on imports
Browse files Browse the repository at this point in the history
(other than the possible seek for custom importers)

We were seeking to handle blob markers. This has two major drawbacks:

1. It wasn't possible to use a non-seekable file.  A use case for
   export/import is to copy database data.  An intermediate file, and
   associated I/O, could be avoided using a pipe, but pipes aren't
   seekable.

2. Seeks cause file-buffer data to be discarded, making IO far more
   expensive.

We didn't really need blob markers, because the preceeding blob data
records serve as markers.  (Now we're stuck with them for backward
compatibility.)
  • Loading branch information
jimfulton committed Feb 10, 2018
1 parent de1f24c commit 26a04c7
Showing 1 changed file with 9 additions and 4 deletions.
13 changes: 9 additions & 4 deletions src/ZODB/ExportImport.py
Original file line number Diff line number Diff line change
Expand Up @@ -158,9 +158,15 @@ def persistent_load(ooid):
oids[ooid] = oid = self._storage.new_oid()
return_oid_list.append(oid)

# Blob support
blob_begin = f.read(len(blob_begin_marker))
if blob_begin == blob_begin_marker:
if (len(data) < 99 and 'blob' in data and
isinstance(self._reader.getGhost(data), Blob)
):
# Blob support

# Make sure we have a (redundant, overly) blob marker.
if f.read(len(blob_begin_marker)) != blob_begin_marker:
raise ValueError("No data for blob object")

# Copy the blob data to a temporary file
# and remember the name
blob_len = u64(f.read(8))
Expand All @@ -169,7 +175,6 @@ def persistent_load(ooid):
cp(f, blob_file, blob_len)
blob_file.close()
else:
f.seek(-len(blob_begin_marker),1)
blob_filename = None

pfile = BytesIO(data)
Expand Down

0 comments on commit 26a04c7

Please sign in to comment.