Send to Device fails with memory error

[Dhardrian]

The "Send to device" command is intermittently failing. I am updating meta data on existing books on the iDevice (i.e. Marvin configured to update rather than do nothing on existing). I have been able to trigger this error by sending as few as 901 books. Sending an entire library of 10930 books fails on every attempt.

Marvin version: 2.7
iDevice: iPad Model ME392LL/A
Calibre version: 1.34.0
Marvin XD plugin version:
iOS reading applications plugin version:
Operating system: Windows 7, 64-bit
iTunes version: 11.1.5.5

The specific steps to recreate the problem:
Attach iPad: Run Marvin: Select Get Books: Select Calibre
In Calibre, select "Send to Device"

Description of the problem:
Send to device fails with memory error.

[Dhardrian]

Forking issue to IOSRA as requested

[Dhardrian]

Attempt to send 10K+ resulted in lockup. CMD traceback:

C:\Users\Rob>calibre-debug -g > debug_log.txt
Exception ValueError: 'I/O operation on closed file' in <bound method ZipFile.__
del__ of <calibre.utils.zipfile.ZipFile instance at 0x48FB64B8>> ignored
Exception ValueError: 'I/O operation on closed file' in <bound method ZipFile.__
del__ of <calibre.utils.zipfile.ZipFile instance at 0x25DAD9E0>> ignored
Exception ValueError: 'I/O operation on closed file' in <bound method ZipFile.__
del__ of <calibre.utils.zipfile.ZipFile instance at 0x499D20A8>> ignored
Exception in thread Thread-5:
Traceback (most recent call last):
File "threading.py", line 808, in bootstrap_inner
File "site-packages\calibre\gui2\device.py", line 409, in run
File "site-packages\calibre\gui2\device.py", line 98, in run
File "site-packages\calibre\gui2\device.py", line 68, in job_done
File "site-packages\calibre__init.py", line 202, in prints
MemoryError

C++ exception thrown in slot: bad allocation
Receiver name: "qt_scrollarea_viewport" Receiver class: QWidget Event type: 12

[GRiker]

I've built a version of iOSRA that should make some progress towards supporting very large downloads.

Please download and install from this link.

The change I made is to break up large 'Send to device' requests into chunks. By default, I've set the chunk size to 100 books. The chunk size shouldn't impact performance significantly, but if this works we can experiment with making it bigger.

I'd like to ask that you test this in steps:

• Try sending 100 books whose metadata will be updated (in other words, the books are already in Marvin, and you have selected 'Update metadata' in Marvin options. Note the Running time in the Jobs window after this completes.
• Try sending 500 books whose metadata will be updated, note the Running time after the job completes.
• Try sending 2000 books, note running time.

If this works, let's move on to the next test:

• Try sending your entire library. Note the running time.

It's entirely possible that these tests will still fail, so it would be helpful to run in debug mode and capture the debug log. You can email it to me at the address I previously provided.

I'm going to close the other issue you opened in the MXD issue tracker. The rule of thumb for deciding where to open an issue is "If the bug occurs while the MXD window is open, create the issue in the MXD bug tracker. Otherwise, create the issue in the iOSRA tracker".

[Dhardrian]

100 book test - success
Upload 100 books to device – 2m 9s
Send metadata to device – 25s

500 book test - success
Upload 500 books to device – 5m 37s
Send metadata to device – 32s

2000 book test - success
Upload 2000 books to device – 20m 3s
Send metadata to device – 1m 1s

10930 book test - failed
Upload 10930 books to device – failure at 79m 59s

Debug log being sent via secondary address

---

Additional details:
Exception in thread Thread-5:
Traceback (most recent call last):
File "threading.py", line 808, in bootstrap_inner
File "site-packages\calibre\gui2\device.py", line 399, in run
File "site-packages\calibre\gui2\device.py", line 271, in detect_device
File "site-packages\calibre\devices\scanner.py", line 330, in is_device_connec
ted
File "C:\Users\Rob\AppData\Local\Temp\calibre_za71sr\zhterx_ios_local_db\Marvi
n_overlays.py", line 743, in is_usb_connected
File "calibre_plugins.ios_reader_apps.__init", line 145, in __log_location
AttributeError: 'NoneType' object has no attribute '_getframe'

---

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 86, in run
File "site-packages\calibre\gui2\device.py", line 571, in _upload_books
File "C:\Users\Rob\AppData\Local\Temp\calibre_za71sr\zhterx_ios_local_db\Marvi
n_overlays.py", line 1802, in upload_books
File "C:\Users\Rob\AppData\Local\Temp\calibre_za71sr\zhterx_ios_local_db\Marvi
n_overlays.py", line 1752, in _upload_subset
File "C:\Users\Rob\AppData\Local\Temp\calibre_za71sr\zhterx_ios_local_db\Marvi
n_overlays.py", line 2937, in _stage_command_file
File "site-packages\calibre\devices\idevice\libimobiledevice.py", line 656, in
write
File "site-packages\calibre\devices\idevice\libimobiledevice.py", line 954, in
_afc_file_write
MemoryError

---

C:\Users\Rob>calibre-debug -g > debug_log_10930.txt
Exception ValueError: 'I/O operation on closed file' in <bound method ZipFile.__
del__ of <calibre.utils.zipfile.ZipFile instance at 0x36289670>> ignored
Exception ValueError: 'I/O operation on closed file' in <bound method ZipFile.__
del__ of <calibre.utils.zipfile.ZipFile instance at 0x36FA0850>> ignored
Exception ValueError: 'I/O operation on closed file' in <bound method ZipFile.__
del__ of <calibre.utils.zipfile.ZipFile instance at 0x28B4D6C0>> ignored

[GRiker]

Please download and install an updated version from this link. This should get through the full set of books. It still may crash later in the process when trying to report the results.

[GRiker]

I've created a new issue for iOSRA beta tester #80. Please go to that issue, then click the Subscribe button in the Notifications area. That will make it easier for me to notify all of the beta testers when a new version is available.

[GRiker]

@Dhardrian, I've found another bug related to multiple file transfers. This will require another code change within calibre proper, not the plugin. Unfortunately, that means waiting another release cycle of calibre to get the patch in place. Sorry.

[Dhardrian]

Let me know if/when I can help

Sent from my iPad

On May 10, 2014, at 3:14 PM, "GRiker" <notifications@github.commailto:notifications@github.com> wrote:

@Dhardrianhttps://github.com/Dhardrian, I've found another bug related to multiple file transfers. This will require another code change within calibre proper, not the plugin. Unfortunately, that means waiting another release cycle of calibre to get the patch in place. Sorry.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/79#issuecomment-42756008.

[GRiker]

Thanks, and sorry for the delay. I've already submitted the patch to Kovid for the next release, which should be next Friday. I will ping you when it's available.

Are you by any chance running calibre from source? If so, you could test the change immediately. But if you're not, it's probably easier to just wait for the next release.

[Dhardrian]

Greg,

I am not running from source. Timing is good, I am headed to Corpus Crisiti tomorrow, but will be back before Friday. Please send information about running from source if you have it. I could setup to do that later this week.

Rob

Sent from my iPad

On May 11, 2014, at 11:05 AM, "GRiker" <notifications@github.commailto:notifications@github.com> wrote:

Thanks, and sorry for the delay. I've already submitted the patch to Kovid for the next release, which should be next Friday. I will ping you when it's available.

Are you by any chance running calibre from source? If so, you could test the change immediately. But if you're not, it's probably easier to just wait for the next release.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/79#issuecomment-42778111.

[GRiker]

You can read about setting up a calibre development environment at this link. If you decide to do so, let me know after you get it running, and then I'll tell you what to do next.

[Dhardrian]

Re-opening issue. Problems have recurred in newer versions of Calibre and plugin