New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Communication errors when try to send ebooks to Marvin on Ipad3 #81

Closed
Kimke7 opened this Issue May 10, 2014 · 38 comments

Comments

Projects
None yet
2 participants
@Kimke7

Kimke7 commented May 10, 2014

Calibre 1.36 (problem occurred with Calibre 1.35)
Using iOSRA 1.3.9
IPAD 3
Since I had trouble with connection times, I decided to delete library from IPAD. Polish all ebooks in my Calibre library - then send back to my Marvin Library on my Ipad. I only have the iOSRA plugin installed. As I try to transfer ebooks to Marvin - I am able to send one or a few at a time, but when I do larger numbers (as low as 25 - though once I was able to send 100 but the 2nd time got the error), I get a communication error:

calibre, version 1.36.0
ERROR: Error: Error communicating with device

'NoneType' object has no attribute 'value'

Traceback (most recent call last):
File "site-packages/calibre/gui2/device.py", line 86, in run
File "site-packages/calibre/gui2/device.py", line 571, in _upload_books
File "/var/folders/t4/grwrt7qs39qgs4fz2w15yc_m0000gp/T/calibre_1.36.0_tmp_pfygS8/gTS0tK_ios_local_db/Marvin_overlays.py", line 1431, in upload_books
File "site-packages/calibre/devices/idevice/libimobiledevice.py", line 267, in copy_to_idevice
File "site-packages/calibre/devices/idevice/libimobiledevice.py", line 835, in _afc_file_close
AttributeError: 'NoneType' object has no attribute 'value'

Once I get this error, if I try to do another, it just shows status waiting and progress unavailable - ultimately making me force quit the application to get out of it as no other attempts to disconnect device or close calibre works. I then close out Marvin and reopen Marvin and Calibre and I can get it to send small amounts of books, but then eventually errors again if send more than a few.

I have tried to do a debug report to send two times. Both times, after I got the error then clicked to eject device and discounted Marvin (in Marvin) – neither which showed within Calibre as disconnecting. I then clicked the red close button and calibre closed (without me having to force quit it and it doesn’t show in the force quite as an option) but no debug report showed up and I can’t figure out where on the Mac it was (even when I searched based on the tmp file name that had been given). Between the first and second debugging tries – I reset my Ipad to factory settings and then restored to backup. That did not make a difference.
I can then only reopen calibre if I reboot my mac – other wise it will not open after this error.
Help. Please. ☺

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 10, 2014

First of all, please update to the latest build of the iOSRA plugin from this link. I've made some changes with respect to sending many books at the same time for another beta tester who was trying to send 10,000 books at once.

Let me know what happens.

How many books were you trying to send when the error occurred?

G

@Kimke7

This comment has been minimized.

Kimke7 commented May 10, 2014

Will update iOSRA plugin now. It would do the error from trying to send 25 or higher.... It just depended as I got it to send 100 once. Will update you how lastest iOSRA build works.

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 10, 2014

I'll be working for the next several hours. Thanks for your patience, there are lots of changes going on with this new release, but it should be worth the effort.

G

@Kimke7

This comment has been minimized.

Kimke7 commented May 10, 2014

Did a couple smaller transfers - then tried a larger one (a few hundred) and got the error.

calibre, version 1.36.0
ERROR: Error: Error communicating with device

'NoneType' object has no attribute 'value'

Traceback (most recent call last):
File "site-packages/calibre/gui2/device.py", line 86, in run
File "site-packages/calibre/gui2/device.py", line 571, in _upload_books
File "/var/folders/t4/grwrt7qs39qgs4fz2w15yc_m0000gp/T/calibre_1.36.0_tmp_OD2Yct/vQYIa6_ios_local_db/Marvin_overlays.py", line 1540, in upload_books
File "/var/folders/t4/grwrt7qs39qgs4fz2w15yc_m0000gp/T/calibre_1.36.0_tmp_OD2Yct/vQYIa6_ios_local_db/Marvin_overlays.py", line 1423, in _upload_subset
File "site-packages/calibre/devices/idevice/libimobiledevice.py", line 267, in copy_to_idevice
File "site-packages/calibre/devices/idevice/libimobiledevice.py", line 835, in _afc_file_close
AttributeError: 'NoneType' object has no attribute 'value'

I had it debugging, but once again while I disconnected device from Calibre and Marvin - it did not show it disconnected within Calibre and when I closed Calibre by hitting red dot - the report didn't appear. Had to reboot mac to get calibre to open again at this point.
I understand if you are working on other things - just trying to get back with info so when you have time to look at it, you can.
Just let me know what you need me to do. Thanks.

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 10, 2014

I really need to see a debug log. Let's try the following steps:
Copy and paste the following two lines into your terminal window (one at a time), then press return after each one.

calibre-debug -c "from calibre.utils.config import JSONConfig; JSONConfig('plugins/iOS reader applications').set('development_mode',True)"

calibre-debug -c "from calibre.utils.config import JSONConfig; JSONConfig('plugins/iOS reader applications').set('debug_libimobiledevice',True)"

Next, launch in debug mode, but printing straight to the console:

calibre-debug -g

That should print everything to the console. After it crashes, you can select and copy all of the content. If it's too big to paste into a new comment, you can email it to me.

@Kimke7

This comment has been minimized.

Kimke7 commented May 10, 2014

Just sent debug log to your email address.

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 10, 2014

Never mind, it was in my junk folder :)

@Kimke7

This comment has been minimized.

Kimke7 commented May 10, 2014

Hey - it's not junk... ;-) Just had sent it again when I got message you received it.

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 10, 2014

Sigh. Unfortunately, I can't open the file. Can you send it as a text file?

@Kimke7

This comment has been minimized.

Kimke7 commented May 10, 2014

yep.... just a sec.

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 10, 2014

Are you on Mac or Windows?

@Kimke7

This comment has been minimized.

Kimke7 commented May 10, 2014

mac

@Kimke7

This comment has been minimized.

Kimke7 commented May 10, 2014

which encoding is best to use when saving as text file? It defaults as Mac OS

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 10, 2014

That should be fine.

@Kimke7

This comment has been minimized.

Kimke7 commented May 10, 2014

sent

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 10, 2014

Waiting for the email. I know it's big, so it's taking a while.

In the meantime, I've been looking at the code and I see one possible culprit, although I'm puzzled as to why I haven't see the same error. Unfortunately, this bit of code is in the main part of calibre, not the plugin. I can give you directions to try a patch, it will require some new steps. Are you up for that?

@Kimke7

This comment has been minimized.

Kimke7 commented May 10, 2014

If you can tell me what to do - I can follow directions pretty well. :-)

@Kimke7

This comment has been minimized.

Kimke7 commented May 10, 2014

check your junk email as I just realized I sent it under my other email so it may have ended up there.

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 10, 2014

I'm sending you a file 'libimobiledevice.py' via gmail. We're going to patch it into calibre. It will only affect this plugin, and you can easily undo it by simply deleting the file. Here are the steps:

  1. Open a Finder window, go to the Applications folder.
  2. Find 'calibre' in the Applications folder.
  3. Right-click on calibre, then Show package contents
  4. Within the Contents folder, navigate down to Contents/Resources/Python/site-packages/calibre/devices/idevice
  5. Within that folder, you will see three files: init.pyo, libimobiledevice.pyo, and parse_xml.pyo.
  6. Drag the file I sent you via email into that folder. There should now be 4 files in that folder.
  7. Try again. No need to run in debug mode.

(I have the new log file you sent.)

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 10, 2014

I'm studying the debug log.

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 10, 2014

"Debugging is like being the detective in a crime movie where you are also the murderer."

I didn't originate that quote, but it's so true.

Thanks to your exhaustive debug log, I can see that the patch I sent you was in fact at least part of the problem. It's a nasty bug. It may even require rebooting in order to clear the side-effects. The problem is all with the plugin, though, so no danger to your library or anything else.

To be totally squeaky clean, the best sequence after installing the patch I sent you would be to reboot your Mac and try again. At this point, you don't need to run in debug mode, just see if it works as expected.

If you're already past that, and you've made some progress, give me a status update.

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 10, 2014

I spotted another small (unrelated, nonfatal) bug in the plugin while reading through the debug file. When convenient, please update the plugin from this link.

I believe that the patch I emailed you should at least resolve the crash you were experiencing. If not, or you encounter a different problem, let me know. I'm going to take a break, I'll be back to coding in the morning, but I'll be reading email.

@Kimke7

This comment has been minimized.

Kimke7 commented May 10, 2014

I have been testing larger loads - been able to do 300, 500 and now testing around 1000 to see if there are any issues. I will do the new plugin after the 1000. I will do more testing tomorrow as it's my bedtime. Keeping fingers crossed that these changes work.

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 10, 2014

Thanks for the update. You win a gold star for helping me track this one down. Till tomorrow.

@Kimke7

This comment has been minimized.

Kimke7 commented May 11, 2014

Well, I have been sending sets of around 1000 at a time without any problems using the patch and updated plugin. Takes around 20 mins or so for that number - so I haven't tried larger. Almost got my whole library transferred again (only a couple more sets to go). Much nicer to do the large transfer than the small 10 or so at a time (felt like I would never get it transferred plus it kept crashing). Let me know if you want me to do any other tests around this - otherwise I say we got it working. Thanks again!

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 11, 2014

Great news! I'm confident that the patch resolved the problem. I'm finding that it takes about 2.5 minutes per 100 books transferred, but there's no problem with thousands of books at a time.

When you have a moment, would you please update to the latest build of the iOSRA plugin (same link as before). After installing and restarting, connect to Marvin as usual. After initialization is complete, click the down arrow next to the red Marvin icon ('Device'), then select Configure this device.

In the Help section, click Device diagnostics. Click Show details, then click the Copy to clipboard button, then paste that diagnostic info into a new comment here.

@Kimke7

This comment has been minimized.

Kimke7 commented May 11, 2014

calibre, version 1.36.0
ERROR: Unhandled exception: ValueError:max() arg is an empty sequence

calibre 1.36 isfrozen: True is64bit: True
Darwin-13.1.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '13.1.0', 'Darwin Kernel Version 13.1.0: Wed Apr 2 23:52:02 PDT 2014; root:xnu-2422.92.1~2/RELEASE_X86_64')
Python 2.7.5
OSX: ('10.9.2', ('', '', ''), 'x86_64')
Traceback (most recent call last):
File "calibre_plugins.ios_reader_apps.config", line 554, in device_diagnostics
File "calibre_plugins.ios_reader_apps.config", line 467, in _format_installed_plugins_info
ValueError: max() arg is an empty sequence

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 11, 2014

Update from this link, try again.

G

@Kimke7

This comment has been minimized.

Kimke7 commented May 11, 2014

calibre, version 1.36.0
Connected device profile:

Kathleen’s iPad

Click Show details for summary.

------------------------------------ System ------------------------------------
calibre 1.36 isfrozen:True is64bit:True
OS X 10.9.2
library: 8,762 EPUBs, 6,541 MOBIs, 16 PDFs

----------------------------------- iDevice ------------------------------------
iOSRA version: 1.3.9
iOS version: 7.1.1
model: iPad3,1
FSTotalBytes: 61,263,085,568
FSFreeBytes: 44,309,905,408
PasswordProtected: False

------------------------------------ Marvin ------------------------------------
device books: 8757
init time: 00:15

------------------------------------ Prefs -------------------------------------
created_under: 1.3.4
debug_libimobiledevice: True
debug_plugin: True
development_mode: True
device_booklist_cache_limit: 10
device_booklist_caching: False
kindle_enabled_formats: [u'MOBI']
kindle_supported_formats: [u'MOBI']
marvin_edit_collections_cb: False
marvin_protect_rb: False
marvin_replace_rb: False
marvin_update_rb: True
plugin_diagnostics: True
preferred_reader_app: Marvin

--------------------------- Device booklist caching ----------------------------
enabled: False
available space: 41.27 GB
allocation factor: 10%
allocated space: 4.13 GB

------------------------------------ Caches ------------------------------------
iOSRA
booklist.db (local) 2014-05-11 13:36:10 390,681,600
booklist.db (remote) 2014-05-10 19:44:24 49,999,872
content_hashes.db (remote) None 0
mainDb.sqlite (remote) 2014-05-11 13:15:32 10,183,680

MXD
installed_books.zip None 0

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 11, 2014

Let's do some cleanup to get rid of the extra debug settings from yesterday. Copy and paste each of the two commands below to your terminal window then press return:

calibre-debug -c "from calibre.utils.config import JSONConfig; JSONConfig('plugins/iOS reader applications').set('development_mode',False)"

calibre-debug -c "from calibre.utils.config import JSONConfig; JSONConfig('plugins/iOS reader applications').set('debug_libimobiledevice',False)"

Finally, we're back to the point where we started. Your initialization times are now much more reasonable, 0:15 seconds to initialize. When we started down this path, it was taking several minutes to initialize, which I suspected was due to one of the other plugins you had installed.

If you decide to reinstall those plugins, please keep an eye on the initialization time to see if/when it jumps up, and let me know which plugin caused the increased load time.

Also, if you're using Marvin XD, please update to the most recent build from this link.

@Kimke7

This comment has been minimized.

Kimke7 commented May 11, 2014

do I do the terminal window things with calibre open or closed? I would assume closed but checking to be sure.

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 11, 2014

Correct, exit calibre and do the terminal commands while calibre is not running.

@Kimke7

This comment has been minimized.

Kimke7 commented May 11, 2014

Did the terminal stuff - got the data about device (though you might not need it). I will let you know as I add plugins about the times. It's not the initializing that takes the longest - its the analyzing books that takes forever. lol I got all my book back on my device so takes a bit of time (15-20 mins), but it works so that's what I care about. Appreciate all the help.

calibre, version 1.36.0
Connected device profile:

Kathleen’s iPad

Click Show details for summary.

------------------------------------ System ------------------------------------
calibre 1.36 isfrozen:True is64bit:True
OS X 10.9.2
library: 8,753 EPUBs, 6,534 MOBIs, 16 PDFs

----------------------------------- iDevice ------------------------------------
iOSRA version: 1.3.9
iOS version: 7.1.1
model: iPad3,1
FSTotalBytes: 61,263,085,568
FSFreeBytes: 44,310,282,240
PasswordProtected: False

------------------------------------ Marvin ------------------------------------
device books: 8756
init time: 00:15

------------------------------------ Prefs -------------------------------------
created_under: 1.3.4
debug_libimobiledevice: False
debug_plugin: True
development_mode: False
device_booklist_cache_limit: 10
device_booklist_caching: False
kindle_enabled_formats: [u'MOBI']
kindle_supported_formats: [u'MOBI']
marvin_edit_collections_cb: False
marvin_protect_rb: False
marvin_replace_rb: False
marvin_update_rb: True
plugin_diagnostics: True
preferred_reader_app: Marvin

--------------------------- Device booklist caching ----------------------------
enabled: False
available space: 41.27 GB
allocation factor: 10%
allocated space: 4.13 GB

------------------------------------ Caches ------------------------------------
iOSRA
booklist.db (local) 2014-05-11 14:16:57 390,673,408
booklist.db (remote) 2014-05-10 19:44:24 49,999,872
content_hashes.db (remote) None 0
mainDb.sqlite (remote) 2014-05-11 13:55:13 10,183,680

MXD
installed_books.zip None 0

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 11, 2014

The 15-20 minute analyzing books phase you're referring to is after adding books to Marvin, correct?

@Kimke7

This comment has been minimized.

Kimke7 commented May 11, 2014

yeah... I enjoyed the quick times when I only had a few hundred books on Marvin - now I am over the 8500 again - so it's about what I had been getting before. :-)

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 12, 2014

Apologies for revisiting this, but could you be very specific about when the long load times occur?

For example, I know that it takes a long time to send books from calibre to Marvin, that's not going to change. I'm trying to figure out if there's any additional optimization that I can do.

When you say you were enjoying the quick times, walk me through what you were doing when you experienced the increasing delays as you added more books to Marvin.

@Kimke7

This comment has been minimized.

Kimke7 commented May 13, 2014

I was commenting on the time it took after getting the books (that we got down to 15 secs) when it first connects to Marvin. Then it takes time to analyze the books before you can do anything (with the library totally loaded it takes 15-20 mins versus a few minutes when I only had a small part of the library loaded).

@GRiker

This comment has been minimized.

Collaborator

GRiker commented May 13, 2014

The time to analyze the books - is that when you click the blue Marvin XD icon?

@GRiker GRiker closed this May 17, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment