Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Amazon/Kindle AZW-format ebooks NOT stripping DRM with Calibre plugin #52

Closed
kellybell opened this issue Dec 28, 2015 · 67 comments
Closed

Comments

@kellybell
Copy link

I am using the latest version of Calibre and the plug-in (both were updated to latest versions today) to try to de-DRM my Kindle books in AZW format, and while some will convert, many will not. I am using a bulk-add-and-convert, and some are getting stripped (and will then convert) and some will not. What's going on? In Calibre, I can't tell any difference between the formatting at all - they are showing as the same (AZW). Help?

These are all books which I have purchased, btw. Out of 113 requested books, 65 books DID NOT convert (that's more than half!). I also tried adding them individually, and they don't work that way either.

Some examples:

Example #1:
Convert book 1 of 1 (The Autistic Brain: Thinking Across the Spectrum)
(log details):
Convert book 1 of 1 (The Autistic Brain: Thinking Across the Spectrum)
Resolved conversion options
calibre version: 2.47.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0.0,
'book_producer': None,
'change_justification': u'original',
'chapter': u"//[((name()='h1' or name()='h2') and re:test(., '\s((chapter|book|section|part)\s+)|((prolog|prologue|epilogue)(\s+|$))', 'i')) or @Class = 'chapter']",
'chapter_mark': u'pagebreak',
'comments': None,
'cover': u'/var/folders/k6/rpx44f5j73s69qklkkf0b9wr0000gn/T/calibre_2.47.0_tmp_FqtW9f/RwDTjI.jpeg',
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_split_on_page_breaks': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_toc_at_end': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': u'',
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x114285610>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0.0,
'linearize_tables': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.iPad3Output object at 0x114285bd0>,
'page_breaks_before': u"//*[name()='h1' or name()='h2']",
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': u'/var/folders/k6/rpx44f5j73s69qklkkf0b9wr0000gn/T/calibre_2.47.0_tmp_FqtW9f/4e2Iwo.opf',
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': u'',
'search_replace': '[]',
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': None,
'sr1_search': None,
'sr2_replace': None,
'sr2_search': None,
'sr3_replace': None,
'sr3_search': None,
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
Python function terminated unexpectedly: The Autistic Brain: Thinking Across the Spectrum
InputFormatPlugin: MOBI Input running
on /var/folders/k6/rpx44f5j73s69qklkkf0b9wr0000gn/T/calibre_2.47.0_tmp_FqtW9f/aoZnWG.azw
Found KF8 MOBI of type 'standalone'
Traceback (most recent call last):
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 209, in main
return run_entry_point()
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 114, in run_entry_point
return getattr(pmod, func)()
File "site-packages/calibre/utils/ipc/worker.py", line 190, in main
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 31, in gui_convert_override
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
File "site-packages/calibre/ebooks/conversion/plumber.py", line 1051, in run
File "site-packages/calibre/customize/conversion.py", line 241, in call
File "site-packages/calibre/ebooks/conversion/plugins/mobi_input.py", line 43, in convert
File "site-packages/calibre/ebooks/mobi/reader/mobi8.py", line 81, in call
File "site-packages/calibre/ebooks/mobi/reader/mobi6.py", line 144, in check_for_drm
calibre.ebooks.DRMError: The Autistic Brain: Thinking Across the Spectrum

Example #2
Convert book 94 of 113 (The Magician King: A Novel (The Magicians Book 2))
Convert book 94 of 113 (The Magician King: A Novel (The Magicians Book 2))
Resolved conversion options
calibre version: 2.47.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0.0,
'book_producer': None,
'change_justification': u'original',
'chapter': u"//[((name()='h1' or name()='h2') and re:test(., '\s((chapter|book|section|part)\s+)|((prolog|prologue|epilogue)(\s+|$))', 'i')) or @Class = 'chapter']",
'chapter_mark': u'pagebreak',
'comments': None,
'cover': u'/var/folders/k6/rpx44f5j73s69qklkkf0b9wr0000gn/T/calibre_2.47.0_tmp_FqtW9f/Y1jOJT.jpeg',
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_split_on_page_breaks': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_toc_at_end': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': u'',
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x10c7a1590>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0.0,
'linearize_tables': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.iPad3Output object at 0x10c7a1b50>,
'page_breaks_before': u"//*[name()='h1' or name()='h2']",
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': u'/var/folders/k6/rpx44f5j73s69qklkkf0b9wr0000gn/T/calibre_2.47.0_tmp_FqtW9f/MrAz0t.opf',
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': u'',
'search_replace': '[]',
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': None,
'sr1_search': None,
'sr2_replace': None,
'sr2_search': None,
'sr3_replace': None,
'sr3_search': None,
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
Python function terminated unexpectedly: The Magician King: A Novel (The Magicians Book 2)
InputFormatPlugin: MOBI Input running
on /var/folders/k6/rpx44f5j73s69qklkkf0b9wr0000gn/T/calibre_2.47.0_tmp_FqtW9f/kfThgW.azw
Found KF8 MOBI of type 'standalone'
Traceback (most recent call last):
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 209, in main
return run_entry_point()
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 114, in run_entry_point
return getattr(pmod, func)()
File "site-packages/calibre/utils/ipc/worker.py", line 190, in main
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 31, in gui_convert_override
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
File "site-packages/calibre/ebooks/conversion/plumber.py", line 1051, in run
File "site-packages/calibre/customize/conversion.py", line 241, in call
File "site-packages/calibre/ebooks/conversion/plugins/mobi_input.py", line 43, in convert
File "site-packages/calibre/ebooks/mobi/reader/mobi8.py", line 81, in call
File "site-packages/calibre/ebooks/mobi/reader/mobi6.py", line 144, in check_for_drm
calibre.ebooks.DRMError: The Magician King: A Novel (The Magicians Book 2)

Example #3:
Convert book 88 of 113 (Until Proven Guilty (J. P. Beaumont Novel Book 1))
Convert book 88 of 113 (Until Proven Guilty (J. P. Beaumont Novel Book 1))
Resolved conversion options
calibre version: 2.47.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0.0,
'book_producer': None,
'change_justification': u'original',
'chapter': u"//[((name()='h1' or name()='h2') and re:test(., '\s((chapter|book|section|part)\s+)|((prolog|prologue|epilogue)(\s+|$))', 'i')) or @Class = 'chapter']",
'chapter_mark': u'pagebreak',
'comments': None,
'cover': u'/var/folders/k6/rpx44f5j73s69qklkkf0b9wr0000gn/T/calibre_2.47.0_tmp_FqtW9f/rLS4L5.jpeg',
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_split_on_page_breaks': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_toc_at_end': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': u'',
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x115405590>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0.0,
'linearize_tables': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.iPad3Output object at 0x115405b50>,
'page_breaks_before': u"//*[name()='h1' or name()='h2']",
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': u'/var/folders/k6/rpx44f5j73s69qklkkf0b9wr0000gn/T/calibre_2.47.0_tmp_FqtW9f/f_5WpY.opf',
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': u'',
'search_replace': '[]',
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': None,
'sr1_search': None,
'sr2_replace': None,
'sr2_search': None,
'sr3_replace': None,
'sr3_search': None,
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
Python function terminated unexpectedly: Until Proven Guilty (J. P. Beaumont Novel Book 1)
InputFormatPlugin: MOBI Input running
on /var/folders/k6/rpx44f5j73s69qklkkf0b9wr0000gn/T/calibre_2.47.0_tmp_FqtW9f/pf5xeW.azw
Traceback (most recent call last):
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 209, in main
return run_entry_point()
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 114, in run_entry_point
return getattr(pmod, func)()
File "site-packages/calibre/utils/ipc/worker.py", line 190, in main
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 31, in gui_convert_override
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
File "site-packages/calibre/ebooks/conversion/plumber.py", line 1051, in run
File "site-packages/calibre/customize/conversion.py", line 241, in call
File "site-packages/calibre/ebooks/conversion/plugins/mobi_input.py", line 35, in convert
File "site-packages/calibre/ebooks/mobi/reader/mobi6.py", line 148, in extract_content
File "site-packages/calibre/ebooks/mobi/reader/mobi6.py", line 144, in check_for_drm
calibre.ebooks.DRMError: Until Proven Guilty (J. P. Beaumont Novel Book 1)

@kellybell kellybell changed the title Several Amazon/Kindle AZW-format ebooks NOT stripping DRM with Calibre plugin Amazon/Kindle AZW-format ebooks NOT stripping DRM with Calibre plugin Dec 28, 2015
@ElleKayEm
Copy link

Post a log of importing one of the books.

@kellybell
Copy link
Author

Could you tell me how to do that, exactly? Where do I find the log files in Calibre? What I pasted is the log file of the attempted conversion for three books (and they failed bc the DRM was still present), which is the only log I can find.

@ElleKayEm
Copy link

Remove the book from calibre. Then look for "Restart in debug mode" in the Preferences menu. Once it reopens, add the book again. Close calibre, and the log will open.

@kellybell
Copy link
Author

I have 497 books in my kindle library, all of which I downloaded today, none of which are listed by name by the Kindle app (so there's no way to find a book by its name that doesn't strip DRM properly). This is going to be a ridiculous amount of effort. Is there not any other way to give you what you want?

Also when I restart Calibre, aren't I going to lose my Job Log, which tells me which books didn't import/convert correctly? If so, that's going to really suck, since there's no way to export that so I can reference it later...

@ElleKayEm
Copy link

Pick a book you know has DRM. Remove and redownload it in Kindle for PC. Then sort your Kindle folder by date.

@kellybell
Copy link
Author

ok. thanks. I don't have a PC btw - I have a Mac (running latest version of OS X software as well).

Also, do you have an idea of how long Calibre should take to restart in debug mode? I'm at 13 minutes and counting, and it's still stuck on the load screen (says, "initializing user interface). The menus are all there in the background (but they're greyed-out), and I can see the library loaded behind the load screen, but obv nothing works). I'm running 2.47, updated today, no other problems that I've noticed.

Normally, it takes <1 minute to load Calibre.

@kellybell
Copy link
Author

And btw I already tried force-quitting, restarting Calibre normally (it was fine), and then restarting in debug mode again. Both times, same behavior (stuck on load screen).

@ElleKayEm
Copy link

No, it shouldn't take that long to open in debug mode. You could try another way to open in debug mode explained in the plugin Read Me and also on the FAQs page of the Alf blog:

On Macintosh, open the Terminal application (in your Utilities folder).
You should now have a text-based command-line window open. Make sure that calibre isn’t running.

Now type in ‘calibre-debug -g’ (without the ‘’) and press return key. Calibre will launch in debug mode, and various diagnostic information will be written to the terminal window.

Import the ebook into calibre. More information will be written to the terminal window.

Now copy the output from the terminal window.

@kellybell
Copy link
Author

I tried - this is what happened:
$ calibre-debug -g
-bash: calibre-debug: command not found

@kellybell
Copy link
Author

I tried running it from within the Applications directory as well, just in case. No joy. And yes I made sure that calibre was not running before I did any of this.

@kellybell
Copy link
Author

are you sure there shouldn't be a space between "calibre" and "-debug"? That's usually how such commands run, assuming that calibre-debug isn't the name of a separate application...

@kellybell
Copy link
Author

How's this for weird? From Terminal, I get this listing for the main Applications folder (the user-specific Apps folder is empty):
screen shot 2015-12-27 at 8 56 10 pm

But when I take a screen shot of the SAME (blessed) Applications folder, I see calibre just fine (see attached). Not only that, but I have always installed Calibre the default method, i.e. by dragging and dropping the app into the Apps folder included in the installer. So it's not that I'm putting it in the wrong place. But how is it that calibre is hidden in Terminal, and visible in the Finder?

screen shot 2015-12-27 at 8 54 08 pm

@ElleKayEm
Copy link

No space between calibre and -debug. Space before -g.

I'm stumped and headed off to bed. I'll send an email to a friend who's familiar with Mac to see if he can take a look at this and help you.

@kellybell
Copy link
Author

Yes well I copied and pasted your instructions into the CLI, so I know I got it right. I tried typing it manually both ways too, just to be sure. IDK what's going on, but this is very frustrating.

@kellybell
Copy link
Author

I take it back - calibre IS listed in the Apps folder, it's just NOT IN ALPHA ORDER, for some reason, and it's listed as calibre.app (even though I didn't change the name of the app or anything). Is this why I can't get calibre-debug to run? Is there some invisible character or is it the .app at the end that's messing up the CLI instructions? See attached, look at end of list.
screen shot 2015-12-27 at 9 00 36 pm

@kellybell
Copy link
Author

well that didn't work either, so that's not it:
KBELL-MB-3:Applications kellybell$ calibre.app-debug -g
-bash: calibre.app-debug: command not found

And this didn't work either:
KBELL-MB-3:Applications kellybell$ calibre.app
-bash: calibre.app: command not found

I give up. Hopefully I'll hear back from someone tomorrow.

@net-buoy
Copy link

There are some variations as to where the applications are placed.
if you look in /Applications/calibre.app you should see a folder named Contents, and in the Contents folder there should be a MacOS folder, and the actual apps will be there.
$ls -al /Applications/calibre.app/Contents/MacOS
should produce a listing
the info at http://manual.calibre-ebook.com/cli/cli-index.html will be correct except for the initial suggestion that the applications live inside a calibre directory under MacOS

@net-buoy
Copy link

and it is not calibre.app calibre.app is essentially a container.
inside that container you will find all the actual applications.
so ...
$ /Applications/calibre.app/Contents/MacOS/calibre-debug --gui

@net-buoy
Copy link

to find the app you would potentially do
$ sudo find / -name calibre-debug -print
and that will confirm where the application in fact lives on your system

@kellybell
Copy link
Author

OK I got it running. I'm going to see if it will break out of the initialization routine, and will report back.

@kellybell
Copy link
Author

It's converting this time. I double-checked to make sure that this book DID NOT strip DRM or convert to another format without throwing the DRM error BEFORE I tried this. So why does it work in debug mode and not at all in regular mode? Here's the dump:

KBELL-MB-3:MacOS kellybell$ /Applications/calibre.app/Contents/calibre-debug.app/Contents/MacOS/calibre-debug -g
calibre 2.47 isfrozen: True is64bit: True
Darwin-15.2.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '15.2.0', 'Darwin Kernel Version 15.2.0: Fri Nov 13 19:56:56 PST 2015; root:xnu-3248.20.55~2/RELEASE_X86_64')
Python 2.7.9
OSX: ('10.11.2', ('', '', ''), 'x86_64')
Successfully initialized third party plugins: DeDRM
Starting up...
Started up in 24.56 seconds with 11314 books
DeDRM v6.3.4: Trying to decrypt B000OIZSKA_EBOK.azw
Using Library AlfCrypto DLL/DYLIB/SO
MobiDeDrm v0.41.
Copyright © 2008-2012 The Dark Reverser et al.
MOBI header version 8, header length 264
Extra Data Flags: 3
Decrypting Kindle Format 8 ebook: Without Fail (Jack Reacher, Book 6)
Found 4 keys to try after 0.2 seconds
Crypto Type is: 2
File is encoded with PID DHJyxuCKQY.
Decrypting. Please wait . . . . . done
Decryption succeeded after 0.4 seconds
DeDRM v6.3.4: Finished after 0.4 seconds
Added Without Fail (Jack Reacher, Book 6) to db in: 0.6
Added 1 books in 7.4 seconds
Worker Launch took: 1.17990803719
2015-12-27 21:20:37.408 calibre-debug[10539:3259394] Communications error: <OS_xpc_error: <error: 0x7fff77303b90> { count = 1, contents =
"XPCErrorDescription" => <string: 0x7fff77303f40> { length = 22, contents = "Connection interrupted" }
}>
Job: 1 Convert book 1 of 1 (Without Fail (Jack Reacher, Book 6)) finished
Convert book 1 of 1 (Without Fail (Jack Reacher, Book 6))
Resolved conversion options
calibre version: 2.47.0
{'asciiize': False,
...
--- etc --- (removed lots of boring conversion data)
...
Looking for large trees in text/part0017.html...
No large trees found
EPUB output written to /var/folders/k6/rpx44f5j73s69qklkkf0b9wr0000gn/T/calibre_2.47.0_tmp_vjz0fg/toJ90L.epub

2015-12-27 21:21:38.329 Calibre[10581:3260762] deliverNotificationWithTitle()

  • Notification delivered.

@net-buoy
Copy link

rofl, they all point to the same place, as they are placeholders for the gui apps you see if you view the contents from Finder (i.e. if you go to calibre in Finder, right click, and select "Show Package Contents" you will see an icon for calibre-debug, and that icon is linked to the actual command that you can find in /Applications/calibre.app/Contents/MacOS -- I once suggested that the calibre docs be fixed, but my suggestion was apparently not taken to heart :-(

Have you tried clicking on the calibre-debug icon?

@net-buoy
Copy link

I am no expert, but as I understand it, there is no reason for the import to work in debug mode and not in regular mode. So, delete these books, exit calibre, start calibre in regular mode, and add the book, then exit calibre. restart calibre and then try to convert. Beyond that you will have to get the apprentice's help

@kellybell
Copy link
Author

It's working fine now that I know how to run the app - that's not the problem. In fact, I just ran the whole process again, starting from scratch (deleting book in Calibre, archiving and re-downloading it in Kindle so I could tell which book it is, importing and converting) and it ALL WORKS FINE (strips DRM and converts w/out errors), AS LONG AS I RUN IN DEBUG MODE. But it DOES NOT work, for these same books, if I'm running in regular mode. So what do I do? Import all my Kindle books in debug mode from now on?

Is this a calibre bug or a plug-in bug? I would say plug-in, but IDK...

FWIW, I noticed that when I imported the test books in debug mode, they came in as AZW3 format, whereas in the regular-mode import (which failed), the books came in as AZW. Hmmm.

@kellybell
Copy link
Author

net-buoy, I already did all that. please read above. And no offense, but I kinda really DO want to talk to the experts on this one, bc basic troubleshooting I can do for myself. This is definitely a bug of some kind.

@kellybell
Copy link
Author

Oh also - when I ran debug mode from Terminal, it initialized VERY quickly. Same as regular mode. So something is definitely wrong with this version of Calibre in terms of starting debug mode from the GUI. So that's one calibre bug for sure that needs reporting, I assume.

@kellybell
Copy link
Author

but, net-buoy, thank you very much for demystifying WTF was going on with the file structure/app command in the CLI. I've never tun into that one before. Thank you.

@net-buoy
Copy link

Beats me - I suppose you could delete calibre and start from scratch (which is what is recommended in the FAQ if nothing else works)
And if that doesn't work ask Paul

@kellybell
Copy link
Author

well, hopefully the person who started helping me on this thread will come back tomorrow and tell me what they want me to do. Technically, I have a solution now (even if it's a workaround). But I need to know if I should report this to calibre or what...

Thanks again for your help with the package Contents silliness. Ugh.

@ElleKayEm
Copy link

Ok, I hadn't heard any other Mac users not being able to follow the instructions as posted on the website and in the ReadMe file (which I quoted here.) And previously that was the only way to do it since it wasn't available from within calibre.

Yes, I'm aware you have the DeDRM installed and working in debug mode. But if it was working previously, none of the imported books would still have the AZW extension. Even those that didn't have DRM to begin with would now have AZW3, MOBI, or HTMLZ.

@kellybell
Copy link
Author

kellybell commented Dec 29, 2015 via email

@kellybell
Copy link
Author

So are you saying that only books that import in AZW3 format can possibly
be de-DRM'd?

If so that's very confusing, bc I know for certain that most of the books
that were imported as AZW in regular mode and successfully deDRM'd
DEFINITELY had DRM in place to begin with.

A very small percentage were mobi-format books that I'd imported to read on
my kindle from other sources (they imported fine of course). But most were
books bought by me from Amazon.

@ElleKayEm
Copy link

Yes, the extension changes from AZW to AZW3 (or MOBI, etc.) if the DeDRM plugin is working. Some of them must have already been DRM-free.

@kellybell
Copy link
Author

Yes like I said, a small number were DRM-free. But out of the 50+ books
that were successfully converted, MOST of them were NOT DRM-free. I know
this for a fact bc I purchased them myself in Amazon.

@ElleKayEm
Copy link

Ok, in my experience, if everything is working correctly with the DeDRM plugin, the extension always changes from the generic AZW extension to the extension of the actual underlying format. If it has an AZW extension and it can be opened/converted, then it never had DRM to begin with. Again, this is how it should work or, at least, how it has worked.

@net-buoy
Copy link

At this point things might be a bit confused so it might help if you did a little matrix of what works and what doesn't. We have start-up, import, and convert versus calibre and calibre-debug via the finder, and calibre and calibre-debug via the CLI. I thought that I understood you to say that everything worked from the CLI, but nothing worked via the finder. If that is the case there are in my mind two explanations: 1) the contents necessary for the GUI is corrupted (in which case the best thing to do is delete, empty trash, and reinstall from a NEW download, or 2) there is something set differently between the GUI environment and the CLI environment, which is causing the GUI initialization to hang. As I mentioned above, there should be a calbre log accessible via console which should provide some possible guidance as to where calibre is hanging.

However, it seems pretty clear that it is a calbre issue, and the first thing I would do, is make it easy on yourself and delete calibre and empty trash, download a fresh copy, and start it, making sure that it opens properly without the plugin.

Then import a non-DRM book to make sure everything works.

Now, after you are sure that calibre is functioning correctly, download the latest plugin archive and install. Then exit calibre and restart. Now you have calibre working with the installed plugin.

Then, and only then, import and see what happens

@kellybell
Copy link
Author

Using the regular calibre GUI, roughly half the imports stripped DRM
and converted successfully. As best I can tell, the older books are
the only common denominator. The books that are over around 2-3 years
old converted just fine, whereas newer books did not.

  • these older books are showing up as AZW format, yet they still strip
    and convert.
  • The newer books ALSO import as AZW, but DRM is not stripped.
  • The plugin manager says that the plugin is enabled and working fine.
  • when I try running calibre-debug from the GUI, the app freezes at
    the load screen, saying "initializing UI".

Using the CLI to run calibre-debug, everything works as expected.

  • ALL books import as AZW3, no matter their age.

Generally:

  • This install of calibre is <6 months old, on a machine that was
    wiped clean with a brand new OS install at that same time.
  • Calibre 2.47 upgrade was installed yesterday (before plugin)
  • No other calibre plugins are being used except this one (de-DRM),
    and it was just installed for the first time yesterday, following to
    the exact directions for install
  • this machine has not been modded extensively. Nothing else is
    breaking or acting weird either, and it's in use >12 hours/day.

Hope this helps. At this point, I'm planning to just continue using
the CLI, because the ginormous hassle of exporting, deleting,
re/installing etc is just way too much work that I don't really have
time for right now.

Net-buoy, I'll never forget your package contents trick. Muy bueno.
I'm not a Mac coder - my background is in OOP in Java, Ruby and PHP,
with some C++. But I've had a constant, steady stream of Macs since
1984, and I learned my lesson the hard way about modding the
Mac-native environment many years ago; that's why I use a VM. I've
never coded or packaged apps to run on a Mac either, so I've never
learned that trick. So - many thanks.

Thanks again to all, for your kind assistance.

@pdurrant
Copy link
Contributor

Any books imported into calibre that retain their .azw extension have not been decrypted by the DeDRM plugin.

You mention older books and newer books. The decryption key for Kindle books depends on the device they were destined for. Books for eInk kindles require a serial number, books for Kindle for Mac depend on the particular installation of Kindle for Mac.

I suspect that if you wiped all your Kindle books, wiped Kindle for Mac, reinstalled Kindle for Mac and re-downloaded your books, all your problems (except restarting in Debug Mode) would go away.

But that is only a guess, since we haven't been able to see anything wrong in logs.

@ElleKayEm
Copy link

I'm thinking the DeDRM plugin just stopped working partway through the initial import. That can happen when you import a large batch at once.

For future reference, are the instructions on Alf's FAQs page for opening the calibre GUI in debug mode from the Terminal application missing something or just not working in this case?

@kellybell
Copy link
Author

Below

I'm thinking the DeDRM plugin just stopped working partway through the
initial import. That can happen when you import a large batch at once.

That was my initial thought too, though individual imports of books still
gave same results, both ways. Paul says there was no initial DRM on those
that succeeded bc they imported as AZW, though many in that list were
bestsellers, like some of the Jack Reacher series. And I've never known
Amazon not to vociferously protect its bestsellers. Not to mention, when I
import those same books in debug mode, they import in AZW3 format; whereas
GUI import brings all in as AZW, with DRM and without. So something IS
clearly wrong with the standard-mode de-DRM import process.

For future reference, are the instructions on Alf's FAQs page for opening
the calibre GUI in debug mode from the Terminal application missing
something or just not working in this case?

The faqs DO need to be edited. See net-buoy's comment in thread - he's
pointed it out to Alf before but no result.


Reply to this email directly or view it on GitHub
#52 (comment)
.

@ElleKayEm
Copy link

Yeah, a Jack Reacher book definitely had DRM since it's from one of the big publishers.

A reinstall of calibre might fix your problem. (Your library and plugins will remain.) But if you've already got your books DeDRMed, I understand not wanting to bother.

Net-buoy mentioned telling the calibre folks that their info was incorrect. But the Apprentice Alf blog and Apprentice Harper's GitHub here aren't connected to calibre. Alf's FAQs say to open the Terminal application and type calibre-debug -g. What should it say?

@kellybell
Copy link
Author

kellybell commented Dec 29, 2015 via email

@pdurrant
Copy link
Contributor

It should say just to use the restart in debug mode command on the preferences menu.

@kellybell
Copy link
Author

kellybell commented Dec 29, 2015 via email

@pdurrant
Copy link
Contributor

My feeling is that this shows a problem with your system that needs to be sorted out before doing any further troubleshooting.

@ElleKayEm
Copy link

Ah, in Windows, the calibre directory is in the system path so it doesn't matter what folder you have the command prompt open to.

Agree with Paul that the Alf blog should give instructions for the simpler GUI method (since it should work if there isn't a larger calibre issue). Those instructions were just written before that was available.

@kellybell
Copy link
Author

kellybell commented Dec 29, 2015 via email

@pdurrant
Copy link
Contributor

Your experience of using the GUI restart in debug mode does not match mine. We are both running the same system software and calibre version. Therefore there's a difference that we don't know about, but that difference is causing you problems. Since the problem persists when DeDRM isn't installed, I suggest, again, that you work with the calibre support at MobileRead to find the cause of the problem and fix it.

I suspect that that will fix your problem using DeDRM.

@apprenticeharper
Copy link
Owner

Remove DeDRM from your calibre installation. Does it now restart in debug mode from the menu? If not, get help at the calibre forum, as that means there's something wrong with your installation of calibre.that has nothing to do with DeDRM. Open a new issue once that's sorted.

@kellybell
Copy link
Author

Will do.

Kelly Bell
Founder and CTO
Gotham City Drupal
917.446.1555

On Jan 13, 2016, at 22:41, Apprentice Harper notifications@github.com
wrote:

Remove DeDRM from your calibre installation. Does it now restart in debug
mode from the menu? If not, get help at the calibre forum, as that means
there's something wrong with your installation of calibre.that has nothing
to do with DeDRM. Open a new issue once that's sorted.


Reply to this email directly or view it on GitHub
#52 (comment)
.

@AdamWill
Copy link

AdamWill commented Mar 19, 2017

FWIW, I just had exactly the same experience as the reporter here: I bought four books from Amazon.com , tried to import all four to Calibre in 'regular' mode, and only one got decrypted (showed up as AZW3 and could be converted to epub), the other three did not. I tried removing those three and importing all four again (since I can't tell which of the four is the one that was already imported successfully), still didn't work for the other three. Then I restarted Calibre in debug mode and tried importing again, this time the three recalcitrant books got decrypted fine. Strange...

@ElleKayEm
Copy link

This was a calibre issue, not specific to the DeDRM plugin. Best place to look for the solution is over at the MobileRead calibre forum.

@apprenticeharper
Copy link
Owner

It is usually better to open a new issue rather than comment on an old issue (2015!) where the symptom might be the same, but the cause is almost certainly not.

I would be interested in working with you, AdamWIll, to work out why you had this experience. If you're willing to spend a little time on it, please open a new issue.

@AdamWill
Copy link

If I get a bit of time I will, but I'm not that invested, to be honest - I got it working and that's all I needed. I just figured it was worth dropping a note since I came across this report and it was pretty much an identical experience.

@apprenticeharper
Copy link
Owner

I'm happy to work with anyone on possible differences in operation between normal and debug calibre modes, but in a new issue, please.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants