Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Any timeline to DeDrm to work with latest Kindle for PC Version? #240

Open
MadMik1964il opened this issue Jan 4, 2023 · 70 comments
Open

Comments

@MadMik1964il
Copy link

Question / bug report

As for today, Amazon doesn't let older versions of Kindle for PC to download files. And current version of DeDrm doesn't work with new ver 1.39 files (files are imported as KFX-ZIP). Please help!!!

Which version of Calibre are you running?

6.1

Which version of the DeDRM plugin are you running?

v10.0.3

If applicable, which version of the Kindle software are you running?

1.39

Log output

No response

@noDRM
Copy link
Owner

noDRM commented Jan 4, 2023

Does this apply to all books? I believe there's some books that require the newest Kindle 4 PC, but most books should still work with the old one.

@federicorosso1993
Copy link

Does this apply to all books? I believe there's some books that require the newest Kindle 4 PC, but most books should still work with the old one.

I tried a few, right now I'm able to download books with the old version but not manga/comics. Luckily the manga I bought even in the new version come without KFX (I was able to do it both for italian manga and Claymore that I own the english version).
Still last week I was able to download manga with the old version so I think they will change it in a short amount of time.

@iamsivil
Copy link

iamsivil commented Jan 4, 2023

Seems like this may be a relatively new thing. I was able to download a book that released approximately 30 hours ago on Kindle for PC 1.17 as of this morning but 2 books released since then (one 20 hours ago and the other 6 hours ago) were blocked from being downloaded and required upgrading to the latest Kindle version.

@j-howell
Copy link

j-howell commented Jan 4, 2023

I am curious what Amazon has changed. I tried some very recently published books and they all downloaded for me. Can someone with this problem provide links to some of the books that are not working for you using K4PC 1.17?

@MadMik1964il
Copy link
Author

Actually I was using 1.26 for a year or so and it was fine. Until yesterday. Tried to get few books, and it is not working anymore.
I returned them, but we do need a solution.

@ElleKayEm
Copy link

Still hanging in here with Kindle for PC 1.17. See noDRM's response to a similar question here: #239 (comment)

@MadMik1964il
Copy link
Author

Well. looks I might need to buy Kindle device, damn it...

@ElleKayEm
Copy link

That won't help either if the book is only available in the latest KFX format.

@UnKnoWn-Consortium
Copy link

UnKnoWn-Consortium commented Jan 4, 2023

I am not entirely sure but it appears they have switched off azw download at old Kindle PC versions for books published after 2nd January 2023 (give or take).

Edit: AZW download for transfer over USB to Kindle devices are still working as of now. In other words we lose access to high-res resource packs (when available)...

@MadMik1964il
Copy link
Author

Well, looks I have to find spare $200 bucks to buy the device (used my phone for reading so far)...

@ElleKayEm
Copy link

Again, you don't have that option if the book is only available in KFX format. And if you're going to get a Kindle just for download and transfer via usb, you can get one for a lot less.

@j-howell
Copy link

j-howell commented Jan 5, 2023

I am curious what Amazon has changed. I tried some very recently published books and they all downloaded for me.

I take back what I wrote earlier. Version 1.17 is no longer letting me download newly published books.

@ElleKayEm
Copy link

You can still download older books with 1.17? Can you download newly published books via one of the other methods for avoiding kfx? Or at least the latest kfx?

@j-howell
Copy link

j-howell commented Jan 5, 2023

You can still download older books with 1.17? Can you download newly published books via one of the other methods for avoiding kfx? Or at least the latest kfx?

Only newly published books seem to be affected. New purchases of books published last year still work in 1.17. Download & transfer targeted to an e-ink Kindle still works even for the newly published ones. I will try other app versions tomorrow.

@j-howell
Copy link

j-howell commented Jan 5, 2023

I tested Kindle for PC versions from 1.17 through the current 1.39. Only version 1.39 is capable of downloading newly published books.

@ElleKayEm
Copy link

Thanks for testing and reporting back.

@VorpalQ
Copy link

VorpalQ commented Jan 6, 2023

Again, you don't have that option if the book is only available in KFX format. And if you're going to get a Kindle just for download and transfer via usb, you can get one for a lot less.

I decided to test this, since I have an old Kindle Paperwhite laying around. So I dusted it off and registered it to my Amazon account. I bought and downloaded a book that was published on Jan 4th, and so could only be downloaded on 1.39. When I used the Kindle PC App, the book came in KFX-ZIP and could not have the DRM removed. When I instead used the 'Download & Transfer via USB' option from 'Content & Device Management' on Amazon's website, it downloaded an AZW3 file that I was then able to import to Calibre and remove the DRM from.

@MadMik1964il
Copy link
Author

Thanks for testing. I"ll go look for old Kindle on ebay.

@ElleKayEm
Copy link

Yes, it is only blocked from "download & transfer" if it's a book which is only available in the latest KFX format. A very small category usually expensive textbooks and certain languages.

@amnotbananaama
Copy link

amnotbananaama commented Jan 6, 2023

Echoing the OP, now that the problem is more clearly defined. While the current issue (some Kindle books published after 1/3/2022 are KFX format only and cannot be stripped by DeDRM) is minor at present, the number of inaccessible books will continue to grow with time. Not to mention, there remains the possibility of Amazon switching to KFX format only for all books in the future. Is there any foreseeable way that DeDRM could support the KFX format in the future?

Of course, I realize that is a lot to ask, and I would like to sincerely thank the maintainer(s) for their work on DeDRM, which has been invaluable to me as a user.

@ElleKayEm
Copy link

noDRM said in another issue:

The past has shown that A) figuring out how to break the new Amazon DRMs is a ton of work, and B) Amazon already has multiple new DRM versions "available" that they can switch to in a day or two, so the effort would be totally wasted.

For now, even most new books are still available in the older decryptable format. It's just that the only way to get them in that format is now "download & transfer via usb" for an E-ink Kindle. Only a few books are truly KFX only. On the flip side, there are a few books (manga and comics) which don't come in KFX at all (yet). Those can be downloaded with the latest Kindle for PC and still be DeDRMed. Trouble is all of this could change and it's hard to tell before buying what you might get stuck with.

@federicorosso1993
Copy link

I think once I cannot remove the drm anymore I will switch to buying from kobo since right now Adobe drm is easier to remove but I also think the only 2 ways to deal with this kind of new drm from amazon in the future will be to:

  1. Figure out the pattern to deal with KFX drm (I don't think it's a random/manual way to encode it every time in a really different kind of way... I think there should be a pattern or they would not be able to change it so fast).
  2. Go full OCR so find a way to automate the process to acquire amazon ebooks in image format and OCR them. Right now it's possible to take screenshot from both the Kindle PC app and the kindle website (read.amazon.com) so it should be possible to automate the process. This should be possible for kobo too. The only problem with this is that OCR is not perfect and screenshot quality is not perfect for manga/comics, the good thing is that even without any working tool it will still be possible to do something like that manually.

I don't think it's a good idea to buy old kindle right now, they can change the "download & transfer via usb" anytime they want, just buy from kobo and remove adobe drm instead of amazon drm.
And I'm not sure manga and comics will stay KFX-free forever.

@j-howell
Copy link

j-howell commented Jan 6, 2023

  1. Figure out the pattern to deal with KFX drm (I don't think it's a random/manual way to encode it every time in a really different kind of way... I think there should be a pattern or they would not be able to change it so fast).

Amazon has implemented around 40 different DRM algorithms for KFX. They are built into the various Kindle apps and device firmware as highly obfuscated code.

Each algorithm is identified with the letter V followed by a number. So far DeDRM handles V1 though V10. Amazon is currently using V11, which DeDRM attempts to decrypt but does not do correctly.

As things stand now anyone who does figure out how to defeat the current V11 DRM has a strong incentive to keep it to themself. Keeping it quiet would mean that at least they could make use of it. Making it known to the world would just result in Amazon switching to V12, causing all of that hard work to be for nothing.

In order to defeat KFX DRM in DeDRM all of the thirty or so remaining DRM algorithms would need to be reverse engineered. If any are gotten wrong, which is likely without a way to test them properly, Amazon would immediately use that one for now and then keep adding more new ones over time.

Even if someone managed to get all of the algorithms correct, DRM removal would only work until Amazon deployed software with more new algorithms and blocked downloads to the older software.

@noDRM
Copy link
Owner

noDRM commented Jan 6, 2023

Someone on MobileRead posted that apparently Kindle 4 Android in version 7.6.0.39 can still download books from 2023, and downloads books in one of the older KFX formats where the DRM can still be removed; maybe that info is helpful for some people. Though I have not verified if that does actually work; and you'll need to either run that on an Android emulator or on a rooted Android device to be able to extract the key files. So it's probably not a solution for the average user, and I'd imagine that Amazon would force Android users to update soon, too.

As for the different KFX versions, I have to admit I didn't really look too closely at the Amazon DRM in the past. Interesting that the tools only work up to V11 - looking at the OBFUSCATION_TABLE in ion.py I would have expected that all versions in that list can be DeDRM'ed and that the newer versions of K4PC / KFX would be using version numbers not yet in that list.

But I guess when Amazon wasn't yet using versions >10 there was no way to test that DRM removal so they probably have some additional protections on these versions, making that table useless. Interesting.

@j-howell
Copy link

j-howell commented Jan 6, 2023

But I guess when Amazon wasn't yet using versions >10 there was no way to test that DRM removal so they probably have some additional protections on these versions, making that table useless. Interesting.

I looked when the last DeDRM stopped working for KFX and the version I saw in use that could not be decoded was V11. Either the table is wrong or there is something more that needs to be done to decode that DRM version.

Just a suggestion, but it would be nice if the code that handles KFX DRM would print out the DRM version number. That would make it possible to tell which version was being used when dealing with reports of failure.

@ElleKayEm
Copy link

Even if KFX DRM was figured out, I'd rather have azw3 for conversion to epub. If they make it too difficult to get, I'll say goodbye to Amazon as I did with B&N many years ago. Kobo gets most of my business already.

@UPSA007
Copy link

UPSA007 commented Jan 9, 2023

Ya but the problem with kobo is their prices. Most of the books cost anywhere from 10 to 20 percent higher and sometimes double for no bloody reason. Google Play books also has the same problem. I mean who decides the price publishers or the platforms.

@shamanNS
Copy link

shamanNS commented Jan 9, 2023

@noDRM

Someone on MobileRead posted that apparently Kindle 4 Android in version 7.6.0.39 can still download books from 2023, and downloads books in one of the older KFX formats where the DRM can still be removed; maybe that info is helpful for some people.

Hello from "someone" 😄

@ElleKayEm
Copy link

Ya but the problem with kobo is their prices. Most of the books cost anywhere from 10 to 20 percent higher and sometimes double for no bloody reason. Google Play books also has the same problem. I mean who decides the price publishers or the platforms.

I don't see that that often for the books I buy. But Kobo has price matching. You get the difference (plus 10% of the competitor's price) back as credit.

@federicorosso1993
Copy link

I don't see this at all for book and manga that I buy (same price from Amazon and Kobo) but there is no price matching here.
I can generate amazon code from my bank account to buy things (and ebooks) on Amazon with a 3% discount... That's why I usually buy from Amazon instead of Kobo (and since kobo is not drm free but I still need to remove the drm I prefer Amazon). But if there will be no way to remove drm from Amazon I will switch to kobo even if I have to pay 3% more is not a big deal.

@federicorosso1993
Copy link

I know that as a kindle 4 user I'm unable to download some Manga/comics (but they are the one that are not with KFX so I can download and remove drm with the last Kindle pc version)

@chocolatechipcats
Copy link

chocolatechipcats commented Jan 12, 2023

According to users in the MobileRead thread, "download and transfer" is now blocked for Kindle Unlimited borrows. I don't know if this indicates they plan to add more restrictions.

I just tested this with a regular fiction book published today on KU. I am able to still 'download and transfer' and copy it to my Kindle. I think more information is needed on this. Is it a specific type of book perhaps? (Comics or Manga maybe?)

Did purchase it, or borrow? That's something I specifically asked about (no Kindle or Prime to test myself); it applies specifically to borrows (for now).

@VorpalQ
Copy link

VorpalQ commented Jan 12, 2023 via email

@j-howell
Copy link

j-howell commented Jan 12, 2023

I can still use Download & Transfer for Kindle Unlimited borrows. Only a few people have reported it being blocked so far. I suspect that it is being rolled out gradually to reduce the impact on customer service.

@ElleKayEm
Copy link

Or maybe older Kindles that don't do azw3 and these aren't available in mobi? (haven't looked at the MobileRead thread yet)

@chocolatechipcats
Copy link

This post points out Amazon help page stating 'download and transfer' unavailable for borrows:
https://www.mobileread.com/forums/showpost.php?p=4288991&postcount=286

@ElleKayEm
Copy link

Well, that would be good for authors in KU since they only get paid by the percent of the book you read.

@trulow
Copy link

trulow commented Jan 13, 2023

I can still use Download & Transfer for Kindle Unlimited borrows. Only a few people have reported it being blocked so far. I suspect that it is being rolled out gradually to reduce the impact on customer service.

Seems like the limitation is primarily for books published on or after Jan 3, 2023

@chocolatechipcats
Copy link

I can still use Download & Transfer for Kindle Unlimited borrows. Only a few people have reported it being blocked so far. I suspect that it is being rolled out gradually to reduce the impact on customer service.

Seems like the limitation is primarily for books published on or after Jan 3, 2023

Tested KU borrows from before and after that date? The link posted above indicates that borrows unavailable for transfer regardless of date. The January 3 cutoff seems to be for any book being downloadable on K4PC 1.17.

@ElleKayEm
Copy link

ElleKayEm commented Jan 14, 2023

Yes, no KU borrows at all for download & transfer now. (Which is fine. Amazon should have done that for authors all along.) Jan 3 pub date and later purchased books can't be downloaded with any of the older Kindle for PC/Mac -- need to do download & transfer to get in decryptable format. ETA: unless it's a book that doesn't come in KFX at all.

@VorpalQ
Copy link

VorpalQ commented Jan 14, 2023

Seems like the limitation is primarily for books published on or after Jan 3, 2023

Tested KU borrows from before and after that date? The link posted above indicates that borrows unavailable for transfer regardless of date. The January 3 cutoff seems to be for any book being downloadable on K4PC 1.17.

image
Yeah I can confirm it no longer works for KU borrows.

@Patitapetit
Copy link

  1. Figure out the pattern to deal with KFX drm (I don't think it's a random/manual way to encode it every time in a really different kind of way... I think there should be a pattern or they would not be able to change it so fast).

Amazon has implemented around 40 different DRM algorithms for KFX. They are built into the various Kindle apps and device firmware as highly obfuscated code.

Each algorithm is identified with the letter V followed by a number. So far DeDRM handles V1 though V10. Amazon is currently using V11, which DeDRM attempts to decrypt but does not do correctly.

As things stand now anyone who does figure out how to defeat the current V11 DRM has a strong incentive to keep it to themself. Keeping it quiet would mean that at least they could make use of it. Making it known to the world would just result in Amazon switching to V12, causing all of that hard work to be for nothing.

In order to defeat KFX DRM in DeDRM all of the thirty or so remaining DRM algorithms would need to be reverse engineered. If any are gotten wrong, which is likely without a way to test them properly, Amazon would immediately use that one for now and then keep adding more new ones over time.

Even if someone managed to get all of the algorithms correct, DRM removal would only work until Amazon deployed software with more new algorithms and blocked downloads to the older software.

@j-howell how did you learn this info?
Thank you

@ElleKayEm
Copy link

Apprentice Sakuya, the person who figured out the first few versions, reported back in 2020 that they could tell that there were 28 versions at that time (Kindle for PC/Mac 1.27). Only 1-4 were in use back then.

@j-howell
Copy link

@j-howell how did you learn this info? Thank you

A combination of ways. The discussions of the DeDRM plugin in Apprentice Alf's Blog, Apprentice Harper's GitHub, and NoDRM's GitHub have provided a lot of information over the years. Inspection of the code within DeDRM plugin itself shows what has been done so far. And I have looked at various Kindle apps and firmware to discover what I can about how they work.

@deirdrew
Copy link

Well. looks I might need to buy Kindle device, damn it...

I bought a new 5th Gen Kindle cheap off ebay, manually did all five software updates available from Amazon's website, then was able to register it by using an Amazon OTP number instead of my regular password. You are texted the OTP number right after you change your Amazon account password at their website. Voila, back using KU on new books after Amazon send them to my 5th gen Kindle and I retrieve them to my PC using USB. whew

@ElleKayEm
Copy link

Be aware that the authors don't get paid when you do that.

@deirdrew
Copy link

deirdrew commented Jan 21, 2023 via email

@ElleKayEm
Copy link

I prefer to give people the benefit of the doubt.

@MadMik1964il
Copy link
Author

Isn't Kindle device getting same new KFX files?

@deirdrew
Copy link

deirdrew commented Jan 21, 2023 via email

@recreational-grade
Copy link

I think it depends on the firmware version, I'm running 5.14.3 on a 7th Gen Paperwhite and I'm getting KFX files.

@ElleKayEm
Copy link

Newer Kindles will. Needs to be old enough not to get KFX at all or still get KFX that can be decrypted.

@deirdrew
Copy link

deirdrew commented Jan 21, 2023 via email

@deirdrew
Copy link

deirdrew commented Jan 21, 2023 via email

@chocolatechipcats
Copy link

chocolatechipcats commented Feb 17, 2023

I may have found a workaround involving an older version of Kindle for Android. (Android device not required; I used the Bluestacks emulator!) Start at this post:
https://www.mobileread.com/forums/showthread.php?p=4299269#post4299269

@hirokihokari
Copy link

when i try an older version of Android Kindle, downloading fails with "An app update is required to read this title." The product page lists Android app but does say "latest software update is required".

What is my option here? The book's published in 2021.

@ElleKayEm
Copy link

Possibly a KFX only title -- in which case, maybe no option. Willing to share the ASIN?

@hirokihokari
Copy link

Sure thing. B092MF6W68

@dreikin
Copy link

dreikin commented Feb 24, 2023

That's a recentish Pearson Higher Ed title, so it's KFX-only and latest encryption only.

@hirokihokari
Copy link

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests