-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Need help with understanding JWPUB format #1
Comments
Hi! I had this idea (scrapping jwpub files) somedays ago and was searching for anything about these JW Library files. Appearly, these files have some linking directly with jw.org, but even then, I don't got anything about how this linking works. By the way, I was thinking that bytecode should be a id from words table too, but I also don't think this is a directly id, maybe have some instructions that JW made for it. You're a Jehovah Witness? |
I even sent a couple of emails with a request for documentation, but got a response that said that they are unable to answer my question from this email address. So my idea was to call the number from https://www.jw.org/en/jehovahs-witnesses/contact/united-states/, but recently I didn't had much time, so I didn't do that. And yes, I am |
I've put a lot of time into understanding this format, but still no results worth showing. It's sad that most of the new publications are PDF/JWPUB only, PDF just doesn't scale well, and JWPUB is ugh., I still have an idea - scraping |
Haha, I don't think they would simply give us their code, sadly. Anyways, scrapping After all, all we can do is trial and error. We have at least a hint, that files works like a Epub, with XML files inside it, the difference it's hard modified and for some reason there's binary code that isn't a match with a list of words. I will try doing something with my knowledge with Python, I don't know that I will help in something, but at least I will try for fun. I really like the fact to use JW Library in PC, and it's sad that Watchtower don't have ported it to some Linux distro. I don't think it will go for a long time, maybe some day they release a version for a famous Linux distro. A final question... I saw with your project that you use the app API from JW but, this is allowed? Isn't a violation from some of the App's terms of use? |
Hello! So, I made some experiencies with the JWPUB file to know how it works and I think I got some hot things working! First of all, content is directly related with the page and don't accept something new in (maybe because content have a fixed size bytes and I inserted more than that? I don't know). Furthermore, the Words table don't work the way we thought, I changed a word in this table and all I got is the way I find it on the book, now I need to search by "subjecters" instead "subject", and after all, the word in the documents keep the same. So, after all, I got a "How to Remain in God's Love" Book with the subjects section with title "Edited Subjects" and a blank "Letter from the Governing Body". EDIT: I read the documentation that you gave, maybe the begin and end can be the initial byte and the final byte to be converted, but there's the question: Converted in what if it's not an index from words table? |
Wow! That's great! I lost so much time with the Have you seen things below the sentence You also need to have there:
Which is quite short, so my guess was that it use Or another scenario the |
Okayyy I think I got a problem with the customized JWPUB and I don't know what exactly was charging it. I saw what was in the jwpub converting doc before and yeah, it could be it but... There's something strange with it and I don't know what exactly happened. I changed a lot of things in the original db because I thought I was compacting it with a new jwpub file with my code but no, and when I fixed that, I had changed a lot in the DB and I think I got a corrupted publication (Or modified so long that it's don't load anything). Remembering that I changed just one content column. But this is really strange, I didn't saw that yesterday but even then it's strange how it's going. After all, there's a lot of things working behind the jwpub specifications, there's even a schema specification for publication view and, with words table, there's some strange tables that's is like a pre compiled search. I'm really thinking about what some of a reading program forum responded to a request to create a support for the JW files, they said these files have requests for the JW API. I don't trust in everything, but this really was stuck in my mind, but even then doesn't make any sense, why a 100mb or + will need from JW? And if it's, how the pioneers book are distributed? |
I'll check the network thing tonight.. I'll download a publication and just watch for the traffic in burp suite, that should clarify if the requests are sent there or not. |
So first of all, I had some problems with android studio, then it was just late and I forgot to reply. |
Yea.. but if we fail that's the only option.
Not really - it can be converted on the go, and then just kept in some html format.
That's sad :( I wish that there would be a decent watchtower library app made with gtk ;p
Since I don't reupload the content, it is legal, but I'm not a lawyer https://www.jw.org/en/terms-of-use/ and even if it's against the terms.. sigh. I'm not switching back to android, so I'll continue to develop this app. (sorry for late reply.. I missed this comment) |
No no! It's okay, brother! I too don't have so much time for searching more these days, after all, I'm still have 15 years old and have some homework to do here for school. ^^ But I will still following the project flow, if I can get something new here, I make a new response on this issue. And if you can't got any new thing from the JWPUB convertion, you still have a more easy task to do, like the video player :) (I really like the way the PC JW Library app can be easily "hacked" to have a new video on, lol) |
Ignore what this weirdo said. It's here: https://github.com/Miaosi001/JW-Library-macOS/blob/main/JWLibrary/SubViews/PubbView.swift |
@MrCyjaneK did you figure out how to read Document.Content? |
I'm not working on this app anymore, spending time on open source alternative to something that is clearly using DRM when it shouldn't (can somebody give me one single reason for which it is worth to encrypt such content when it is freely available?) And the elephant in the room. WHY isn't the app open source in the first place? Until somebody gives me answers to that questions I'm not going to work on this project. wol.jw.org is enough for me.
|
Security? If anyone could get a publication and easily edit it the risk of spreading misleading information would be very high. |
@darioragusa As they can do with .epub, .mobi, and .pdf. Also there is a tool for that used widely in the internet, you can sign things with PGP that would allow 3rd party apps to be developed and would cause less risk (currently we can edit the publications - drm is defectivebydesign.org). |
@MrCyjaneK I know you can edit the other formats without problems but the most of us use the JW Library app. I download a jwpub knowing that it comes from jw.org or the app and I trust the content. It's not a random txt file sent by a random guy opened with Word or Adobe Reader which may or may not contain the correct informations. An example: if I send to my grandma an EPUB she my be not able to open it but, if I send a jwpub she taps the file, a trusted app she always use show up and for her it's all ok: a normal article with the reliable content that is supposed to be there. A jwpub can still be edited but it's not a thing that anyone with basic knowledge of Word can do: less editors -> less edited files. Perhaps I'm totally wrong but those are my two cents. |
The thing is current method allows editing, and signing would make it impossible while allowing moders like us to easily read the content |
I don't know much about signing files, but I guess that the app should have a key and using this key with (something, idk) they should get a value. It's like checking the hash? If a bit changes the value is different? |
It's like checking if the content was modified, the content can be signed to verify that it was created by somebody and after modding it the signature will not match. It's like encrypting but you can see the content and can't modify it. |
Ok, but this way they shouldn't save the signatures for every version of every article in every publication in every language? |
pgp signatures do not add a lot of extra size to publication so I don't consider this a problem. (hence you could sign a sha512sum of publication and get similar result) + you can sign them as they are served to download. |
If the signature is stored with the publication what stops me to change it? |
You can change it - you can even sign it with your key but it will be invalid
|
I have a database extracted from the jwpub file. Like for example, nwtsty.db. Some fields have BLOB and I can't see the data using sql. How can I see the data? |
@GuyMicciche #1 (comment) ChatGPT is actually pretty good in converting that message into code (worked for golang) |
what are you working on btw? |
If you need help knowing how a JWPUB works or the styles you can use, I recommend you take a look at the Reviw wiki |
I have a python script that I already got headway with. Give a link to the db file, and it generates the correct hashes. But right now I'm having trouble getting the data from "Content" blob. should I share the script here? |
Hello @MrCyjaneK,
I want to keep a local JWPUB archive. Do you have any information on how you were able to load the entire catalog (I only want it in one language – that's only ~4 GB). I am using the API url How have you implemented this? Thanks 🤗 |
@geimist you need catalog.db
gzip -d it and enjoy sqlite catalog :) |
Yes, I have that. |
I have a working python version. |
@geimist - provide all parameters that you find to the url, if something is undefined provide an empty string. |
I would like to know if there is a way to convert .doc (win word) or pdf files into .jwpub format, or if there isn't a system already done online, an idea on how to do it |
1 similar comment
I would like to know if there is a way to convert .doc (win word) or pdf files into .jwpub format, or if there isn't a system already done online, an idea on how to do it |
Yes, MaxBas70. There is a way. You have to create the SQLite database that defines the publication (with all its content, metadata, words index etc.), add the artwork as separate files, pack everything together, attach some more required metadata, pack it once more and there you have the JWPUB file. Since the JWPUB file is not a publicly documented format (most of us here have looked into JWPUB files to see how they are structured without any support from brothers serving in MEPS Programming), there isn't any publicly available piece of software that takes a random Word document and converts it into JWPUB. |
Thanks Arthur, at this point I ask you if it was possible to ### create an online converter that launches a procedure for the conversion. I know many who would be interested and would be willing to pay a monthly or processing fee to have, for example, speech outlines directly on JWL. I can provide the technological structure to do it and the marketing to sell it. If you give me a hand, if it's possible. |
Aren't some of the public talk outlines already available in JWPUB on JW Hub? |
it is already rather in depth explained in this very issue, I suggest reading all of the comments in here. |
I'm not sure if creating a "JWPUB online converter software" is legal to charge money. You won't find a direct WORD or PDF to JWPUB converter. This specific conversion can only be done through MEPS. MEPS provides an interface similar to that of WORD and allows the creation of digital publications. After you finish publishing and sending it to print, you can export it in formats such as PDF, WPUB, EPUB, and JWPUB. Have you already tried Reviw and its wiki? It can help you create a JWPUB easily and for free. |
Not all, some are in pdf or word. The problem is that if I download a pattern, I am forced to use it pure and make the notes alongside. Instead it would be more useful to upload my word or pdf outline, which I customized and have it in the main window. Or it would be useful to have a system to position the free text box as you like as in the W study. Can anyone explain to me how to use Rewind in practice? |
Good evening, I'm creating a custom .jwpub, but I'd like some help with the .db tables. There are some that I don't know what to put: sqlite_stat1, word, searchindex, DocumentParagraph, and many others. I already have the html, manifest and blob, but it doesn't open. I believe there's something missing. I was told it would be the paragraph, but I don't know how to fill it in. If you can help, please. Thank you. I can share the .zip for help (I use google translator)😂 |
Hello there. I'm working on a service that helps with a personal study and I also need help / guidance around how to make my own JWPub file. Idea of my service is the following: save your writings while listening on a congregation / assembly or while listening a video in the service. It splits your writings with AI into pieces so you can have your own index: when you read The Bible and tap on the gem icon, you can see part of your writing, tap on it and see the whole writing - of a speech or a video. My progress so far:
Details
Then I zip zipped db + the manifest file into one "mrsrch.jwpub" file. And it can't be installed on JW Library at this point. I feel a bit lost on what I can do next to make it work. Do you have any ideas? I'd appreciate any help 🙏 |
Te recomiendo la wiki de Reviw: https://www.github.com/livrasand/Reviw/wiki its free |
Thanks I'll give it a try. Though, my Spanish to English translator sometimes gives me something crazy. Maybe I'm missing something, but at least from what I understood, Reviw wants me to send them files so they create JWPub file for me, via GH issues. That's not how I want to approach the issue. But maybe I didn't understand their wiki. |
Is something like that, but Reviw i use to create a .db file, after that i create my own .jwpub (CyberChef i use for create my blobs). He helping with somethings i dont know, i believe he help you with your file and changing ideas. That is my .jwpub, i'm working does 2 weeks and work well. He help me with something. So i believe is a good thing use there for asnwer your questions. RefDocument table i create one, is complicated but cool. I will add other talks, for now I have only made 1 |
If you need more specific help, I can help you. Me and the Reviw community have created all these JWPUBs: livrasand.github.io Send me an email and we can talk about how to help you |
hi everyone what's up ? |
Sorry for late, I was in vacation. What is "design interface (Ui)"? I'm not dev, just a normal guy |
It's User Interface, probably they were referring to Graphical User Interface, the window that the application show the text and images |
I have no idea how to get words out of
Content
in.db
file located in jwpub archive. what I know. So any help is needed.The text was updated successfully, but these errors were encountered: