EPUB format

[lvh]

Right now the only high-quality format we have is PDF. People don't like PDFs on e-book readers. They want EPUB instead. Turns out that producing a good EPUB file is hard.

[lvh]

FWIW, pandoc just totally choked on the TeX:

[~/Code/Crypto101/book]$ pandoc -v
pandoc 1.8.2.1
Compiled with citeproc support.
Copyright (C) 2006-2011 John MacFarlane
Web:  http://johnmacfarlane.net/pandoc
This is free software; see the source for copying conditions.  There is no
warranty, not even for merchantability or fitness for a particular purpose.
[~/Code/Crypto101/book]$ cat Crypto101.tex | pandoc -f latex -t epub -o Crypto101.epub
pandoc:
Error:
"source" (line 30, column 1):
unexpected "\\begin", "\\end" or '\\'
expecting block, white space or end of input
Literate haskell feature, Pattern match failure in do expression at src/Text/Pandoc/Readers/LaTeX.hs:895:3-30, not a footnote or thanks command, Pattern match failure in do expression at src/Text/Pandoc/Readers/LaTeX.hs:444:3-19 or Pattern match failure in do expression at src/Text/Pandoc/Readers/LaTeX.hs:496:3-18

Apparently pandoc doesn't agree that \maketitle is a thing?

[acecodes]

You could run it through Calibre and then clean up the resulting ePub. I just did a preliminary test and it seemed to handle the TeX without any issues. The formatting (headers, spacing, etc.) is not very good, but that's only because converting from PDF to almost anything else is a massive pain. If you have any kind of skill with HTML and CSS, you could fix it yourself in a few hours. I've done this with tons of books before, so if you have any questions feel free to ask.

[lvh]

Right; I'd expect LaTeX to EPUB to work better since it's not typeset to a
particular page size. Can the Calibre process be automated? I only know it
as a GUI app.
On Mar 20, 2014 8:08 PM, "Ace" notifications@github.com wrote:

You could run it through Calibre and then cleaning up the resulting ePub.
I just did a preliminary test and it seemed to handle the TeX without any
issues. The formatting (headers, spacing, etc.) is not very good, but
that's only because converting from PDF to almost anything else is a
massive pain. If you have any kind of skill with HTML and CSS, you could
fix it yourself in a few hours. I've done this with tons of books before,
so if you have any questions feel free to ask.

Reply to this email directly or view it on GitHubhttps://github.com//issues/62#issuecomment-38208474
.

[acecodes]

Calibre can be used in the command line. Here's a page that lays out the specifics of how to use it: http://manual.calibre-ebook.com/cli/ebook-convert.html.

No, it's not completely automated, but I don't know of any PDF conversion tools that are. From what I understand, PDF is structured as a series of images, and pulling text and formatting data from images is very difficult. You can tweak the conversion process to provide better conversions, but no matter what tool you use, you'll need to do some manual work. Whether or not such an effort is worth it is entirely up to you.

[ghost]

Isn't that quite an old version of pandoc?

Either that or https://groups.google.com/forum/m/#!topic/pandoc-discuss/VVOdG5Sg13I thinks you may need braces around maketitle

[dfc]

Is there any chance you could remove Crypto101.tex from gitignore? I noticed that when I do an org-export emacs creates an auto directory this is not in the .gitignore. This made me wonder if there is anything unique in your emacs setup that affects org's export process.

I ask because I would like to help out with the latex and pandoc parts of the project and it would be easier if I knew that we were both starting with the same input.

[lvh]

Indeed, my version of pandoc is ancient. I forgot that pandoc is a cabal package, not one managed by my package manager... I've updated pandoc:

[~/Code/Crypto101/book]$ pandoc -v                                                                                                                                                             *[master][]
pandoc 1.12.3.3
Compiled with texmath 0.6.6, highlighting-kate 0.5.6.1.
Syntax highlighting is supported for the following languages:
    actionscript, ada, apache, asn1, asp, awk, bash, bibtex, boo, c, changelog,
    clojure, cmake, coffee, coldfusion, commonlisp, cpp, cs, css, curry, d,
    diff, djangotemplate, doxygen, doxygenlua, dtd, eiffel, email, erlang,
    fortran, fsharp, gnuassembler, go, haskell, haxe, html, ini, java, javadoc,
    javascript, json, jsp, julia, latex, lex, literatecurry, literatehaskell,
    lua, makefile, mandoc, markdown, matlab, maxima, metafont, mips, modelines,
    modula2, modula3, monobasic, nasm, noweb, objectivec, objectivecpp, ocaml,
    octave, pascal, perl, php, pike, postscript, prolog, python, r,
    relaxngcompact, restructuredtext, rhtml, roff, ruby, rust, scala, scheme,
    sci, sed, sgml, sql, sqlmysql, sqlpostgresql, tcl, texinfo, verilog, vhdl,
    xml, xorg, xslt, xul, yacc, yaml
Default user data directory: /Users/lvh/.pandoc
Copyright (C) 2006-2013 John MacFarlane
Web:  http://johnmacfarlane.net/pandoc
This is free software; see the source for copying conditions.  There is no
warranty, not even for merchantability or fitness for a particular purpose.

And then it does build an epub, but with two remaining issues: images and math. Here's a screenshot of the resulting epub in iBooks:

So I read the fine manual. The MathJax page appears to suggest MathJax is the de facto standard for EPUB math, which is nice because I already use Mathjax regularly :)

However, rendering with cat Crypto101.tex | pandoc -f latex -t epub --mathjax -o Crypto101.epub didn't help, the math still looks like it does in the above document :/

[lvh]

By the way @dfc

• I added the TeX source in 5c25e90. A little worried that it will get outdated because people won't always recompile when editing the org source, though.
• dfcdfc is totally not the best HN color, I checked

[dfc]

I think pandoc needs one or more of the +tex_math_extensions turned on to deal with the math correctly. My math skills are as useful as configuring #lvhlvh as a hex color so I do not use the math extensions enough to remember when to use what option.

That being said I can track this down (aka brute force it) if you would like to focus on the content of the book.

[lvh]

I just enabled a pandoc extension called tex_math_dollars. Ballin'.

Seriously though, same issue with:

cat Crypto101.tex | pandoc -f latex -t epub+tex_math_dollars+tex_math_single_backslash --mathjax -o Crypto101.epub

It does appear that pandoc is perfectly aware of the math there (looking at the escapes) but just isn't doing a good job convincing mathjax to render it.

Interestingly, neither the output of that or the output of this:

pandoc --from latex --to html5+tex_math_dollars+tex_math_single_backslash --mathjax="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" --output Crypto101.html Crypto101.tex

actually has a reference to a Mathjax <script> tag...

[lvh]

So, the output of org-mode compiling to HTML looks pretty decent, just that Mathjax can't figure out what \xor means since it's usually spelled \oplus.

[lvh]

One of the big hurdles in this so far appears to be the PDF images, some browsers just don't render them or render links instead (I imagine most EPUB readers are in that category). Fortunately there is machinery in place for rendering both Metapost illustrations and hand-drawn PBM illustrations to SVG; it just has to be hooked up :)

[dfc]

It looks like pandoc's org-mode reader is progressing. I am going to see how it handles the org source.

[lvh]

Awesome! Thanks for working on this, @dfc!

[adammenges]

Yea, cheers @dfc!

[ghost]

Any updates?

[lvh]

Nope. pandoc has evolved a lot (particularly its org ingester), so feel free to try if EPUB is something you care about.

[NightMachinery]

I tried org to epub via pandoc, the mathy parts were incorrect. It also spewed a few warnings:

[WARNING] Could not convert TeX math '\xor', rendering as TeX:
  \xor
      ^
  unexpected eof
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'a \xor (b \xor c) = (a \xor b) \xor c', rendering as TeX:
  a \xor (b \xor c) = (a \xor b) \xor c
         ^
  unexpected "("
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'a \xor b = b \xor a', rendering as TeX:
  a \xor b = b \xor a
         ^
  unexpected "b"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'a \xor a = 0', rendering as TeX:
  a \xor a = 0
         ^
  unexpected "a"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '0
    \xor 0 = 0', rendering as TeX:
    \xor 0 = 0
         ^
  unexpected "0"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '1 \xor 1 = 0', rendering as TeX:
  1 \xor 1 = 0
         ^
  unexpected "1"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'a \xor 0 = a', rendering as TeX:
  a \xor 0 = a
         ^
  unexpected "0"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '0 \xor 0 = 0', rendering as TeX:
  0 \xor 0 = 0
         ^
  unexpected "0"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '1 \xor 0 = 1', rendering as TeX:
  1 \xor 0 = 1
         ^
  unexpected "0"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'a \xor b \xor a = b', rendering as TeX:
  a \xor b \xor a = b
         ^
  unexpected "b"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '\begin{align*}
  a \xor b \xor a & = a \xor a \xor b & \; & \text{(second rule)} \\
                  & = 0 \xor b        & \; & \text{(third rule)} \\
                  & = b               & \; & \text{(fourth rule)}
  \end{align*}
  ', rendering as TeX:
  a \xor b \xor a & = a \xor a \xor b & \;
         ^
  unexpected "b"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '\begin{align*}
  73 \xor 87 & = 0b1001001 \xor 0b1010111 \\
             & = \begin{array}{*{7}{C{\widthof{$\xor$}}}c}
                     1    & 0    & 0    & 1    & 0    & 0    & 1    & \quad \text{(left)}\\
                     \xor & \xor & \xor & \xor & \xor & \xor & \xor & \\
                     1    & 0    & 1    & 0    & 1    & 1    & 1    & \quad \text{(right)}\\
                 \end{array} \\
             & = \begin{array}{*{7}{C{\widthof{$\xor$}}}}
                     0    & 0    & 1    & 1    & 1    & 1    & 0
                 \end{array} \\
             & = 0b0011110 \\
             & = 30 \\
  \end{align*}
  ', rendering as TeX:
  73 \xor 87 & = 0b1001001 \xor 0b1010111
          ^
  unexpected "8"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '\begin{align*}
  c_1 \xor c_2
  &= (p_1 \xor k) \xor (p_2 \xor k) && (\text{definition})\\
  &= p_1 \xor k \xor p_2 \xor k && (\text{reorder terms})\\
  &= p_1 \xor p_2 \xor k \xor k && (a \xor b = b \xor a) \\
  &= p_1 \xor p_2 \xor 0 && (x \xor x = 0) \\
  &= p_1 \xor p_2 && (x \xor 0 = x)
  \end{align*}
  ', rendering as TeX:
  c_1 \xor c_2
           ^
  unexpected "c"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '\begin{eqnarray*}
  C_j \xor P_j
  &=& (P_j \xor K) \xor P_j \\
  &=& K \xor P_j \xor P_j \\
  &=& K \xor 0 \\
  &=& K
  \end{eqnarray*}
  ', rendering as TeX:
  C_j \xor P_j
           ^
  unexpected "P"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'k = c_i \xor p_i', rendering as TeX:
  k = c_i \xor p_i
               ^
  unexpected "p"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'p_i = c_i \xor k', rendering as TeX:
  p_i = c_i \xor k
                 ^
  unexpected "k"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'x
  \xor S(x) = 0', rendering as TeX:
  \xor S(x) = 0
       ^
  unexpected "S"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'x \xor S(x)=\texttt{0xff}', rendering as TeX:
  x \xor S(x)=\texttt{0xff}
         ^
  unexpected "S"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '\begin{align*}
  C_M & = E(k, IV_M \xor P_M) \\
      & = E(k, IV_M \xor (IV_M \xor IV_A \xor G)) \\
      & = E(k, IV_A \xor G)
  \end{align*}
  ', rendering as TeX:
  C_M & = E(k, IV_M \xor P_M) \\
                    ^
  unexpected "\\"
  expecting "&", "\\\\", white space or "\\end"
[WARNING] Could not convert TeX math '\begin{align*}
  P^{\prime}_1 & = D(k, C_1) \xor IV \\
               & = D(k, C_1) \xor k \\
               & = P_1
  \end{align*}

  \begin{align*}
  P^{\prime}_2 & = D(k, Z) \xor C_1 \\
               & = R
  \end{align*}

  \begin{align*}
  P^{\prime}_3 & = D(k, C_1) \xor Z \\
               & = D(k, C_1) \\
               & = P_1 \xor IV
  \end{align*}
  ', rendering as TeX:
  e}_1 & = D(k, C_1) \xor IV \\
                     ^
  unexpected "\\"
  expecting "&", "\\\\", white space or "\\end"
[WARNING] Could not convert TeX math 'P^{\prime}_3 = P_1 \xor IV', rendering as TeX:
  rime}_3 = P_1 \xor IV
                     ^
  unexpected "I"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '(P_1 \xor IV) \xor P_1 = IV', rendering as TeX:
  (P_1 \xor IV) \xor P_1 = IV
            ^
  unexpected "I"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '\begin{align*}
  P^{\prime}_{i + 1}
  & = P_{i + 1} \xor X \\
  & = P_{i + 1}
    \xor \mathtt{ZZZZZZZZZ}
    \xor \mathtt{;admin=1;} \\
  & = \mathtt{ZZZZZZZZZ}
    \xor \mathtt{ZZZZZZZZZ}
    \xor \mathtt{;admin=1;} \\
  & = \mathtt{;admin=1;} \\
  \end{align*}
  ', rendering as TeX:
  & = P_{i + 1} \xor X \\
                ^
  unexpected "\\"
  expecting "&", "\\\\", white space or "\\end"
[WARNING] Could not convert TeX math '\begin{eqnarray*}
  D(C_i)[b] \xor r_b \xor \mathtt{01} \xor \mathtt{02} & = & \mathtt{01} \xor \mathtt{01} \xor \mathtt{02} \\
  & = & \mathtt{02}
  \end{eqnarray*}
  ', rendering as TeX:
  D(C_i)[b] \xor r_b \xor \mathtt{01} \xor
                 ^
  unexpected "r"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '\begin{equation}
  x \leftarrow x \xor (y \madd z) \lll n
  \end{equation}
  ', rendering as TeX:
   \leftarrow x \xor (y \madd z) \lll n
                     ^
  unexpected "("
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '\xor', rendering as TeX:
  \xor
      ^
  unexpected eof
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'p_i
  \xor s_i \xor s_i = p_i', rendering as TeX:
  \xor s_i \xor s_i = p_i
       ^
  unexpected "s"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '\begin{equation}
  y \equiv g^x \pmod{p}
  \end{equation}
  ', rendering as TeX:
  y \equiv g^x \pmod{p}
                    ^
  unexpected "{"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '\begin{equation}
  m_A \equiv g^{r_A} \pmod{p}
  \end{equation}
  \begin{equation}
  m_B \equiv g^{r_B} \pmod{p}
  \end{equation}
  ', rendering as TeX:
  equiv g^{r_A} \pmod{p}
                     ^
  unexpected "{"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'm \equiv g^r \pmod{p}', rendering as TeX:
  m \equiv g^r \pmod{p}
                    ^
  unexpected "{"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '\begin{equation}
  s \equiv (g^{r_A})^{r_B} \pmod{p}
  \end{equation}
  ', rendering as TeX:
  ^{r_A})^{r_B} \pmod{p}
                     ^
  unexpected "{"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '(g^{r_A})^{r_B} \equiv (g^{r_B})^{r_A} \pmod{p}', rendering as TeX:
  ^{r_B})^{r_A} \pmod{p}
                     ^
  unexpected "{"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'm_A \equiv g^{r_A} \pmod{p}', rendering as TeX:
  equiv g^{r_A} \pmod{p}
                     ^
  unexpected "{"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'm_B \equiv g^{r_B} \pmod{p}', rendering as TeX:
  equiv g^{r_B} \pmod{p}
                     ^
  unexpected "{"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '\begin{equation}
  C \equiv M^e \pmod{N}
  \end{equation}
  ', rendering as TeX:
  C \equiv M^e \pmod{N}
                    ^
  unexpected "{"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'P^e \equiv P^1
  \equiv P \pmod N', rendering as TeX:
  \equiv P \pmod N
                 ^
  unexpected "N"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '448 \pmod {512}', rendering as TeX:
  448 \pmod {512}
            ^
  unexpected "{"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '448 \pmod {512}', rendering as TeX:
  448 \pmod {512}
            ^
  unexpected "{"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '\begin{align*}
  t_1 &\equiv m_1 \cdot a + b \pmod p \\
  t_2 &\equiv m_2 \cdot a + b \pmod p
  \end{align*}
  ', rendering as TeX:
  iv m_1 \cdot a + b \pmod p \\
                     ^
  unexpected "\\"
  expecting "&", "\\\\", white space or "\\end"
[WARNING] Could not convert TeX math '\begin{align*}
    t_1 - t_2 &\equiv (m_1 \cdot a + b) - (m_2 \cdot a + b) \pmod p \\
    &\Downarrow \text{(remove parentheses)} \\
    t_1 - t_2 &\equiv m_1 \cdot a + b - m_2 \cdot a - b \pmod p \\
    &\Downarrow \text{($b$ and $-b$ cancel out)} \\
    t_1 - t_2 &\equiv m_1 \cdot a - m_2 \cdot a \pmod p \\
    &\Downarrow \text{(factor out $a$)} \\
    t_1 - t_2 &\equiv a \cdot (m_1 - m_2) \pmod p \\
    &\Downarrow \text{(flip sides, multiply by inverse of $(m_1 - m_2)$)} \\
    a &\equiv (t_1 - t_2)(m_1 - m_2)^{-1} \pmod p
  \end{align*}
  ', rendering as TeX:
   (m_2 \cdot a + b) \pmod p \\
                     ^
  unexpected "\\"
  expecting "&", "\\\\", white space or "\\end"
[WARNING] Could not convert TeX math '\begin{align*}
  t_1 &\equiv m_1 \cdot a + b \pmod p \\
  &\Downarrow \text{(reorder terms)}\\
  b &\equiv t_1 - m_1 \cdot a \pmod p
  \end{align*}
  ', rendering as TeX:
  iv m_1 \cdot a + b \pmod p \\
                     ^
  unexpected "\\"
  expecting "&", "\\\\", white space or "\\end"
[WARNING] Could not convert TeX math '\pmod{p}', rendering as TeX:
  \pmod{p}
       ^
  unexpected "{"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'g \equiv 2^{(p-1)/q} \pmod{p}', rendering as TeX:
  v 2^{(p-1)/q} \pmod{p}
                     ^
  unexpected "{"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math 'y \equiv g^x \pmod{p}', rendering as TeX:
  y \equiv g^x \pmod{p}
                    ^
  unexpected "{"
  expecting "%", "\\label", "\\nonumber" or whitespace
[WARNING] Could not convert TeX math '\begin{eqnarray*}
  s_1 - s_2 & \equiv & k^{-1} (H(m_1) + xr) - k^{-1} (H(m_2) + xr) \pmod q \\
  & \equiv & k^{-1} \left( (H(m_1) + xr) - (H(m_2) + xr) \right) \pmod q \\
  & \equiv & k^{-1} (H(m_1) + xr - H(m_2) - xr) \pmod q \\
  & \equiv & k^{-1} (H(m_1) - H(m_2)) \pmod q
  \end{eqnarray*}

...