Back to the collaboration to sprint formats #60

Closed
mlinksva opened this Issue May 16, 2016 · 8 comments

Comments

Projects
None yet
3 participants
@mlinksva
Collaborator

mlinksva commented May 16, 2016

I think the book has been worked on in 4 formats:

  1. On PubSweet in HTML at the book sprint. Input: Original. Output: PDF, HTML(?), epub(?)
  2. Jekyll static website in markdown and HTML. Reason: maintain and edit in git(?). Input: manual with some scripting conversion from HTML above(?). Output: HTML.
  3. Gitbook in markdown and HTML. Reason: get back PDF and epub outputs(?). Input: manual with some scripting conversion from markdown and HTML above(?). Output: HTML, epub, PDF.
  4. LaTeX. Reason: PDF (and thus print) output from Gitbook unsatisfactory, LaTeX beautiful. Input: manual with some scripting (pandoc?) conversion from Gitbook markdown and HTML. Output: PDF.

As of the 2016-05-22 release (3) and (4) will still be used, to output HTML and epub, and PDF, respectively.

I think @clemsos did the vast majority of the conversion work.

Is this an accurate recounting?

Two hypothetical questions:

(a) If we did any work on the book after the birthday release, would it make sense to maintain both Gitbook and LaTeX versions, keeping them in sync manually? Or could the LaTeX version (nicely and easily) be used to output HTML and epub?

(b) If you were starting a similar book project -- a period of intense collaboration, followed by long-term editing and publication, with zero tools predetermined -- what would you do? Start and stick with pubsweet? Start and stick with LaTeX? Something else?

Curious what anyone thinks, but I guess especially @clemsos and @christopheradams. Thanks for reading my 99% idly curious questions.

@mlinksva mlinksva added the question label May 16, 2016

@clemsos

This comment has been minimized.

Show comment
Hide comment
@clemsos

clemsos May 17, 2016

Contributor

Some precisions about the formats:

  1. On PubSweet in HTML at the book sprint. Input: Original. Output: PDF, HTML(i don't think so), epub(yes) . Problem was that we needed to maintain a local version of Pubsweet at the booksprint place (connection problems) while integrating external contributors + export the whole content every night for proofreading by another member of book sprint team. The software was buggy and unstable plus, the whole process didn't allow easy import/export. One of the problem was that the export was possible only with a specific version of Firefox because it relies on an early specification for page display that was abandonned at an early stage. That meant that none of us could reproduce the build of the PDF. So, after the book sprint, we switched to the 2) solution :
  2. Jekyll static website in markdown and HTML. Reason: maintain and edit in git(yes, we are a bunch of nerds), escape from Pubsweet (quite literally), make it readable as a website. Input: manual with some scripting conversion from HTML above(?). Output: HTML.
    Problem was that we now had to export the PDF from Jekyll, which didn't had an obvious solutions. I went through all possible solutions including Latex, printable html, python scripting, etc (see #12). At some point I ran accross Gitbook
  3. Gitbook in markdown and HTML. Reason: get back PDF and epub outputs(yes). Input: manual with some scripting conversion from markdown and HTML above(yes, I had to reformat some of the markdown to fit gitbook formatting). Also we had to custom gitbook itself to build a clean ebook that will go on Lulu thanks @christopheradams (costoffreedom-book/#5) Output: HTML, epub, PDF.
  4. LaTeX. Reason: PDF (and thus print) output from Gitbook unsatisfactory, LaTeX beautiful. Input: manual with some scripting (pandoc : yes mostly pandoc from gitbook markdown, then manual work to adapt) conversion from Gitbook markdown and HTML. I made bash scripts that automate the process (it reads the gitbook summary and convert it to Latex), but still a fair amount of work to do afterwards. Output: PDF.
Contributor

clemsos commented May 17, 2016

Some precisions about the formats:

  1. On PubSweet in HTML at the book sprint. Input: Original. Output: PDF, HTML(i don't think so), epub(yes) . Problem was that we needed to maintain a local version of Pubsweet at the booksprint place (connection problems) while integrating external contributors + export the whole content every night for proofreading by another member of book sprint team. The software was buggy and unstable plus, the whole process didn't allow easy import/export. One of the problem was that the export was possible only with a specific version of Firefox because it relies on an early specification for page display that was abandonned at an early stage. That meant that none of us could reproduce the build of the PDF. So, after the book sprint, we switched to the 2) solution :
  2. Jekyll static website in markdown and HTML. Reason: maintain and edit in git(yes, we are a bunch of nerds), escape from Pubsweet (quite literally), make it readable as a website. Input: manual with some scripting conversion from HTML above(?). Output: HTML.
    Problem was that we now had to export the PDF from Jekyll, which didn't had an obvious solutions. I went through all possible solutions including Latex, printable html, python scripting, etc (see #12). At some point I ran accross Gitbook
  3. Gitbook in markdown and HTML. Reason: get back PDF and epub outputs(yes). Input: manual with some scripting conversion from markdown and HTML above(yes, I had to reformat some of the markdown to fit gitbook formatting). Also we had to custom gitbook itself to build a clean ebook that will go on Lulu thanks @christopheradams (costoffreedom-book/#5) Output: HTML, epub, PDF.
  4. LaTeX. Reason: PDF (and thus print) output from Gitbook unsatisfactory, LaTeX beautiful. Input: manual with some scripting (pandoc : yes mostly pandoc from gitbook markdown, then manual work to adapt) conversion from Gitbook markdown and HTML. I made bash scripts that automate the process (it reads the gitbook summary and convert it to Latex), but still a fair amount of work to do afterwards. Output: PDF.
@clemsos

This comment has been minimized.

Show comment
Hide comment
@clemsos

clemsos May 17, 2016

Contributor

(a) the latex version reflects only partly the content of the markdown. I added the final tex files that are now used to create the PDF and made changes directly into them (see 2b5d8185229368c116f2fb6bddeff7c0d8eb5656 ). Before that it was generated from the makdown.

(b) we can discuss that at length. I personally think Gitbook has an UI that will allow to write a book quite easily. Sadly, they dont open source it. Pubsweet has shown painful at the end, even though it was useful during the sprint. What I will do is acknowledge that to have a fine printable PDF build of a book takes more than a week and therefore a book sprint that aims at printing something should include a budget for someone to create a clean PDF release using Latex.

Contributor

clemsos commented May 17, 2016

(a) the latex version reflects only partly the content of the markdown. I added the final tex files that are now used to create the PDF and made changes directly into them (see 2b5d8185229368c116f2fb6bddeff7c0d8eb5656 ). Before that it was generated from the makdown.

(b) we can discuss that at length. I personally think Gitbook has an UI that will allow to write a book quite easily. Sadly, they dont open source it. Pubsweet has shown painful at the end, even though it was useful during the sprint. What I will do is acknowledge that to have a fine printable PDF build of a book takes more than a week and therefore a book sprint that aims at printing something should include a budget for someone to create a clean PDF release using Latex.

@clemsos

This comment has been minimized.

Show comment
Hide comment
@clemsos

clemsos May 17, 2016

Contributor

All the scripts and things I have tried show me that there is no easy way to really automatize something. The best tool here is totally pandoc (what a great software!). You could build a pipeline w pandoc to have sth like online writing stored in some text input > .doc/mardown > html > latex (which may be the most convenient workflow), but someone still have to refine things and make it clean at the end.

Contributor

clemsos commented May 17, 2016

All the scripts and things I have tried show me that there is no easy way to really automatize something. The best tool here is totally pandoc (what a great software!). You could build a pipeline w pandoc to have sth like online writing stored in some text input > .doc/mardown > html > latex (which may be the most convenient workflow), but someone still have to refine things and make it clean at the end.

@christopheradams

This comment has been minimized.

Show comment
Hide comment
@christopheradams

christopheradams May 17, 2016

Collaborator

I agree with the points @clemsos made.

I made an additional refinement to the process for the Waiting poetry book. I kept the gitbook repo as the original "source", and then added it as a subtree to a print version repository. I wrote a Makefile which uses pandoc to convert the markdown files, plus sed and some perl regex to make any print-specific fixes. I did write my own LaTeX preamble to define the layout and input the files in the right order.

The final process is automated. I can just update the gitbook subtree and run make to get the final output.

Collaborator

christopheradams commented May 17, 2016

I agree with the points @clemsos made.

I made an additional refinement to the process for the Waiting poetry book. I kept the gitbook repo as the original "source", and then added it as a subtree to a print version repository. I wrote a Makefile which uses pandoc to convert the markdown files, plus sed and some perl regex to make any print-specific fixes. I did write my own LaTeX preamble to define the layout and input the files in the right order.

The final process is automated. I can just update the gitbook subtree and run make to get the final output.

@mlinksva

This comment has been minimized.

Show comment
Hide comment
@mlinksva

mlinksva May 17, 2016

Collaborator

Thanks @clemsos and @christopheradams I enjoyed reading.

So I can find later

I didn't look closely at the 2nd. @clemsos so on (a) if we made any post-release additions etc, they should be done in the gitbook version?

On (b) it seems there isn't any great solution for someone creating a book who can't deal at least a bit with LaTeX.

I wonder if https://www.sharelatex.com/ could be an end-to-end solution?

It seems that pubsweet is actively worked on at https://gitlab.coko.foundation/pubsweet/core ... hopefully it is or will be mature when (as I hope) I eventually participate directly in another booksprint.

Collaborator

mlinksva commented May 17, 2016

Thanks @clemsos and @christopheradams I enjoyed reading.

So I can find later

I didn't look closely at the 2nd. @clemsos so on (a) if we made any post-release additions etc, they should be done in the gitbook version?

On (b) it seems there isn't any great solution for someone creating a book who can't deal at least a bit with LaTeX.

I wonder if https://www.sharelatex.com/ could be an end-to-end solution?

It seems that pubsweet is actively worked on at https://gitlab.coko.foundation/pubsweet/core ... hopefully it is or will be mature when (as I hope) I eventually participate directly in another booksprint.

@mlinksva

This comment has been minimized.

Show comment
Hide comment
@mlinksva

mlinksva May 17, 2016

Collaborator

I guess it is obvious to you two, but for anyone reading this who is baffled by the issue title, it is a play on http://protocultural.org/ "back to the caves to surge forward", another #freebassel related project source at https://github.com/protocultural/protocultural.org 😄

Collaborator

mlinksva commented May 17, 2016

I guess it is obvious to you two, but for anyone reading this who is baffled by the issue title, it is a play on http://protocultural.org/ "back to the caves to surge forward", another #freebassel related project source at https://github.com/protocultural/protocultural.org 😄

@clemsos

This comment has been minimized.

Show comment
Hide comment
@clemsos

clemsos May 19, 2016

Contributor

😄 wasnt that obvious hehe !

One more note about compiling books etc. The main reasons that force me to edit directly .tex files instead of working with the versions generated by the Makefile were

  • multiple fonts/languages : english, chinese and arabic. Latex makes it completely painful to integrate pieces of chinese text inside English. A bit better for Arabic but still. You could maybe make a script that will detect language but then sounds like a bit overkill maybe...
  • inappropriate use of Markdown tags : we used the "quote" tag in our .md files to add author names. Well, that was a mistake. We should have gone with plain styling. it will have make things easier.
  • details, details, details : for instance, we have some poetry that requires no indentation.

That being said, I think one of the main constrain of this book was the heterogeneous nature / form of the texts. If you have a book with a clear unity in form and content (for instance a novel in a single language) the process is much easier. The problem with this one was having different languages and different types of text.

Contributor

clemsos commented May 19, 2016

😄 wasnt that obvious hehe !

One more note about compiling books etc. The main reasons that force me to edit directly .tex files instead of working with the versions generated by the Makefile were

  • multiple fonts/languages : english, chinese and arabic. Latex makes it completely painful to integrate pieces of chinese text inside English. A bit better for Arabic but still. You could maybe make a script that will detect language but then sounds like a bit overkill maybe...
  • inappropriate use of Markdown tags : we used the "quote" tag in our .md files to add author names. Well, that was a mistake. We should have gone with plain styling. it will have make things easier.
  • details, details, details : for instance, we have some poetry that requires no indentation.

That being said, I think one of the main constrain of this book was the heterogeneous nature / form of the texts. If you have a book with a clear unity in form and content (for instance a novel in a single language) the process is much easier. The problem with this one was having different languages and different types of text.

@clemsos

This comment has been minimized.

Show comment
Hide comment
@clemsos

clemsos May 20, 2016

Contributor

hey,

Just to celebrate the end of this endless editing session, I have put together all recipes and tools in a rep, based on what @christopheradams and I have been working on with Latex/Gitbook

I called it booksuite : https://github.com/clemsos/booksuite
Should be as easy as 1) fork or clone , 2) add markdown files 3) add links in in the summary 4) build with make

It is currently hosted in my guthub but we should move the rep into some shared organization

Contributor

clemsos commented May 20, 2016

hey,

Just to celebrate the end of this endless editing session, I have put together all recipes and tools in a rep, based on what @christopheradams and I have been working on with Latex/Gitbook

I called it booksuite : https://github.com/clemsos/booksuite
Should be as easy as 1) fork or clone , 2) add markdown files 3) add links in in the summary 4) build with make

It is currently hosted in my guthub but we should move the rep into some shared organization

@mlinksva mlinksva closed this May 20, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment