support (e.g. as plugin) for processing pandoc JSON format (like filters but via network/unix socket)

[jonassmedegaard]

Seems gitit does not support applying Pandoc filters.

I am setting up a workflow similar to http://gitit.net/collaborative%20authoring. Some of my needs for the resulting PDF design can be solved by use of a custom XeLaTeX template, but some need tweaking the text content. I currently use perl regex but now try recode that as Pandoc filters. I then look at gitit for allowing authors to edit content and instantly fetch a resulting draft PDF - but would then need my Pandoc filters applied as part of the export.

If curious, my concrete use is for editing reports at the European Parliament (didn't succeed for http://islandsofresilience.eu/ and now try again for http://euwiki.org/EUT/2nd-edition). My draft messy code is as at http://source.epfsug.biks.dk/?p=eut.git.

[jgm]

What you want is a plugin.

Filters do the same thing in pandoc, but for gitit, it makes more
sense to load the code when the server starts.

See the documentation on plugins.

+++ Jonas Smedegaard [Dec 17 14 08:29 ]:

Seems gitit does not support applying Pandoc filters.

I am setting up a workflow similar to http://gitit.net/collaborative%20authoring. Some of my needs for the resulting PDF design can be solved by use of a custom XeLaTeX template, but some need tweaking the text content. I currently use perl regex but now try recode that as Pandoc filters. I then look at gitit for allowing authors to edit content and instantly fetch a resulting draft PDF - but would then need my Pandoc filters applied as part of the export.

If curious, my concrete use is for editing reports at the European Parliament (didn't succeed for http://islandsofresilience.eu/ and now try again for http://euwiki.org/EUT/2nd-edition). My draft messy code is as at http://source.epfsug.biks.dk/?p=eut.git.

---

Reply to this email directly or view it on GitHub:
#477

[jonassmedegaard]

Makes sense to implement this feature as a plugin, I guess.

If you mean a unique gitit plugin for each filtering need, I am not sure I agree, however: I write my Pandoc filter in Perl. Transforming that into a long-running service processing Pandoc JSON over a network socket is easy - far easier than learning Haskell to rewrite as a gitit plugin. I imagine it is similar for other non-Haskell filter frameworks as well.

[jonassmedegaard]

You mention plugin documentation - I have failed to locate that on my own: Could you pass me a URL, so that I can check what exists instead of making assumption.

[jgm]

It should be here:
https://hackage.haskell.org/package/gitit-0.10.6.1/docs/Network-Gitit-Interface.html

I don't know why the docs for this module didn't build on Hackage.

Anyway, if you do 'cabal haddock' in the source directory after building
gitit with the -fplugins option, you should find it at
dist/doc/html/gitit/Network-Gitit-Interface.html.

Or you can just look directly at the comments in the source code for that module.

+++ Jonas Smedegaard [Dec 17 14 11:20 ]:

You mention plugin documentation - I have failed to locate that on my own: Could you pass me a URL, so that I can check what exists instead of making assumption.

---

Reply to this email directly or view it on GitHub:
#477 (comment)

[jgm]

You could always write a Haskell plugin that just shells out to your perl program, though this might more difficult than just rewriting in Haskell.

+++ Jonas Smedegaard [Dec 17 14 11:13 ]:

Makes sense to implement this feature as a plugin, I guess.

If you mean a unique gitit plugin for each filtering need, I am not sure I agree, however: I write my Pandoc filter in Perl. Transforming that into a long-running service processing Pandoc JSON over a network socket is easy - far easier than learning Haskell to rewrite as a gitit plugin. I imagine it is similar for other non-Haskell filter frameworks as well.

---

Reply to this email directly or view it on GitHub:
#477 (comment)

[jonassmedegaard]

Quoting John MacFarlane (2014-12-17 20:36:13)

You could always write a Haskell plugin that just shells out to your
perl program, though this might more difficult than just rewriting in
Haskell.

That sounds less elegant to me than extending the existing
language-agnostic Pandoc JSON serialization interface:

• Path handed to Pandoc is executed, and JSON is piped through
• URL handed to gitit is resolved, and JSON is sent through
  • Jonas

• Jonas Smedegaard - idealist & Internet-arkitekt

• Tlf.: +45 40843136 Website: http://dr.jones.dk/

  [x] quote me freely [ ] ask before reusing [ ] keep private

[jgm]

+++ Jonas Smedegaard [Dec 17 14 12:33 ]:

Quoting John MacFarlane (2014-12-17 20:36:13)

You could always write a Haskell plugin that just shells out to your
perl program, though this might more difficult than just rewriting in
Haskell.

That sounds less elegant to me than extending the existing
language-agnostic Pandoc JSON serialization interface:

• Path handed to Pandoc is executed, and JSON is piped through
• URL handed to gitit is resolved, and JSON is sent through

But using perl and JSON serialization for this is not very
elegant in the first place. After all, we have the AST in a nice
in-memory pandoc data structure. A proper gitit plugin just
modifies this directly -- the shell is not involved at all, and no
JSON serialization/deserialization needs to be done. In a
web application, this seems a far better approach.

Note also that the pandoc filters interface is part of the pandoc
executable, not the pandoc library, which gitit uses.

[jonassmedegaard]

Quoting John MacFarlane (2014-12-17 22:23:09)

+++ Jonas Smedegaard [Dec 17 14 12:33 ]:

Quoting John MacFarlane (2014-12-17 20:36:13)

You could always write a Haskell plugin that just shells out to your
perl program, though this might more difficult than just rewriting
in Haskell.

That sounds less elegant to me than extending the existing
language-agnostic Pandoc JSON serialization interface:

• Path handed to Pandoc is executed, and JSON is piped through
• URL handed to gitit is resolved, and JSON is sent through

But using perl and JSON serialization for this is not very elegant in
the first place. After all, we have the AST in a nice in-memory pandoc
data structure. A proper gitit plugin just modifies this directly --
the shell is not involved at all, and no JSON
serialization/deserialization needs to be done. In a web application,
this seems a far better approach.

Note also that the pandoc filters interface is part of the pandoc
executable, not the pandoc library, which gitit uses.

Sure, learning Haskell and using solely that language is certainly the
most elegant.

Seems gitit is not gonna be as friendly towards simple folks like me as
Pandoc (the executable) is, so I guess I will simply have to stay away
from gitit (as both the alternatives of learning Haskell and paying
someone else to write plugins for me are less suitable for me).

Feel free to close this bugreport, if I understand you correctly that
you do not find it relevant to support the neds I describe.

• Jonas

• Jonas Smedegaard - idealist & Internet-arkitekt

• Tlf.: +45 40843136 Website: http://dr.jones.dk/

  [x] quote me freely [ ] ask before reusing [ ] keep private

[jgm]

+++ Jonas Smedegaard [Dec 17 14 18:14 ]:

Seems gitit is not gonna be as friendly towards simple folks like me as
Pandoc (the executable) is, so I guess I will simply have to stay away
from gitit (as both the alternatives of learning Haskell and paying
someone else to write plugins for me are less suitable for me).

Feel free to close this bugreport, if I understand you correctly that
you do not find it relevant to support the neds I describe.

It can be kept open. I can certainly see the use for this.
But, because of my other commitments, I've been in minimal
maintenance mode for gitit for some time. If someone wants
to contribute a patch for this feature, I'd consider it.

[jonassmedegaard]

Quoting John MacFarlane (2014-12-18 05:04:54)

+++ Jonas Smedegaard [Dec 17 14 18:14 ]:

Seems gitit is not gonna be as friendly towards simple folks like me
as Pandoc (the executable) is, so I guess I will simply have to stay
away from gitit (as both the alternatives of learning Haskell and
paying someone else to write plugins for me are less suitable for
me).

Feel free to close this bugreport, if I understand you correctly that
you do not find it relevant to support the neds I describe.

It can be kept open. I can certainly see the use for this.
But, because of my other commitments, I've been in minimal
maintenance mode for gitit for some time. If someone wants
to contribute a patch for this feature, I'd consider it.

I appreciate that you recognize its value. Fully understandable that it
is not of highest priority for you. :-)

Let's hope someone comes along with similar needs and better Haskell
skills than me.

• Jonas

• Jonas Smedegaard - idealist & Internet-arkitekt

• Tlf.: +45 40843136 Website: http://dr.jones.dk/

  [x] quote me freely [ ] ask before reusing [ ] keep private

[RGD2]

I get the impression that adding pandoc filter support of any kind to gitit is nontrivial because pandoc is being used as a library, and filter use (especially thirdparty coughpantablescough) may not have been factored into the design of pandoc when used that way. Well, at least the one time the word 'filter' seems to show up on pandoc's API help page is in the context of xss security. Discuss?

I would absolutely love it if someone were to add a way to import-in place another file with some kind of shell filter... this would be enough to support pandoc filters (since the option could be given) but would also allow things like using a bash one-liner to grab a particular figure out of a pdf, for example. Or a png graph as freshly-generated by a gnuplot. Or even that after an ngspice call.

Alternative workaround for now is to have the 'desktop' cloud9 ide installed on the same machine as gitit, and use it to update wikidata there and push commits directly. This also gives a way to run a shell, but it means that generating/updating such content is an extra step or two, and the resulting files are static graphics which gitit will show on the page.

I guess for this issue, one could exploit a git hook in gitit's wikidata repository to process certain source documents automatically when they change, and push a commit with the updated 'display' files. That would have much the same ultimate effect, although with having to decode gits documentation to do so.