Lua filter can't be used on pandoc-citeproc output

[jkr]

Native lua filters are used before external filters. This means that you can't use a lua filter on the output of other filters.

This is an issue for me because I have some filters that fix up pandoc-citeproc output (removing repeated parenthetical citations in MLA output). While my use is a bit niche, I can imagine more mainstream use-cases: a filter styling the refs Div that pandoc-citeproc produces.

Any thoughts on the best way to deal with this? Should pandoc-citeproc be a special case? I can always do two passes with a JSON intermediate, but that seems to go against the spirit of using the lua filters in the first place. (Of course, for the time being, I can keep using my old haskell filter for this special case.)

[jkr]

Another example that I came across while converting my filters. One targeted LaTeX style uses a list environment for its works-cited list, so I need a filter that can run through the bibliography entries and put an \item in front of them (as well as wrapping the whole thing in \begin{citedlist}/\end{citedlist}). So, again, this is a filter that needs to run on the other side of pandoc-citeproc.

[jgm]

@tarleb, any thoughts? I actually think it might make sense for all lua filters to run after regular filters, because one of the main things that will be difficult to achieve in a lua filter is pandoc-citeproc, and Jesse is certainly right that we might want to manipulate its output. Another option would be revising things so that filters and lua-filters could be mixed in any order, and would be run in the order presented on the command line. That would take a bit of re-plumbing in Text.Pandoc.App. +++ Jesse Rosenthal [Dec 25 17 15:35 ]:

…

Another example that I came across while converting my filters. One targeted LaTeX style uses a list environment for its works-cited list, so I need a filter that can run through the bibliography entries and put an \item in front of them (as well as wrapping the whole thing in \begin{citedlist}/\end{citedlist}). So, again, this is a filter that needs to run on the other side of pandoc-citeproc. — You are receiving this because you are subscribed to this thread. Reply to this email directly, [1]view it on GitHub, or [2]mute the thread. References 1. #4196 (comment) 2. https://github.com/notifications/unsubscribe-auth/AAAL5BD7oXZvwDj_mRSZjk-MEU3EMah4ks5tD8C9gaJpZM4RMTrW

[tarleb]

I'm not sure. This sounds sensible, but the opposite order can be useful as well, as evident from this discussion: https://groups.google.com/d/topic/pandoc-discuss/-l-1QkPoG9k/discussion And the complexity inherent to special cases always scares me. I guess my preferred solution would be to provide a way to run pandoc-citeproc from within lua filters. E.g., if there was a simple way to transform a document to json, one could do json = ??doc_to_json?? (doc) doc_with_refs_json = pandoc.pipe('pandoc-citeproc', {}, json) doc_with_refs = pandoc.read(doc_with_refs_json, 'json') We could provide a helper function for this. Or another idea: adding a command line switch '--disable-json-filters' and a Lua function `run_json_filters(doc)` would allow users to gain full control over when JSON filters are run in their filtering pipeline.

[tarleb]

John MacFarlane <notifications@github.com> writes:

Another option would be revising things so that filters and lua-filters could be mixed in any order, and would be run in the order presented on the command line. That would take a bit of re-plumbing in Text.Pandoc.App.

I had somehow missed that second paragraph. I'm in favor if this idea.

[jkr]

@tarleb: I see that you're starting work on a run_json_filter approach. This seems a bit more convoluted (from a user's point of view) than John's suggestion (filters in any order), but since I don't understand the difficulties of the implementation, I'm in no position to object.

What I would suggest, though, if you go with this approach, you package a pandoc-citeproc.lua filter with pandoc-citeproc, that just runs pandoc.run_json_filter(pandoc-citeproc). This would allow us to maintain some composibility, that would be lost if we had to call one filter from another.

Anecdotal justification: I use filters in a pipeline, and I like to be able to mix and match (e.g., pandoc-citeproc -> fix citations -> get rid of small caps, but sometimes just pandoc-citeproc -> get rid of small caps, and sometimes just fix citations -> get rid of small caps). It would be cumbersome to have to build one filter into another, and I couldn't compose them as easily.

[tarleb]

Oh, please do object! I wasn't convinced that adding a function to lua counts as a solution either, and I agree with you that jgm's solution is preferable. I'll work on the run_json_filter some more, but will scratch the claim that it could solve this issue.

[mb21]

I may be missing something, but as a user, I’d prefer just having them run in the order specified by —filter

[jkr]

Mauro Bieg <notifications@github.com> writes:

I may be missing something, but as a user, I’d prefer just having them run in the order specified by `—filter`

If you're suggesting putting all filters (JSON and lua) under the same option tag, there seem to be a couple of issues there: 1. pandoc looks in different places for the two sorts of filters ($DATADIR for lua filters, $PATH for JSON filters). This is because JSON filters are executables, while lua filters can only run properly inside the embedded lua. 2. You could have a lua JSON filter (I believe @tarleb wrote a library last year) in your path, so it would be hard to distinguish the two.

[jkr]

Albert Krewinkel <notifications@github.com> writes:

Oh, please *do* object! I wasn't convinced that adding a function to lua counts as a solution either, and I agree with you that jgm's solution is preferable.

It *does* seem like an nice capability to have, and I can imagine a future in which the lua filters are the "official" filters, and json filters are a niche option, so this could be really useful. Out of curiosity (and apologies for asking questions that were probably covered back when these were being built), is there a reason why `applyLuaFilters` and `applyFilters` have to be separated by `extractMedia` and `applyTransforms`? ~~~ ...

=> applyLuaFilters datadir (optLuaFilters opts) format => maybe return extractMedia (optExtractMedia opts) => applyTransforms transforms => applyFilters readerOpts datadir filters' [format]

... ~~~ It seems offhand like both sorts of filters should be working with the same AST.

[mb21]

you're suggesting putting all filters (JSON and lua) under the same
option tag

Yes.

pandoc looks in different places for the two sorts of filters

So --lua-filter looks in $DATADIR. But --filter looks first in $PATH and if it doesn't find anything looks in $DATADIR (says the MANUAL). So as a user, I could use --filter to order all my non-lua and lua filters. The case where I have a lua-filter on my PATH and I need to order them should be pretty rare (and the user should then probably just remove the lua-filter from the PATH).

[tarleb]

Out of curiosity, […] is there a reason why applyLuaFilters and applyFilters have to be separated by extractMedia and applyTransforms?

Initially, Lua filters were run after JSON filters; this changed when we added the pandoc.mediabag module, which requires that maybe return extractMedia (optExtractMedia opts) is run after Lua filters are done. I suppose the relative position of applyTransforms wasn't changed, which is why it's run between both filter runs.

[tarleb]

You could have a lua JSON filter (I believe @tarleb wrote a library last year) in your path, so it would be hard to distinguish the two.

I wrote this useful monstrosity, which isn't really a filter in the classical sense but a custom writer which outputs JSON. I don't think there is a lua port of pandocfilters as of now. It should be safe to implement @mb21's proposal. I still don't like special cases, but it might make sense in this case.

[jgm]

I think it's probably still worth keeping the --lua-filter vs --filter distinction; that tells pandoc whether to treat a lua file as a pandoc lua filter or a regular json filter. But respecting the command line order of --lua-filter and --filter would be good for many reasons, and would solve this issue.

We need to make sure we don't run into trouble with the mediabag stuff, but that shouldn't be a big obstacle.

[jgm]

I've made some changes, I'll put up code to a branch soon.

[jgm]

See the filter-changes branch.
Note that this contains incompatible API changes in Text.Pandoc.App and would have to appear as pandoc 2.1. I'll probably release some fixes in the 2.0 series first.

[tarleb]

@jgm what are your thoughts on #4197, do you see this as a desirable feature? If so, should this be part of the 2.0 series, or should I rebase it vs your filter-changes branch?

[jgm]

@tarleb I guess I don't see a compelling reason to add #4197, since we now have a cleaner way of interleaving lua filters and regular json filters. But if you think there's some task that would require running a json filter from a lua filter, perhaps you could describe the use case in more detail in the PR?

[jgm]

Closed by 8b575db