"~" and "*" not expanding before calling subprocesses

[grea09]

I want to create global styles with odt reference files. So I write this in the yaml file :

rapport:
    all:
        metadata:
            smart: true
            standalone: true
            numbersections: true
            secPrefix:
                - section
                - sections
            figPrefix:
                - figure
                - figures
            eqnPrefix:
                - equation
                - equations
            tblPrefix:
                - table
                - tables
            markdown_in_html_blocks: true
            bibliography: "~/.panzer/bibliography/zotero.bib"

        filter:
            - run: pandoc-crossref
            - run: pandoc-science.py
            - run: pandoc-citeproc
            - run: pandoc-svg.py
        postflight:
            - run: odt2pdf
        cleanup:
            - run: clean

    odt:
        commandline:
            reference-odt: "`~/.panzer/template/rapport.odt`"

The problem is that I get that output :

         ----- pandoc read -----
         running
         ----- style definitions -----
         global:
           rapport        
         ----- document style -----
         style:
           rapport
         full hierarchy:
           rapport
         writer:
           odt
         ----- run list -----
         filter:
          1  pandoc-crossref "~/.panzer/filter/pandoc-crossref"
          2  pandoc-science  "~/.panzer/filter/pandoc-science.py"
          3  pandoc-citeproc "pandoc-citeproc"
          4  pandoc-svg      "~/.panzer/filter/pandoc-svg.py"
         postflight:
          5  odt2pdf         "~/.panzer/postflight/odt2pdf"
         cleanup:
          6  clean           "~/.panzer/cleanup/clean"
         ----- filter -----
         [1/6] pandoc-crossref odt
         [2/6] pandoc-science odt
         [3/6] pandoc-citeproc odt
ERROR:     !pandoc-citeproc: Could not find ~/.panzer/bibliography/zotero.bib
ERROR:   failed to receive json object from filter---skipping filter
         [4/6] pandoc-svg odt
         ----- pandoc write -----
         pandoc writing with options:
           --reference-odt=~/.panzer/template/rapport.odt
ERROR:     !pandoc: ~/.panzer/template/rapport.odt: openBinaryFile: does not exist (No such file or directory)
         ----- postflight -----
         [5/6] odt2pdf
ERROR:     !Error: source file could not be loaded
         ----- cleanup -----
         [6/6] clean

Everything works with complete absolute paths but not with symbols that require expansion. I tried to use commands using the wildcard "*" that works in the terminal but not with Panzer. I think it is an issue with Python not expanding them unless told otherwise.

Thanks for the great tool, I use it even for redacting quite complex conference papers.

[msprev]

Thanks very much for the kind words!

There are a couple of different things going on here and we need to unpick them.

First, the bibliography metadata field. If this metadata field is set, and pandoc-citeproc is then run as a filter, pandoc-citeproc looks for the bibliography in that location. All panzer does is it allows this metadata field to be set as part of the style (so conveniently sweeps it away from the document's source file). Now, in principle, this metadata field can contain any sequence of characters. It is up to pandoc-citeproc to do with this field it what it wills. At the moment, pandoc-citeproc does not expand ~ to the current directory. You can verify this for yourself. If you set explicitly set the bibliography field inside the YAML block of a document and use a ~, run pandoc + pandoc-citeproc (don't run panzer!), then you get the same error. Correct me if I'm wrong about this.

The problem you are experiencing is not of panzer's making, but panzer can provide a solution. All you need to is write a little filter that you style runs before pandoc-citeproc. That filter looks for '~' in your bibliography metadata field and substitutes the user's home directory. It's a very simple filter to write. I've had similar requests for expanding ~ in other metadata fields, and this is by far the best and most general solution. (For sometimes, one might not want ~ expanded!).

Second, the reference-odt field. This one is completely different. It appears inside the commandline section of the style. That does not create in a metadata field in the document that pandoc sees. Rather, it controls how panzer runs pandoc -- which command line options it executes pandoc. These fields in the style can only mean one thing. I can see here a plausible for panzer doing the expansion, as these function just like normal command line arguments. I will implement this. Can you please send me you reference odt file (or an equivalent) so that I can test it?

[msprev]

Right... I've thought more about this. I think on balance it is better not to implement it for reference-odt. I could, in principle, implement it for this and any other fields that appear inside commandline. That is because those fields have a known meaning -- I know which ones ~ expansion would make sense for, and which ones it would not. But what about the args fields of scripts and filters? It would be odd to expand ~ for commandline but not there. But those fields have unknown semantics. Some may refer to files for which expansion makes sense, others may not. And it would be confusing to do expansion in commandline, but not in args.

An alternative option would be to use Python's shell=True option for running every one of panzer's subprocesses. I'm not keen to do this. Pretty much everything I've read recommends against using this option. First, it introduces extra security vulnerability. Second, it creates a new and hairy problem of the user needing to manually escape all arguments of every subprocess according to the rules for whichever shell is being invoked on the machine it is being run on. This will create a nest of complex and hard-to-find bugs. The convenience of ~ expansion wouldn't be worth it.

So... I think on balance the best and least confusing thing is to not allow ~ expansion inside a style definition. The same applies to * expansion.

If this is a really sore point for you, then you could write a little program that invokes panzer but first goes through your style definition file and does any expansion you like? You would yourself have a sense of which fields you want to expand and which you do not.

[grea09]

I found a workaround for this. I use a preflight script to create a symbolic link to ~/.panzer in the working directory. This way I can use .panzer to access the configuration and template files.
The only problem I see with this is that it is not platform independent but that is a minor issue for me.

Thank you for your detailed response.

[msprev]

Great! That is an ingenious workaround. Thank you for sharing it.