tectonic claims biber can't handle external paths

[aquohn]

While using latexmk, my folder structure would look like

.
|--code
   |-- ...
|--report
   |-- report.tex
   |-- ...
|--images
   |-- ...
|--presentation
   |-- presentation.tex
   |-- ...
|--project.bib

I would then load project.bib in my .tex files with \addbibresource{../project.bib}, so multiple documents can share the same bibliographic database. However, when trying to compile this with tectonic and using \usepackage[backend=biber]{biblatex}, it complains that error: relative parent paths are not supported for the external tool. Got path ``../project.bib``. However, this compiles fine with a legacy latexmk-based workflow.

[Neved4]

Hey @aquohn thx for the issue! 💜
Can I suggest using the search-path option to workaround your issue?

Remove ../ from \addbibresource and then compile with tectonic -Z search-path=../ test.tex
Good luck!

MWE test.tex:

\documentclass{article}
  \usepackage{lmodern}
  \usepackage[style=philosophy-modern]{biblatex}

  \addbibresource{test.bib}

\begin{document}
  \title{Bibliography with biber}
  \author{Lewis Carrol}
  \date{August 27, 2022}
  \maketitle

  Nunc auctor nunc in nisi suscipit tempus \cite{black1973poa}.

  \printbibliography
\end{document}

test.bib file:

@article{     black1973poa,
  title     = {The pricing of options and corporate liabilities},
  author    = {Fishcer Black and Myron Scholes},
  doi       = {10.1086/260062},
  journal   = {Journal of Political Economy},
  number    = 3,
  pages     = {637--654},
  publisher = {UChicago Press},
  volume    = 81,
  year      = 1973,
}

[mskblackbelt]

Is there any way to implement this workaround with tectonic -X build and the Tectonic.toml file? I tried tossing in a search_path = '../..', but tectonic threw an error on that one.

[Neved4]

Hey @mskblackbelt! I believe experimental features like those enabled with -Z are not configurable through Tectonic.toml but I'm not 100% sure if I'm recalling it properly. 😅

I'd ping @pkgw and wait for his response.

As far as I know you can't do tectonic -X build -Z search-path=../ either, despite -Z unstable options and -X commands being part of the same V2 which, note is not the default. I don't know if some unstable options will be merged with the other V2 or if they'll be replaced with more robust built-in alternatives baked inside tectonic itself.
That'd be an interesting topic awesomely suited for discussions! 💜

[aquohn]

Hi @mnrvwl thanks for the suggestion! While working around the issue is simple, I would like to know why tectonic isn't capable of supporting the behaviour that latexmk supports, and if it can be patched in. I would prefer not to edit the file when switching between tools, and it would be nice if this could be supported as default behaviour.

[Neved4]

@aquohn You're right!

I share your love for latexmk, is an incredible convenient tool and it'd be awesome if tectonic had smoother edges so it had all the latexmk niceties + the awesome tectonic things that latexmk lacks.

Some of my projects are rather complex: documents split over several files and across complex directory trees, so manually writing the relative paths on a per-file basis and keeping track of it is like finding a needle in a stack.

To solve this I personally use the TEXINPUTS variable in a centralised place because it allows me to define all the paths properly: export TEXINPUTS="../:${TEXINPUTS}", and add as many declarations as I need, in a similar fashion to how'd you use UNIX's $PATH. In tectonic I use -Z search-path=../, and throw as many -Z as needed.

Going this route allows me to have simple source declarations like:

\addbibresource{arthistory.bib}
\addbibresource{philosophy.bib}
\addbibresource{astronomy.bib}

Where I only have to specify the endless (e.g., ../../../../) paths once —declared in TEXINPUTS or -Z—, instead of dozens of times:

\addbibresource{../../../../arthistory.bib}
\addbibresource{../../../../philosophy.bib}
\addbibresource{../../../../astronomy.bib}

Admittedly it relies, to an extent, in setting up the tools and the environment —note latexmk also has a .latexmkrc file that supports the ensure_path() function— yet it's a more elegant and cleaner way to play nice and not edit the file when switching between tools 🚀

I totally support that tectonic handles external paths with biber in the future —both with relative paths and in Tectonic.toml— for it'll allow for more authoring options 💯

[pkgw]

Regarding the interplay between -Z options and the Tectonic.toml mode: the basic idea is that things that are supported through -Z in the V1 or tectonic -X compile mode would be exposed by entries in the Tectonic.toml file. However, a wrinkle here is that the -Z options are, as advertised, unstable, while keywords added to Tectonic.toml are intended to be supported indefinitely. That's why there isn't a policy of, say, ensuring that every -Z option has a corresponding Tectonic.toml keyword.

For this specific case, things are bit more complex because of the way that Tectonic wants to invoke biber in an isolated, temporary directory to make sure that its operation is encapsulated. It is hard for me to think of a tractable way to set up that working directory with relative paths that reach out of it.

I think that the -Z shell-escape-cwd unstable option might help here, since that turns off the temporary directory functionality.

[mskblackbelt]

It looks like the shell_escape_cwd should work in Tectonic.toml, but I can't figure out the syntax to use $(pwd) as a key in the TOML file. I'd like to access a bib file located one level above the TOML file (two above the _preamble.tex file). When I try using shell_escape = true and shell_escape_cwd = '$(pwd)/..', I get the following output:

note: "version 2" Tectonic command-line interface activated
Running TeX ...
warning: accessing absolute path `/dev/null`; build may not be reproducible in other environments
warning: open of input /dev/null.tex failed
caused by: access to the path `/dev/null.tex` is forbidden
warning: open of input /dev/null.tex failed
caused by: access to the path `/dev/null.tex` is forbidden
error: failed to create file `$(pwd)/../Plot_Intro.log`
caused by: No such file or directory (os error 2)
error: failed to execute the shell-escape command "uname -s > "Plot_Intro.w18"": execution of the request failed
error: ifplatform.sty:92: Package catchfile Error: File `Plot_Intro.w18' not found.

See the catchfile package documentation for explanation.
Type  H <return>  for immediate help
error: halted on potentially-recoverable error as specified

Additionally, shell_escape = true doesn't seem to be inferred by the shell_escape_cwd option in the TOML file, contrary to the description of the -Z options. If I leave out the shell_escape = true and only set shell_escape_cwd, I get an error when pygments tries to run telling me to run with the --shell-escape flag.

Any suggestions on what's happening here (mostly the first situation, I can live with having to set both values).

[pkgw]

@mskblackbelt Thanks for pointing out about the default value for shell_escape = true. I just filed #966 which should fix that.

As for your example, $(pwd) is a construct that only works in the shell. Maybe it will work to just write:

shell_escape_cwd = ".."

If not, if you can post or send me (peter@newton.cx) a minimal example demonstrating the failure, I think that would help me understand what's going on.

[mskblackbelt]

@pkgw Switching the directory to .. works for compilation, but doing that means the directory is littered with TeX intermediates (.aux, .bbl, .xdv, etc.) files. Is this unintended, or a consequence of specifying a shell_excape_cwd instead of leaving the default temp directory? If a consequence, is there a way to request that Tectonic copy over a couple of files to the compilation directory?

My project involves a number of lab guides that use a shared header file and a shared bibliography. While I could symlink them into each individual src folder, it would be nice to just give the relative file path in the individual TeX files, then have Tectonic compile labs as I change them.

I guess I'm avoiding symlinks because I don't know how well they're honored in git repositories copied between systems. If I'm worrying over nothing, maybe you can let me know.

[pkgw]

@mskblackbelt Sorry for the slow follow-up here. I can tell you that Git should handle symlinks about as well as you could help — it recognizes them and tracks them in the repo history. I'm not sure what exactly it does if the symlink tries to point outside of the repo, but hopefully that's not relevant in this case. I also don't know exactly how it deals with them on Windows, but on Unixy systems they should pretty much work as if Git wasn't involved.