Discarding file paths causes issues

[rm-dr]

While working on #14, I came across the following problem:

fithesis4 does not work correctly with tectonic. The package comes with many different .def files, all of which have the same name and are thus ignored in the bundle.

Two example problem files are below. These are versions of the document class, stored under each department's directory.

• texmf-dist/tex/latex/fithesis/locale/mu/law/fithesis-english.def
• texmf-dist/tex/latex/fithesis/locale/mu/med/fithesis-english.def

A MWE is below, which is a stripped version of this template.

\documentclass[
	digital,
	twoside,
	nolof,
	nolot
]{fithesis4}

\usepackage[main=english]{babel} 

\thesissetup{
	date          = \the\year/\the\month/\the\day,
	university    = mu,
	faculty       = sci,
	type          = bc,
	programme     = NA,
	field         = Sport Management,
	department    = Department of Social Sciences and Sport Management,
	author        = Jane Doe,
	gender        = f,
	advisor       = {Prof. RNDr. John Smith, CSc.},
	title         = The use of LaTeX for the Typesetting of Sports Tables,
	TeXtitle      = The use of \LaTeX\ for the Typesetting of Sports Tables,
	keywords      = {keyword1, keywords2, ...},
	TeXkeywords   = {keyword1, keywords2, \ldots},
	abstract      = {This is the abstract}
}

\begin{document}
	test
\end{document}

This provides correct styling with vanilla XeLaTeX and TeXlive, but fails with tectonic.

fithesis4 is a very specific package that few people will use, but it highlights an issue with tectonic's bundle system. Packages that are written in a similar way will also break.

[rm-dr]

An update:

fithesis4 uses the full path of each file whenever it calls \input. Tectonic isn't ready for this, and CoreBridgeState.input_open() returns a null pointer for those files. This is why the above MWE outputs a blank document.

I've temporarily fixed this by patching fithesis4 in the bundle, but that's not a real solution.
Who knows how many other packages are silently failing? We can't possibly patch them all.

[vlasakm]

I think that TeX Live tries to mostly keep the files in TeX search paths (i.e. non-source, non-documentation) uniquely named and even disallows common names not prefixed by the name of the package or not using a different file extension (like test.tex), but see below. Perhaps fithesis4 was allowed because the files are not confusable with any other package, but the use of the directory structure is indeed unfortunate in any case.

IIRC TeX was designed at the time where a single directory was assumed (maybe even on the whole system) and classic TeX had no notion of directories or search paths and that these were added and unified with kpathsea and the TeX directory structure (TDS).

I think that Tectonic should support the notion of directories in TeX. TDS depends on them and uses them in ways, which I think manifest very much in the issues you have been uncovering with Tectonic's simplified model. For example, different engine + format combinations use different search paths, e..g:

TEXINPUTS.xelatex       = $TEXMFDOTDIR;$TEXMF/tex/{xelatex,latex,xetex,generic,}//
TEXINPUTS.xelatex-dev   = $TEXMFDOTDIR;$TEXMF/tex/{latex-dev,xelatex,latex,xetex,generic,}//
TEXINPUTS.xeplain       = $TEXMFDOTDIR;$TEXMF/tex/{xeplain,eplain,plain,xetex,generic,latex,}//
TEXINPUTS.xetex         = $TEXMFDOTDIR;$TEXMF/tex/{xetex,plain,generic,latex,}//

As Tectonic AFAIK doesn't strive to be LaTeX-only, I think that we need work with these search paths requirements dynamically at Tectonic runtime, instead of more simply considering the search path statically just when creating the bundle (but even that would be an improvement and get rid of some conflicts).

Some packages deliberately use these engine/format specific paths and include multiple files with the same name, e.g. there is etex.src in tex/plain/etex/etex.src that is used for all TeX engines except for LuaTeX, which has tex/luatex/hyph-utf8/etex.src (earlier) in the search paths. latex vs latex-dev is another case of this.

I think it would be useful to have a notion of TeX flat directories and texmf (TDS compliant) trees. In TeX flat directories every file type is viable, while in texmf trees file types are searched according to the TDS. This is standard in TeX Live, and I think it is quite nice (see for example https://github.com/vlasakm/context#configuration for example use cases). Instead of using environment variables like TEXMFDOTDIR and TEXMF, Tectonic could use something like -Z search-path and -Z search-tree.

[vlasakm]

Also, the latex-dev thing and search paths in general are I think what is problematic in #13 and #14.

[rm-dr]

Closing, since this is solved by #14