Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shell-escape #38

Open
dbrgn opened this issue May 31, 2017 · 16 comments
Open

Shell-escape #38

dbrgn opened this issue May 31, 2017 · 16 comments

Comments

@dbrgn
Copy link

@dbrgn dbrgn commented May 31, 2017

Does tectonic support the --shell-escape flag? It is used to do things like syntax highlighting with pygments (using the minted.sty package).

@pkgw

This comment has been minimized.

Copy link
Collaborator

@pkgw pkgw commented May 31, 2017

Tectonic very explicitly does not, because the engine is intended to be fully standalone, so that it can have fully reproducible document compiles. Calling out to arbitrary programs that may or may not be available on arbitrary machines breaks that. I really want to discourage the use of shell-escape techniques as much as possible.

However, I believe that biblatex requires a shell-escape to biber (#35), and that seems to be a very popular feature. Tectonic is all about improving the user experience, so I'm willing to be practical.

@dbrgn

This comment has been minimized.

Copy link
Author

@dbrgn dbrgn commented May 31, 2017

The question is if there are alternatives for things like syntax highlighting :)

Might it be possible to write LaTeX packages in Rust? (One of the downsides of that would be that it's not compatible with other implementations anymore though.)

@pkgw

This comment has been minimized.

Copy link
Collaborator

@pkgw pkgw commented May 31, 2017

I just recently had to deal with this — my solution was the listings package, which can do basic syntax highlighting without shell-escape.

I believe the correct solution for this sort of thing is not to attempt to integrate Rust (or, say, Lua) into the TeX engine, but to instead write programs that can generate (La)TeX code that's then fed into the engine. TeX is impossible to parse, but not hard to generate.

@khaledhosny

This comment has been minimized.

Copy link

@khaledhosny khaledhosny commented May 31, 2017

Which is what these programs that are called via shell-escape do :)

@pkgw

This comment has been minimized.

Copy link
Collaborator

@pkgw pkgw commented May 31, 2017

OK yes but they do it at the wrong time :-) You can't encapsulate/embed the TeX engine if it needs to run arbitrary executables at runtime — it should be a handed a complete set of inputs that it can process without needing to talk to the outside world. I don't feel like I've converged on the most persuasive way to make this case but I have an extremely strong engineering/design sense that shell-escape is a bad path to go down.

@dbrgn

This comment has been minimized.

Copy link
Author

@dbrgn dbrgn commented May 31, 2017

It definitely is a bad path, and documentation always warns about that flag, but I don't see a good alternative.

Maybe it could be allowed in an "impure" mode, with proper warnings?

@jtojnar

This comment has been minimized.

Copy link
Contributor

@jtojnar jtojnar commented May 31, 2017

Maybe it could be made to only access an isolated environment like chroot, or even containers with only programs specified in Tectonic.toml available.

wopian added a commit to hibari-moe/api that referenced this issue Feb 19, 2018
Turns out Tectonic isn't as feature complete as it says it is tectonic-typesetting/tectonic#38
@tecosaur

This comment has been minimized.

Copy link

@tecosaur tecosaur commented Oct 1, 2019

@pkgw just my 2c since this seems to just be sitting here.

I really want --shell-escape. It may be bad practice (encapsulation etc.), and I completely get why it feels like a bad idea, but unfortunately a lot of fairly popular, very useful packages depend on this (minted, biber, to name a few).

Would it be possible to add it so that users aren't blocked from using these packages, just strongly discourage it? e.g.

$ tectonic --shell-escape file.tex
*Warning!* --shell-escape is _strongly_ discouraged,
and should be avoided unless absolutely necessary
...
@ratmice

This comment has been minimized.

Copy link
Contributor

@ratmice ratmice commented Oct 1, 2019

If it would be possible to implement minted commands as a rust implementation which doesn't require shell-escape. I have written code to syntax highlight with syntect and convert to tex which does not require shell-escape.

I just don't know yet the right way to export this code to the tex engine, or how to package themes, and language definitions for the bundle, nor whether this should be an external command, feature, or sandboxed webassembly 😬.

All these questions really belong in a separate issue, but i'd much rather put in the effort than see us cave on shell-escape.

@tecosaur

This comment has been minimized.

Copy link

@tecosaur tecosaur commented Oct 1, 2019

Are you proposing creating a fork of minted that uses syntext instead of pygments? If this approach is used, I must admit I have quite a few concerns about the maintanability involved with your own rust-ed forks of minted, biber, TikZ externalise, pythontex, etc.

@ratmice

This comment has been minimized.

Copy link
Contributor

@ratmice ratmice commented Oct 1, 2019

yes, here is an example that is already using tectonic, built into a preprocessor that writes tex like pkgw's comment, recommends.

Reading though reading again he advocates against hooking into rust code to implement tex commands as I advocate above. I understand why, it's a slippery slope as you remind me but given that it can be done without sacrificing reproducibility (and it can) its at least worth considering for only the most popular packages. If only to quell the stalemate.

@tecosaur

This comment has been minimized.

Copy link

@tecosaur tecosaur commented Oct 1, 2019

I see. At the moment I'm tempted by tectonic (hence my commenting here), but the main thing keeping me away is not being able to use my the packages that have become essential to my daily workflow (minted, biber,...). It sounds like at the moment I'll have to stick with the standard engines for --shell-escape, but this may become a viable option in the future.

@pkgw

This comment has been minimized.

Copy link
Collaborator

@pkgw pkgw commented Oct 1, 2019

At this point, it seems to me that it would help a lot of people to add a -Z shell-escape option, so I'm coming around to the prospect of making it available. I certainly don't have the bandwidth to prepare a PR for it, but if someone else does, we'll give it careful consideration.

(I would like to implement any command-line flag with something like a -Z option, cf. #475 which I just created.)

@jjzmajic

This comment has been minimized.

Copy link

@jjzmajic jjzmajic commented Nov 1, 2019

New to using tectonic, but with --shell-escape I really don't see the point. I already see the Makefiles being written. Might as well stick to latexmk. If something like that is necessary, install them like you do latex classes. When tectonic sees [backend=biber] it should know what to do etc.

@durka

This comment has been minimized.

Copy link

@durka durka commented Dec 18, 2019

Do you know where to go in xetex to enable shellescape? I started poking around, but it seems the code to actually run commands is missing. In particular, xetex-shipout.c always prints "runsystem(...)... disabled." Am I looking in the wrong place?

@pkgw

This comment has been minimized.

Copy link
Collaborator

@pkgw pkgw commented Dec 18, 2019

@durka The relevant code was stripped out a long time ago since it got in the way of compilation. Here are some of the relevant commits:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.