Skip to content

Commit

Permalink
First set of shell utilities (see #468)
Browse files Browse the repository at this point in the history
  • Loading branch information
josephwright committed Jul 26, 2018
1 parent dcbed19 commit 7b62e64
Show file tree
Hide file tree
Showing 11 changed files with 209 additions and 0 deletions.
159 changes: 159 additions & 0 deletions l3kernel/l3candidates.dtx
Original file line number Diff line number Diff line change
Expand Up @@ -965,6 +965,53 @@
% Execute \meta{tokens} through shell escape at shipout.
% \end{function}
%
% \begin{function}[added = 2018-07-27]{\sys_shell_cp:nn}
% \begin{syntax}
% \cs{sys_shell_cp:nn} \Arg{source} \Arg{dest}
% \end{syntax}
% Copies the files specified in the \meta{source} (which may include
% wildcards) to the \meat{dest}. The file paths should be specified using
% |/| as a path separator. If unrestricted shell escape is not enabled, no
% action is attempted.
% \end{function}
%
% \begin{function}[added = 2018-07-27]{\sys_shell_mkdir:n}
% \begin{syntax}
% \cs{sys_shell_mkdir:n} \Arg{directory}
% \end{syntax}
% Creates the \meta{directory}, which should should be specified using |/| as
% a path separator. If unrestricted shell escape is not enabled, no action is
% attempted.
% \end{function}
%
% \begin{function}[added = 2018-07-27]{\sys_shell_ren:nnn}
% \begin{syntax}
% \cs{sys_shell_ren:nnn} \Ag{path} \Arg{old} \Arg{new}
% \end{syntax}
% Renames the files in the meta{\path} from the \meta{old} to \meta{new}
% name (which may include wildcards). The \meta{path} should be specified using
% |/| as a path separator. If unrestricted shell escape is not enabled, no
% action is attempted.
% \end{function}
%
% \begin{function}[added = 2018-07-27]{\sys_shell_rm:n}
% \begin{syntax}
% \cs{sys_shell_rm:n} \Arg{files}
% \end{syntax}
% Removes the \meta{files} (which may include wildcards). The file path should
% be specified using |/| as a path separator. If unrestricted shell escape is
% not enabled, no action is attempted.
% \end{function}
%
% \begin{function}[added = 2018-07-27]{\sys_shell_rmdir:n}
% \begin{syntax}
% \cs{sys_shell_rmdir:n} \Arg{directory}
% \end{syntax}
% Removes the \meta{directory}, which should should be specified using |/| as
% a path separator. If unrestricted shell escape is not enabled, no action is
% attempted.
% \end{function}
%
% \section{Additions to \pkg{l3tl}}
%
% \begin{function}[EXP,pTF]{\tl_if_single_token:n}
Expand Down Expand Up @@ -3032,6 +3079,118 @@
% \end{macro}
% \end{macro}
%
% \begin{macro}{\sys_shell_cp:nn}
% Simple Unix-like file copying: at some stage we may need a directory-only
% version as Windows and Unix have different requirements here.
% \begin{macrocode}
\cs_new_protected:Npx \sys_shell_cp:nn #1#2
{
\sys_if_shell_unrestricted:T
{
\sys_shell_now:x
{
\sys_if_platform_unix:T
{ cp~-RLf~ \exp_not:N \tl_to_str:n {#1} ~ \tl_to_str:n {#2} }

This comment has been minimized.

Copy link
@jfine2358

jfine2358 Aug 24, 2018

Quoting this line of code, I've asked this question

What are the security risks in this line of LaTeX code, and what can be done to reduce them? The LaTeX3 project wants TeX macros to be able to do file operations via shell escape.

This comment has been minimized.

Copy link
@godbyk

godbyk Aug 24, 2018

Do we need to worry about quoting or escaping special (per-shell) characters at this level?

For example, if I write \sys_shell_cp:nn { /path~with~spaces/in~it.txt } { /destination/file.txt }, will that work properly?

This comment has been minimized.

Copy link
@josephwright

josephwright Aug 24, 2018

Author Member

At present this is all highly experimental, and spaces are likely out. The reason is that we need to quote them to allow the \input primitive to handle them, but we can't nest a second set of " tokens, which we'd need to allow a path-with-spaces to work on Windows.

I'll need to think about quoting more generally: we may well need to provide some functions in this area.

This comment has been minimized.

Copy link
@godbyk

godbyk Aug 24, 2018

Yeah, quoting and escaping is always a bit tricky—compounded by trying to do it in a cross-platform way.

Without quoting, you could potentially do something like \sys_shell_cp:nn { src dest } { ; evil_command } and have LaTeX call evil_command after copying the file.

Also, should these macros emit a warning for the \sys_if_shell_unrestricted:F case (as opposed to silently failing)?

Finally, I should mention that I'm generally in favor of providing some cross-platform abstractions for common file system operations that are likely to be useful in multiple packages and by document authors. One benefit from a security standpoint is that it provides a smaller attack surface. If there's a bug, fixing it solves the issue for all the packages using the \sys_shell_* macro instead of individual packages having to solve the bug themselves.

This comment has been minimized.

Copy link
@jfine2358

jfine2358 via email Aug 24, 2018

This comment has been minimized.

Copy link
@jfine2358

jfine2358 Aug 24, 2018

  • @godbyk You wrote: [security benefit is] smaller attack surface.

I don't agree, even with completely safe escaping. Once 'unrestricted shell escape' is enabled, the door is completely wide open. We have Bobby Drop Tables.

For me the big attack vector is persuading the user (or system admin or whatever) to enable unrestricted shell escape. Don't help the bad guy use the social authority of the LaTeX project. Dear user, just click on a button, so the document can compile.

This comment has been minimized.

Copy link
@godbyk

godbyk Aug 24, 2018

@jfine2358 There are security issues inherent in using the -shell-escape option to allow LaTeX to run any program it likes. That's why -shell-escape isn't enabled by default.

These macros don't enable shell escape. Neither do they encourage the use shell escape per se. They merely provide a common interface for package authors to perform some basic file system manipulation so they don't have to each write their own (potentially buggy) implementations. If LaTeX 3 handles the quoting and escaping well, then that solves a lot of the more common issues.

It sounds like you're taking issue with the -shell-escape option itself. I think complaints about that feature are better addressed elsewhere (perhaps the TeX Live mailing list?). This discussion is about the particular implementation of these macros.

This comment has been minimized.

Copy link
@jfine2358

jfine2358 Aug 25, 2018

I've added issue

State clearly security aspects of l3sys-shell #472

I suggest we move the discussion there.

\sys_if_platform_windows:T
{
xcopy~/y~ \exp_not:N \@@_path_to_win:n {#1} ~
\exp_not:N \@@_path_to_win:n {#2}
}
}
}
}
% \end{macrocode}
% \end{macro}
%
% \begin{macro}{\sys_shell_mkdir:n}
% Windows (with the extensions) will automatically make directory trees but
% issues a warning if the directory already exists: avoid by including a
% test.
% \begin{macrocode}
\cs_new_protected:Npx \sys_shell_mkdir:n #1
{
\sys_if_shell_unrestricted:T
{
\sys_shell_now:x
{
\sys_if_platform_unix:T
{ mkdir~-p~ \exp_not:N \tl_to_str:n {#1} }
\sys_if_platform_windows:T
{
if~not~exist~
\exp_not:N \@@_path_to_win:n { #1 / nul } ~
mkdir~ \exp_not:N \@@_path_to_win:n {#1}
}
}
}
}
% \end{macrocode}
% \end{macro}
%
% \begin{macro}{\sys_shell_ren:nnn}
% The differences in syntax between Unix and Windows mean that it's not easy
% to provide |mv|: instead we go with |ren|-like syntax but with arguments
% set up to work with Unix too.
% \begin{macrocode}
\cs_new_protected:Npx \sys_shell_ren:nnn #1#2#3
{
\sys_if_shell_unrestricted:T
{
\sys_shell_now:x
{
\sys_if_platform_unix:T
{ mv~-f~ \tl_to_str:n { #1 / #2 } ~ \tl_to_str:n { #1 / #3 } }
\sys_if_platform_windows:T
{
ren~ \exp_not:N \@@_path_to_win:n { #1 / #2 } ~
\exp_not:N \tl_to_str:n {#3}
}
}
}
}
% \end{macrocode}
% \end{macro}
%
% \begin{macro}{\sys_shell_rm:n}
% Deletion: obviously a big health warning here!
% \begin{macrocode}
\cs_new_protected:Npx \sys_shell_rm:n #1
{
\sys_if_shell_unrestricted:T
{
\sys_shell_now:x
{
\sys_if_platform_unix:T
{ rm~-f~ \exp_not:N \tl_to_str:n {#1} }
\sys_if_platform_windows:T
{ del~/f~/q~ \exp_not:N \@@_path_to_win:n {#1} }
}
}
}
% \end{macrocode}
% \end{macro}
%
% \begin{macro}{\sys_shell_rmdir:n}
% When removing a directory, we create it first as that avoids errors in
% the Windows case.
% \begin{macrocode}
\cs_new_protected:Npx \sys_shell_rmdir:n #1
{
\sys_if_shell_unrestricted:T
{
\sys_shell_mkdir:n {#1}
\sys_shell_now:x
{
\sys_if_platform_unix:T
{ rm~-r~ \exp_not:N \tl_to_str:n {#1} }
\sys_if_platform_windows:T
{ rmdir~/s~/q~ \exp_not:N \@@_path_to_win:n {#1} }
}
}
}
% \end{macrocode}
% \end{macro}
%
% \subsection{Additions to \pkg{l3tl}}
%
% \begin{macrocode}
Expand Down
5 changes: 5 additions & 0 deletions l3kernel/testfiles/m3expl001.luatex.tlg
Original file line number Diff line number Diff line change
Expand Up @@ -5853,6 +5853,11 @@ Defining \sys_shell_shipout:x on line ...
Defining \__sys_path_to_win:n on line ...
Defining \__sys_path_to_win:w on line ...
Defining \__sys_path_to_win:N on line ...
Defining \sys_shell_cp:nn on line ...
Defining \sys_shell_mkdir:n on line ...
Defining \sys_shell_ren:nnn on line ...
Defining \sys_shell_rm:n on line ...
Defining \sys_shell_rmdir:n on line ...
Defining \tl_if_single_token_p:n on line ...
Defining \tl_if_single_token:nT on line ...
Defining \tl_if_single_token:nF on line ...
Expand Down
5 changes: 5 additions & 0 deletions l3kernel/testfiles/m3expl001.ptex.tlg
Original file line number Diff line number Diff line change
Expand Up @@ -5887,6 +5887,11 @@ Defining \sys_shell_shipout:x on line ...
Defining \__sys_path_to_win:n on line ...
Defining \__sys_path_to_win:w on line ...
Defining \__sys_path_to_win:N on line ...
Defining \sys_shell_cp:nn on line ...
Defining \sys_shell_mkdir:n on line ...
Defining \sys_shell_ren:nnn on line ...
Defining \sys_shell_rm:n on line ...
Defining \sys_shell_rmdir:n on line ...
Defining \tl_if_single_token_p:n on line ...
Defining \tl_if_single_token:nT on line ...
Defining \tl_if_single_token:nF on line ...
Expand Down
5 changes: 5 additions & 0 deletions l3kernel/testfiles/m3expl001.tlg
Original file line number Diff line number Diff line change
Expand Up @@ -5887,6 +5887,11 @@ Defining \sys_shell_shipout:x on line ...
Defining \__sys_path_to_win:n on line ...
Defining \__sys_path_to_win:w on line ...
Defining \__sys_path_to_win:N on line ...
Defining \sys_shell_cp:nn on line ...
Defining \sys_shell_mkdir:n on line ...
Defining \sys_shell_ren:nnn on line ...
Defining \sys_shell_rm:n on line ...
Defining \sys_shell_rmdir:n on line ...
Defining \tl_if_single_token_p:n on line ...
Defining \tl_if_single_token:nT on line ...
Defining \tl_if_single_token:nF on line ...
Expand Down
5 changes: 5 additions & 0 deletions l3kernel/testfiles/m3expl001.uptex.tlg
Original file line number Diff line number Diff line change
Expand Up @@ -5884,6 +5884,11 @@ Defining \sys_shell_shipout:x on line ...
Defining \__sys_path_to_win:n on line ...
Defining \__sys_path_to_win:w on line ...
Defining \__sys_path_to_win:N on line ...
Defining \sys_shell_cp:nn on line ...
Defining \sys_shell_mkdir:n on line ...
Defining \sys_shell_ren:nnn on line ...
Defining \sys_shell_rm:n on line ...
Defining \sys_shell_rmdir:n on line ...
Defining \tl_if_single_token_p:n on line ...
Defining \tl_if_single_token:nT on line ...
Defining \tl_if_single_token:nF on line ...
Expand Down
5 changes: 5 additions & 0 deletions l3kernel/testfiles/m3expl001.xetex.tlg
Original file line number Diff line number Diff line change
Expand Up @@ -5857,6 +5857,11 @@ Defining \sys_shell_shipout:x on line ...
Defining \__sys_path_to_win:n on line ...
Defining \__sys_path_to_win:w on line ...
Defining \__sys_path_to_win:N on line ...
Defining \sys_shell_cp:nn on line ...
Defining \sys_shell_mkdir:n on line ...
Defining \sys_shell_ren:nnn on line ...
Defining \sys_shell_rm:n on line ...
Defining \sys_shell_rmdir:n on line ...
Defining \tl_if_single_token_p:n on line ...
Defining \tl_if_single_token:nT on line ...
Defining \tl_if_single_token:nF on line ...
Expand Down
5 changes: 5 additions & 0 deletions l3kernel/testfiles/m3expl003.luatex.tlg
Original file line number Diff line number Diff line change
Expand Up @@ -5853,6 +5853,11 @@ Defining \sys_shell_shipout:x on line ...
Defining \__sys_path_to_win:n on line ...
Defining \__sys_path_to_win:w on line ...
Defining \__sys_path_to_win:N on line ...
Defining \sys_shell_cp:nn on line ...
Defining \sys_shell_mkdir:n on line ...
Defining \sys_shell_ren:nnn on line ...
Defining \sys_shell_rm:n on line ...
Defining \sys_shell_rmdir:n on line ...
Defining \tl_if_single_token_p:n on line ...
Defining \tl_if_single_token:nT on line ...
Defining \tl_if_single_token:nF on line ...
Expand Down
5 changes: 5 additions & 0 deletions l3kernel/testfiles/m3expl003.ptex.tlg
Original file line number Diff line number Diff line change
Expand Up @@ -5887,6 +5887,11 @@ Defining \sys_shell_shipout:x on line ...
Defining \__sys_path_to_win:n on line ...
Defining \__sys_path_to_win:w on line ...
Defining \__sys_path_to_win:N on line ...
Defining \sys_shell_cp:nn on line ...
Defining \sys_shell_mkdir:n on line ...
Defining \sys_shell_ren:nnn on line ...
Defining \sys_shell_rm:n on line ...
Defining \sys_shell_rmdir:n on line ...
Defining \tl_if_single_token_p:n on line ...
Defining \tl_if_single_token:nT on line ...
Defining \tl_if_single_token:nF on line ...
Expand Down
5 changes: 5 additions & 0 deletions l3kernel/testfiles/m3expl003.tlg
Original file line number Diff line number Diff line change
Expand Up @@ -5887,6 +5887,11 @@ Defining \sys_shell_shipout:x on line ...
Defining \__sys_path_to_win:n on line ...
Defining \__sys_path_to_win:w on line ...
Defining \__sys_path_to_win:N on line ...
Defining \sys_shell_cp:nn on line ...
Defining \sys_shell_mkdir:n on line ...
Defining \sys_shell_ren:nnn on line ...
Defining \sys_shell_rm:n on line ...
Defining \sys_shell_rmdir:n on line ...
Defining \tl_if_single_token_p:n on line ...
Defining \tl_if_single_token:nT on line ...
Defining \tl_if_single_token:nF on line ...
Expand Down
5 changes: 5 additions & 0 deletions l3kernel/testfiles/m3expl003.uptex.tlg
Original file line number Diff line number Diff line change
Expand Up @@ -5884,6 +5884,11 @@ Defining \sys_shell_shipout:x on line ...
Defining \__sys_path_to_win:n on line ...
Defining \__sys_path_to_win:w on line ...
Defining \__sys_path_to_win:N on line ...
Defining \sys_shell_cp:nn on line ...
Defining \sys_shell_mkdir:n on line ...
Defining \sys_shell_ren:nnn on line ...
Defining \sys_shell_rm:n on line ...
Defining \sys_shell_rmdir:n on line ...
Defining \tl_if_single_token_p:n on line ...
Defining \tl_if_single_token:nT on line ...
Defining \tl_if_single_token:nF on line ...
Expand Down
5 changes: 5 additions & 0 deletions l3kernel/testfiles/m3expl003.xetex.tlg
Original file line number Diff line number Diff line change
Expand Up @@ -5857,6 +5857,11 @@ Defining \sys_shell_shipout:x on line ...
Defining \__sys_path_to_win:n on line ...
Defining \__sys_path_to_win:w on line ...
Defining \__sys_path_to_win:N on line ...
Defining \sys_shell_cp:nn on line ...
Defining \sys_shell_mkdir:n on line ...
Defining \sys_shell_ren:nnn on line ...
Defining \sys_shell_rm:n on line ...
Defining \sys_shell_rmdir:n on line ...
Defining \tl_if_single_token_p:n on line ...
Defining \tl_if_single_token:nT on line ...
Defining \tl_if_single_token:nF on line ...
Expand Down

0 comments on commit 7b62e64

Please sign in to comment.