Problem with gnuplot, compile in parallel and \gpgetvar

[dflvunoooooo]

I don't know if this is even possible. If I want to access a variable from inside a gnuplot CacheMeCode environtment and I enable compile in parallel the compilation fails. I think this is because there is a rerun necessary to load the in parallel created figures, and until this has happend latex does not know anything about the variable. Is that correct? Or is there any way to avoid this? The error message from the main log is:

The file robustExternalize/robExt-52F599C0FDCA78B4CBA2BE02B20B6927.tex already 
exists.
LaTeX Font Info:    Font shape `TU/lmtt/bx/n' in size <10.95> not available
(Font)              Font shape `TU/lmtt/b/n' tried instead on input line 21.
LaTeX Font Info:    External font `cmex10' loaded for size
(Font)              <10.95> on input line 21.
LaTeX Font Info:    External font `cmex10' loaded for size
(Font)              <8> on input line 21.
LaTeX Font Info:    External font `cmex10' loaded for size
(Font)              <6> on input line 21.
Finished to include the file. [1
! Undefined control sequence.
<write> ...paces Variable = \gp@var@test_variable 
                                                  }}{\thepage }{}\protected@...
l.26 \end{document}
                   
The control sequence at the end of the top line
of your error message was never \def'ed. If you have
misspelled it (e.g., `\hobx'), type `I' and the correct
spelling (e.g., `I\hbox'). Otherwise just continue,
and I'll forget about whatever was undefined.

]

Package robExt Warning: Warning: Compiling all missing figures in parallel
(robExt)                with "xargs -t -I "{}" -P 16 sh -c "{}" <
(robExt)                ""robust-externalize
(robExt)                providevars"-robExt-compile-missing-figures.sh"". You
(robExt)                need to rerun LaTeX to include them.


xargs -t -I "{}" -P 16 sh -c "{}" < ""robust-externalize providevars"-robExt-co
mpile-missing-figures.sh"
runsystem(xargs -t -I "{}" -P 16 sh -c "{}" < ""robust-externalize providevars"
-robExt-compile-missing-figures.sh")...executed.



! Package robExt Error: The pdf file
(robExt)               
robustExternalize/robExt-52F599C0FDCA78B4CBA2BE02B20B6927.pdf
(robExt)                is not present. The compilation command "cd
(robExt)                robustExternalize/ && gnuplot -c
(robExt)                "robExt-52F599C0FDCA78B4CBA2BE02B20B6927.tex" &&
(robExt)                echo "ok" >
(robExt)                "robExt-52F599C0FDCA78B4CBA2BE02B20B6927.pdf"" used
(robExt)                to compile the environment on line 21 certainly
(robExt)                failed, see logs above or in
(robExt)               
robustExternalize/robExt-52F599C0FDCA78B4CBA2BE02B20B6927.log.

Type <return> to continue.
 ...                                              
                                                  
l.26 \end{document}
                   

LaTeX does not know anything more about this error, sorry.

There is no other log file. Here is the code:

\documentclass{scrreport}

\usepackage{robust-externalize}
\setPlaceholder*{__ROBEXT_LATEX_ENGINE__}{xelatex}
\runHereAndInPreambleOfCachedFiles{
    \usepackage{gnuplottex}
    \usepackage{gnuplot-lua-tikz}			
}
\robExtConfigure{
    compile in parallel,
}
\cacheTikz

\begin{document}
    
    \begin{figure}
        \centering  
        \begin{CacheMeCode}{gnuplot, tikz terminal={providevars test_variable}}
            test_variable = 5
            plot cos(x) title "Test2"
        \end{CacheMeCode}
        \caption{Variable = \gpgetvar{test_variable}}
    \end{figure}
    
\end{document}

[tobiasBora]

So in parallel mode, the compilation is done at the very end of the compilation of the latex document, so \gpgetvar cannot know the variable since it does not exist yet. So you have two versions to handle this.

Solution 1

The first option is to define a custom \gpgetvar that checks if the variable exists. This way you could get during the first compilation:

instead of an error. Note that you need to compile 3 times for this to converge if you also use \cacheTikz (unless you use compile in parallel after=N to speed up compilation if you have few images).

\documentclass{scrreport}

\usepackage{robust-externalize}
\setPlaceholder*{__ROBEXT_LATEX_ENGINE__}{xelatex}
\runHereAndInPreambleOfCachedFiles{
  % Generate with gnuplot -e "set terminal tikz createstyle"
  % and make sure to copy gnuplot-* files in the robustExternalize/ folder (I'm looking for a nicer solution) 
  \usepackage{gnuplottex}
  \usepackage{gnuplot-lua-tikz}			
}
\robExtConfigure{
  compile in parallel,
}
\cacheTikz

\makeatletter
\NewDocumentCommand{\mygpgetvar}{m}{%
  \ifcsname gp@var@#1\endcsname\gpgetvar{#1}\else \emph{Please recompile to load variable:}\texttt{\detokenize{#1}}\fi%
}
\makeatother

\begin{document}
    \begin{figure}
        \centering  
        \begin{CacheMeCode}{gnuplot, tikz terminal={providevars test_variable}}
            test_variable = 5
            plot cos(x) title "Test2"
        \end{CacheMeCode}
        \caption{Variable = \mygpgetvar{test_variable}}
    \end{figure}
\end{document}

Solution 2

The other solution is to pass the information from the latex file to the gnuplot file (this solution applies to any sort of file) using set placeholder={__TEST_VARIABLE__}{5}:

\documentclass{scrreport}

\usepackage{robust-externalize}
\setPlaceholder*{__ROBEXT_LATEX_ENGINE__}{xelatex}

\runHereAndInPreambleOfCachedFiles{
  \usepackage{gnuplottex}
  \usepackage{gnuplot-lua-tikz}			
}
\robExtConfigure{
  compile in parallel,
  %% Create a new placeholder that you can use later:
  set placeholder={__TEST_VARIABLE__}{5},
}
\cacheTikz

\begin{document}
    
    \begin{figure}
        \centering  
        \begin{CacheMeCode}{gnuplot, tikz terminal={providevars test_variable}}
          test_variable = __TEST_VARIABLE__
          plot cos(test_variable * x) title "Test2"
        \end{CacheMeCode}
        \caption{Variable = \getPlaceholder{__TEST_VARIABLE__}}
    \end{figure}

\end{document}

[tobiasBora]

Ok, I added in the source code a new function \robExtGpgetvar basically equal to mygpgetvar in solution 1, so now you can just do:

\begin{CacheMeCode}{gnuplot, tikz terminal={providevars test_variable}}
test_variable = 5
plot cos(test_variable * x) title "Test2"
\end{CacheMeCode}
Variable is \robExtGpgetvar{test_variable}.

[dflvunoooooo]

Awesome, thank you for all your work! Will test this :)

[dflvunoooooo]

This is working well for a simple plot. But not for a more complex one. I couldn't really figure out why not. There is still the warning: Warning: you need to recompile as the gpgetvar(robExt) variable "Beta" does not exist yet. But I recompiled six times already. Besides, this warning appears twice.
Here is my code:

\documentclass{scrreport}

\usepackage{robust-externalize}
\setPlaceholder*{__ROBEXT_LATEX_ENGINE__}{xelatex}
\runHereAndInPreambleOfCachedFiles{
    \usepackage{gnuplottex}
    \usepackage{gnuplot-lua-tikz}			
}
\robExtConfigure{
    compile in parallel,
}
\cacheTikz

\begin{document}
    
\begin{figure}
    \centering 
    \begin{CacheMeCode}{gnuplot, dependencies={Versuch_4_5_Gruppe_N_am_01_12_2023.txt}, tikz terminal/.expanded={providevars Beta, Gamma, Rnull}}
        set decimalsign locale "de_DE.UTF-8"                       # Setzt , als Dezimaltrennzeichen in den Daten.

        Beta = 1.0
        Gamma = 5
        f(x) = Beta * x**Gamma
        Rnull = 0.665
        fit f(x) "./Versuch_4_5_Gruppe_N_am_01_12_2023.txt" every 1::615::678 using 3:(($6/$5)-Rnull) via Beta, Gamma
        title_f = sprintf('Beta = %.12f',Beta)
        
        ##########################              Plot
        set logscale xy
        plot './Versuch_4_5_Gruppe_N_am_01_12_2023.txt' every 1::615::678 using 3:(($6/$5)-Rnull) linecolor 2 title "erwärmend",\
        f(x) linecolor 7 linewidth 2 title title_f
    \end{CacheMeCode}
    \caption{Test \robExtGpgetvar{Beta}.}
\end{figure}
[Versuch_4_5_Gruppe_N_am_01_12_2023.txt](https://github.com/leo-colisson/robust-externalize/files/13714392/Versuch_4_5_Gruppe_N_am_01_12_2023.txt)
    
\end{document}

Edit: siunitx package not required.

[tobiasBora]

Ahahaah sorry, I just realized I hardcoded the name test_variable in the code, I guess I forgot to replace it with #1 after debugging. This should be fixed now. Sorry for these bugs, you are quite good as finding them, thanks a lot.

[dflvunoooooo]

Ah a classic. It is working now, thank you again :)

[dflvunoooooo]

Is \robExtGpgetvar{} providing a number or a string? Because I usually try to format the variables to avoid far too much points after decimal with \num[round-mode=figures,round-precision=3]{} for example. This also converts the number to the local appearance of numbers (in german for example a , as decimal point). If I try to apply this to the \robExtGpgetvar{} command, siunitx complains about an "invalid number". Here is my code:

\documentclass{scrreport}

\usepackage{robust-externalize}
\setPlaceholder*{__ROBEXT_LATEX_ENGINE__}{xelatex}
\runHereAndInPreambleOfCachedFiles{
    \usepackage{siunitx}
    \usepackage{gnuplottex}
    \usepackage{gnuplot-lua-tikz}			
}

\begin{document}
    
\begin{figure}
    \centering  
    \begin{CacheMeCode}{gnuplot, tikz terminal={providevars banane}}
        banane = 5.000001828383748e-19
        plot cos(x) title "Test2"
    \end{CacheMeCode}
    \caption{Variable = \num[round-mode=figures,round-precision=3]{\robExtGpgetvar{banane}}}
\end{figure}
    
\end{document}

Edit: This does not work if I create my own function and return a 1 in case the variable is not set.

\documentclass{scrreport}

\usepackage{robust-externalize}
\setPlaceholder*{__ROBEXT_LATEX_ENGINE__}{xelatex}
\runHereAndInPreambleOfCachedFiles{
    \usepackage{siunitx}
    \usepackage{gnuplottex}
    \usepackage{gnuplot-lua-tikz}			
}

\makeatletter
\NewDocumentCommand{\mygpgetvar}{m}{%
    \ifcsname gp@var@#1\endcsname\gpgetvar{#1}\else 1\fi%
}
\makeatother

\begin{document}
    
\begin{figure}
    \centering  
    \begin{CacheMeCode}{gnuplot, tikz terminal={providevars banane}}
        banane = 5.000001828383748e-19
        plot cos(x) title "Test2"
    \end{CacheMeCode}
    \caption{Variable = \num[round-mode=figures,round-precision=3]{\mygpgetvar{banane}}}
\end{figure}
    
\end{document}

Edit2: This does also not work with the set placeholder option.

\documentclass{scrreport}

\usepackage{robust-externalize}
\setPlaceholder*{__ROBEXT_LATEX_ENGINE__}{xelatex}

\runHereAndInPreambleOfCachedFiles{
    \usepackage{gnuplottex}
    \usepackage{gnuplot-lua-tikz}			
}
\robExtConfigure{
    compile in parallel,
    %% Create a new placeholder that you can use later:
    set placeholder={__TEST_VARIABLE__}{5},
}
\cacheTikz

\begin{document}
    
    \begin{figure}
        \centering  
        \begin{CacheMeCode}{gnuplot, tikz terminal={providevars test_variable}}
            test_variable = __TEST_VARIABLE__
            plot cos(test_variable * x) title "Test2"
        \end{CacheMeCode}
        \caption{Variable = \num[round-mode=figures,round-precision=3]{\getPlaceholder{__TEST_VARIABLE__}}}
    \end{figure}
    
\end{document}

[tobiasBora]

Oh, the first time it outputs a string with a warning, the second time a number. I just added \robExtGpgetvarNb{foo} that prints 404 by default. If you want another number, just use the optional argument like \robExtGpgetvarNb[-1]{foo}.

Edit: wait, weird that your second test also fail, let me try, I might have pushed a bit fast.

[dflvunoooooo]

Oh, the first time it outputs a string with a warning, the second time a number. I just added \robExtGpgetvarNb{foo} that prints 404 by default. If you want another number, just use the optional argument like \robExtGpgetvarNb[-1]{foo}.

Edit: wait, weird that your second test also fail, let me try, I might have pushed a bit fast.

This does not work for me. Without the \num{} command I get the 404 and the second time I compile it displays the number. But siunitx still complains if I wrap the \robExtGpgetvarNb{} in a \num{} command.

[tobiasBora]

Oh, the new version should work this time. It seems like \siunit needs an expandable command, so we need \NewExpandableDocumentCommand, and this also means that we cannot call a warning inside to say that the variable does not exist yet. But it should not be too much of an issue, as the variable exists before the figure is actually printed. I'll close for now, let me know if you have an issue with this one.

[tobiasBora]

Actually, I’ll reopen until I create also an expandable version of \getPlaceholder.

[tobiasBora]

For now, a quick workaround is to call \getPlaceholderInResult{__TEST_VARIABLE__} before the \num and use \robExtResult inside like:

\documentclass{scrreport}

\usepackage{robust-externalize}
\setPlaceholder*{__ROBEXT_LATEX_ENGINE__}{xelatex}

\runHereAndInPreambleOfCachedFiles{
  \usepackage{siunitx}
  \usepackage{gnuplottex}
  \usepackage{gnuplot-lua-tikz}			
}
\robExtConfigure{
  compile in parallel,
  %% Create a new placeholder that you can use later:
  set placeholder={__TEST_VARIABLE__}{5},
}
\cacheTikz

\begin{document}

\begin{figure}
  \centering  
  \begin{CacheMeCode}{gnuplot, tikz terminal={providevars test_variable}}
    test_variable = __TEST_VARIABLE__
    plot cos(test_variable * x) title "Test2"
  \end{CacheMeCode}
  \caption{Variable =
    \getPlaceholderInResult{__TEST_VARIABLE__}%
    \num[round-mode=figures,round-precision=3]{\robExtResult}}
\end{figure}


\end{document}

% !TeX TXS-program:compile = txs:///pdflatex/[--shell-escape]
% Local Variables:
% TeX-command-extra-options: "--shell-escape"
% End:

[dflvunoooooo]

Oh, the new version should work this time. It seems like \siunit needs an expandable command, so we need \NewExpandableDocumentCommand, and this also means that we cannot call a warning inside to say that the variable does not exist yet. But it should not be too much of an issue, as the variable exists before the figure is actually printed. I'll close for now, let me know if you have an issue with this one.

Great! You are the best! With the newest version \robExtGpgetvarNb{} is working :)

[dflvunoooooo]

For now, a quick workaround is to call \getPlaceholderInResult{__TEST_VARIABLE__} before the \num and use \robExtResult inside like:

\documentclass{scrreport}

\usepackage{robust-externalize}
\setPlaceholder*{__ROBEXT_LATEX_ENGINE__}{xelatex}

\runHereAndInPreambleOfCachedFiles{
  \usepackage{siunitx}
  \usepackage{gnuplottex}
  \usepackage{gnuplot-lua-tikz}			
}
\robExtConfigure{
  compile in parallel,
  %% Create a new placeholder that you can use later:
  set placeholder={__TEST_VARIABLE__}{5},
}
\cacheTikz

\begin{document}

\begin{figure}
  \centering  
  \begin{CacheMeCode}{gnuplot, tikz terminal={providevars test_variable}}
    test_variable = __TEST_VARIABLE__
    plot cos(test_variable * x) title "Test2"
  \end{CacheMeCode}
  \caption{Variable =
    \getPlaceholderInResult{__TEST_VARIABLE__}%
    \num[round-mode=figures,round-precision=3]{\robExtResult}}
\end{figure}


\end{document}

% !TeX TXS-program:compile = txs:///pdflatex/[--shell-escape]
% Local Variables:
% TeX-command-extra-options: "--shell-escape"
% End:

This workaround is working as well. Great!

[dflvunoooooo]

I found another hiccup. If I insert my exported number into a siunitx table, the format settings of this table are not applied. Here is the code:

\documentclass{scrreport}

\usepackage{siunitx}
\usepackage{robust-externalize}
\setPlaceholder*{__ROBEXT_LATEX_ENGINE__}{xelatex}
\runHereAndInPreambleOfCachedFiles{
    \usepackage{gnuplottex}
    \usepackage{gnuplot-lua-tikz}			
}

\begin{document}
    
\begin{figure}
    \centering  
    \begin{CacheMeCode}{gnuplot, tikz terminal={providevars bananen, birnen, aepfel}}
        bananen = 5.00034209230958
        birnen = 5.0864684681684684
        aepfel = 42.684640684608460
        plot cos(x) title "Test2"
    \end{CacheMeCode}
    \caption{Variable = \robExtGpgetvar{bananen}}
\end{figure}

\begin{table}
    \begin{tabular}{S[table-format=1.6] S[table-format=1.6] S[table-format=1.6]}
        \robExtGpgetvarNb{bananen} & \robExtGpgetvarNb{birnen} & \robExtGpgetvarNb{aepfel} \\
    \end{tabular}
\end{table}

\end{document}

Leading to far too long numbers and an overlapping.

But I don't know if this is a robust-extrenalize or siunitx problem.

[tobiasBora]

Leading to far too long numbers and an overlapping.

The last error is independent of this package, as you get exactly the same error if you manually replace the \robExtGpgetvarNb with their variable. After a quick search, you want table-auto-round:

\documentclass{scrreport}

\usepackage{siunitx}
\usepackage{robust-externalize}
\setPlaceholder*{__ROBEXT_LATEX_ENGINE__}{xelatex}
\runHereAndInPreambleOfCachedFiles{
    \usepackage{gnuplottex}
    \usepackage{gnuplot-lua-tikz}			
}

\begin{document}
    
\begin{figure}
    \centering  
    \begin{CacheMeCode}{gnuplot, tikz terminal={providevars bananen, birnen, aepfel}}
        bananen = 5.00034209230958
        birnen = 5.0864684681684684
        aepfel = 42.684640684608460
        plot cos(x) title "Test2"
    \end{CacheMeCode}
    \caption{Variable = \robExtGpgetvar{bananen}}
\end{figure}

\begin{table}
    \begin{tabular}{S[table-format=1.6,table-auto-round] S[table-format=1.6,table-auto-round] S[table-format=1.6,table-auto-round]}
        \robExtGpgetvarNb{bananen} & \robExtGpgetvarNb{birnen} & \robExtGpgetvarNb{aepfel} \\
    \end{tabular}
\end{table}

\end{document}

[tobiasBora]

It was hard to make \getPlaceholder expandable, but I made \getPlaceholderNoReplacement expandable, so for placeholders that do not contain other placeholders (you can evaluate a placeholder to get to that point if needed), no need to use \robExtResult as above, you can just do \num[round-mode=figures,round-precision=3]{\getPlaceholderNoReplacement{__TEST_VARIABLE__}}:

\documentclass{scrreport}

\usepackage{robust-externalize}
\setPlaceholder*{__ROBEXT_LATEX_ENGINE__}{xelatex}

\runHereAndInPreambleOfCachedFiles{
  \usepackage{siunitx}
  \usepackage{gnuplottex}
  \usepackage{gnuplot-lua-tikz}			
}
\robExtConfigure{
  compile in parallel,
  %% Create a new placeholder that you can use later:
  set placeholder={__TEST_VARIABLE__}{5},
}
\cacheTikz

\begin{document}

\begin{figure}
  \centering  
  \begin{CacheMeCode}{gnuplot, tikz terminal={providevars test_variable}}
    test_variable = __TEST_VARIABLE__
    plot cos(test_variable * x) title "Test2"
  \end{CacheMeCode}
  \caption{Variable = \num[round-mode=figures,round-precision=3]{\getPlaceholderNoReplacement{__TEST_VARIABLE__}}}
\end{figure}


\end{document}

% !TeX TXS-program:compile = txs:///pdflatex/[--shell-escape]
% Local Variables:
% TeX-command-extra-options: "--shell-escape"
% End:

[tobiasBora]

Since fully expandable placeholders have little benefit, I guess I will stop here with making more functions expandable. If someone really can't live with the above workarounds, I will see in due time.

[dflvunoooooo]

Leading to far too long numbers and an overlapping.

The last error is independent of this package, as you get exactly the same error if you manually replace the \robExtGpgetvarNb with their variable. After a quick search, you want table-auto-round:

\documentclass{scrreport}

\usepackage{siunitx}
\usepackage{robust-externalize}
\setPlaceholder*{__ROBEXT_LATEX_ENGINE__}{xelatex}
\runHereAndInPreambleOfCachedFiles{
    \usepackage{gnuplottex}
    \usepackage{gnuplot-lua-tikz}			
}

\begin{document}
    
\begin{figure}
    \centering  
    \begin{CacheMeCode}{gnuplot, tikz terminal={providevars bananen, birnen, aepfel}}
        bananen = 5.00034209230958
        birnen = 5.0864684681684684
        aepfel = 42.684640684608460
        plot cos(x) title "Test2"
    \end{CacheMeCode}
    \caption{Variable = \robExtGpgetvar{bananen}}
\end{figure}

\begin{table}
    \begin{tabular}{S[table-format=1.6,table-auto-round] S[table-format=1.6,table-auto-round] S[table-format=1.6,table-auto-round]}
        \robExtGpgetvarNb{bananen} & \robExtGpgetvarNb{birnen} & \robExtGpgetvarNb{aepfel} \\
    \end{tabular}
\end{table}

\end{document}

You are right, I am sorry. I was under the impression, that this happend automatically in another project of mine, but I can not find it and obviously it shouldn't work. Thanks again.

[dflvunoooooo]

It was hard to make \getPlaceholder expandable, but I made \getPlaceholderNoReplacement expandable, so for placeholders that do not contain other placeholders (you can evaluate a placeholder to get to that point if needed), no need to use \robExtResult as above, you can just do \num[round-mode=figures,round-precision=3]{\getPlaceholderNoReplacement{__TEST_VARIABLE__}}:

\documentclass{scrreport}

\usepackage{robust-externalize}
\setPlaceholder*{__ROBEXT_LATEX_ENGINE__}{xelatex}

\runHereAndInPreambleOfCachedFiles{
  \usepackage{siunitx}
  \usepackage{gnuplottex}
  \usepackage{gnuplot-lua-tikz}			
}
\robExtConfigure{
  compile in parallel,
  %% Create a new placeholder that you can use later:
  set placeholder={__TEST_VARIABLE__}{5},
}
\cacheTikz

\begin{document}

\begin{figure}
  \centering  
  \begin{CacheMeCode}{gnuplot, tikz terminal={providevars test_variable}}
    test_variable = __TEST_VARIABLE__
    plot cos(test_variable * x) title "Test2"
  \end{CacheMeCode}
  \caption{Variable = \num[round-mode=figures,round-precision=3]{\getPlaceholderNoReplacement{__TEST_VARIABLE__}}}
\end{figure}


\end{document}

% !TeX TXS-program:compile = txs:///pdflatex/[--shell-escape]
% Local Variables:
% TeX-command-extra-options: "--shell-escape"
% End:

Your are awesome, thank you for your great support! :)

[dflvunoooooo]

Oh, the first time it outputs a string with a warning, the second time a number. I just added \robExtGpgetvarNb{foo} that prints 404 by default. If you want another number, just use the optional argument like \robExtGpgetvarNb[-1]{foo}.

Edit: wait, weird that your second test also fail, let me try, I might have pushed a bit fast.

Is it possible to write a warning in the logs, if the referenced variable wasn't found and 404 was given?

[tobiasBora]

This is solved in latest version, but this works only in lualatex due to some fundamental latex limitations.

[dflvunoooooo]

I see, will think about switching to lualatex. Thank you.

[tobiasBora]

I'm curious, what is the advantage of xelatex compared to lualatex for you?

[dflvunoooooo]

Don't really know. I set everything up a few years ago. I had to switch from pdflatex to one of those two, and if I remember correctly I read that lua was slower and there where some problems with signs over letters (don't know the correct word for it), so I chose xelatex. I was relatively new to latex and wanted a running system primarily.