tlmgr creates nul file on Windows

[netique]

Hi,

every now and then a nul file appears inside my project directory after knitting to PDF. That would be no big deal, but this special file cannot be removed by standard means and it really messes up git.

I've found out that this is due to tlmgr and could finally reproduce the issue with tinytex::tlmgr_update(). Some fancy "debugging" and system process monitoring revealed that {tinytex}'s tlmgr calls curl, shipped with Rtools42 on Windows. curl tries to download something but ends up creating the nul file. Note that this won't happen when you specify some non-default CTAN (I mean not using --repository ctan) as pointed out at https://tug.org/pipermail/tex-live/2019-July/043908.html. Lastly, if I run tlmgr update --list --self in CMD in the same directory, no nul appears as it doesn't use curl (according to Process Monitor EDIT: seems like it does).

Apparently, you can specify what to use in TeX environment table, as described here: https://tug.org/texlive/doc/tlmgr.html#ENVIRONMENT-VARIABLES. I wonder if it can be mitigated somehow on {tinytex} side.

EDIT: The issue seems to originate somewhere around https://github.com/TeX-Live/texlive-source/blob/fc5589360e8051b3c578d09ad6afdc212d375267/texk/tests/TeXLive/TLUtils.pm#L4164. I'll contact TUG mailing list with the issue.

Many thanks!

Session info:

R version 4.1.2 (2021-11-01)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22000), RStudio 2021.9.1.372

Locale:
  LC_COLLATE=Czech_Czechia.1250  LC_CTYPE=Czech_Czechia.1250    LC_MONETARY=Czech_Czechia.1250 LC_NUMERIC=C                  
  LC_TIME=Czech_Czechia.1250    

Package version:
  graphics_4.1.2  grDevices_4.1.2 stats_4.1.2     tinytex_0.36    tools_4.1.2     utils_4.1.2     xfun_0.29      

LaTeX version used: 
  TeX Live 2021 (TinyTeX) with tlmgr 2021-10-04

---

By filing an issue to this repo, I promise that

• I have fully read the issue guide at https://yihui.org/issue/.
• I have provided the necessary information about my issue.
  • If I'm asking a question, I have already asked it on Stack Overflow or RStudio Community, waited for at least 24 hours, and included a link to my question there.
  • If I'm filing a bug report, I have included a minimal, self-contained, and reproducible example, and have also included xfun::session_info('tinytex'). I have upgraded all my packages to their latest versions (e.g., R, RStudio, and R packages), and also tried the development version: remotes::install_github('yihui/tinytex').
  • If I have posted the same issue elsewhere, I have also mentioned it in this issue.
• I have learned the Github Markdown syntax, and formatted my issue correctly.

I understand that my issue may be closed if I don't fulfill my promises.

[netique]

Seems to be fixed very quicky after my message to tex-live mailing list: https://tug.org/pipermail/tex-live/2022-February/047741.html. Maybe nothing needed at {tinytex} side after all.

[cderv]

Thanks a lot for having looked into that !

This is something I needed to investigate as it was reported also in rstudio/tufte#107 and I encountered that to, but I did not manage to pin point the source.

Let's see if that fixes it !

[cderv]

I reinstall the daily version of TinyTeX which bundle daily TeX Live (using tinytex::reinstall_tinytex()) and for now it is working well.

You could also try that on your end ?

[netique]

Nope, nul stil pops up... There is now ongoing discussion in the texlive mailing list, so maybe there will be more patches coming. But it is so weird - when I do the exact same tlmgr command in, say, Desktop directory, it is going quite well without nul.

[cderv]

But it is so weird - when I do the exact same tlmgr command in, say, Desktop directory, it is going quite well without nul.

So that is why I can't reproduce all the time. 🤔

I'll try to follow the discussion but thanks for keeping us updated. Hopefully they will find the source.

[netique]

Now I am 99% sure that system2() or using command prompt in Terminal pane in RStudio causes this behavior. When I open cmd in the same directory, run the same command, no nul appears.

I appended a few simple debugging lines to TLUtils.pm and the code behaves differently in RStudio and in Windows CMD or PowerShell.

sub query_ctan_mirror_curl {
  my $max_trial = 3;
  my $warg = (win32() ? "-w %{url_effective} " : "-w '%{url_effective}' ");
  for (my $i = 1; $i <= $max_trial; $i++) {
    # -L -> follow redirects
    # -s -> silent
    # -w -> what to output after completion
    my $cmd = "$::progs{'curl'} -Ls "
              . "-o " . nulldev() . " "
              . $warg
              . "--connect-timeout $NetworkTimeout "
              . "--max-time $NetworkTimeout "
              . $TeXLiveServerURL;
    ddebug("query_ctan_mirror_curl: cmd: $cmd\n");
    debug("\n\n\n____this cmd wil be run in 10 sec:\n$cmd\n________");
	sleep(10);
    my $url = `$cmd`;
	debug("____nul file should appear now____\n");
	debug("____url was: $url ______\n");
	debug("____sleep now for 20 sec______\n");
	sleep(20);
    if (length $url) {
      # remove trailing slashes
      $url =~ s,/*$,,;
      ddebug("query_ctan_mirror_curl: returning url: $url\n");
      return $url;
    }
    sleep(1);
  }
  return;
}

In system shell, URL reads proper mirror URL, in RStudio or with system2 it reads %url_effective. Pretty interesting.

[cderv]

Pretty interesting indeed!

Regarding system2() behavior, does it happen only inside RStudio or in a R console outside also ?

[netique]

In a plain R console, the nul is generated as well and the mirror URL is still %url_effective which seems wrong.

[cderv]

So could be related to how system2() works and its configuration ?

[netique]

I think so. I'll send a message to R-devel mailing list.

[cderv]

Did you try already how the sdtout and other argument impacts the behavior ?
I did not manage to reproduce yet what you have done to try myself.

[netique]

Yes, I did try. Without success. However, the problem lies in rtools40 curl, which cannot hadle curl -Ls -o nul -w %{url_effective} --connect-timeout 30 --max-time 30 https://mirror.ctan.org command in TLUtils.pm. Using Windows curl explicitly, it works just fine. Waiting to hear more from r-devel mailing list.

[cderv]

Really insteresting all that ! For reference, link to the thread: https://stat.ethz.ch/pipermail/r-devel/2022-February/081474.html

Thank you very much for all this investigation !

[netique]

I think I deciphered this conundrum, see https://tug.org/pipermail/tex-live/2022-February/047748.html

You can fix it in {tinytex} texlive "flavor" or wait for maintainers to push to texlive daily build.

[cderv]

Thanks, that is good news ! We'll probably wait for a fix to be pushed in TeX Live. Then it should be available in TinyTeX daily build.

[yihui]

I'm truly impressed by your debugging skills! I really appreciate the effort.

I was able to reproduce this problem with an older version of TinyTeX. I just reinstalled the latest version, it seems the problem has gone.

[netique]

Thanks, @yihui. May I ask you for TinyTeX versions that you tested? I think the main issue is caused by curl 7.64.1 (x86_64-pc-msys) shipped with rtools40. If I remove rtools40 from the PATH, everything works like a charm.

EDIT: I use TinyTeX version v2022.02, but tlmgr is of revision 60693 (2021-10-04 04:24:25 +0200) in this release of TinyTeX.

[yihui]

Sorry I was wrong. The actual reason was that the first time I tried to compile a document to pdf, tinytex figured that it needed to install a few missing packages (hence needed to access a CTAN mirror), which created this nul file. Then I deleted the nul file, and also upgraded TinyTeX. The second time I compiled the document, the missing packages had been installed, so tlmgr didn't need to access the internet. As a result, I no longer saw the nul file.

I just deliberately deleted a LaTeX package, recompiled the document, and the nul file was still generated (as you would expect).

I was using tlmgr from 2021-12-24, which has definitely not included the fix from TeX Live developers yet.

[cderv]

I think texlive changes are not yet inside our daily build - revisions with change regarding this are 61960 and 61996

Once they are out we could test again against curl 7.64.1 to see if this is fixed or not.

[netique]

Well I am afraid those revisions won't help. The issue really is in "unix" curl that is used primarily in system2() calls if rtools are on PATH (and then even in terminal in RStudio). TeXLive developers seem to be concerned with further changes that are required for this curl on Windows. For standard Widows curl, there really is no issue whatsoever.

However I believe this could be tackled on TinyTeX/user side. There is ENV variable available that can tell tlmgr to use its own curl. See https://www.tug.org/texlive/doc/tlmgr.html#ENVIRONMENT-VARIABLES. I set my TEXLIVE_PREFER_OWN to 1 and it works perfectly.

EDIT: TEXLIVE_DOWNLOADER = wget did not work afterall, but TEXLIVE_PREFER_OWN = 1 is the way. I can see it utilizes

C:\\Users\\netik\\AppData\\Roaming\\TinyTeX\\tlpkg\\installer\\curl\\curl.exe --version
curl 7.66.0 (i386-pc-win32)

TinyTeX curl that works fine.

[yihui]

Great! I just applied the fix. You can test the development version via

remotes::install_github('yihui/tinytex')

Thanks a lot!

[netique]

Works great!