Latexmk Crashes After Loading the Second File in a Multi-File LaTeX Project

[gkapfham]

Once again, thank you for your hard work in the vimtex plugin! I have used it in the past several months to write many papers using LaTeX and vim. Recently, I upgraded vimtex using vim-plug and found that some of the basic features on which I rely no longer work correctly.

Before I go into the details about the problem that I am facing, I will share some of the configurations of the environments in which I have noticed concerns. Currently, I am running Ubuntu 14.04 LTS and I can reproduce the problems described in this issue both with Ubuntu's standard version of vim and a new version of vim that I built from source a few days ago.

Here is the version of vim provided by Ubuntu:

VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Jan  2 2014 19:40:46)
Garbage after option argument: "-version"
More info with: "vim -h"

Here are some more details about this version of vim:

:version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Jan  2 2014 19:40:46)
Included patches: 1-52

Here is the version of vim that I recently compiled:

VIM - Vi IMproved 7.4 (2013 Aug 10, compiled May 31 2015 17:09:42)
Garbage after option argument: "-version"
More info with: "vim -h"

Here are some more details about this version of vim:

:version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled May 31 2015 17:09:42)
Included patches: 1-729

I have also created a minimal .vimrc file; you will notice that it is only slightly different than the one that people commonly use when reporting issues about vimtex.

set nocompatible
filetype plugin indent on
syntax enable
set iskeyword+=:
let maplocalleader=","
let mapleader=","

call plug#begin('~/.vim/bundle')
Plug 'https://github.com/lervag/vimtex.git', {'for': 'tex'}
call plug#end()

let g:vimtex_fold_enabled = 0
let g:vimtex_quickfix_mode = 2
let g:vimtex_quickfix_open_on_warning = 1
let g:vimtex_toc_resize = 0
let g:vimtex_toc_hide_help = 1
let g:vimtex_indent_enabled = 1
let g:vimtex_latexmk_enabled = 1
let g:vimtex_latexmk_callback = 0
let g:vimtex_complete_recursive_bib = 0

By default, this version will call xdg-open and this will lead to the loading of evince. I also have a .latexmkrc file that I took from @lervag's GitHub repository of configuration files and enhanced slightly with other configurations that I found in the discussion of issues for vimtex.

$print_type = 'pdf';
$pdf_mode = 1;
$bibtex_use = 2;
push @generated_exts, "cb";
push @generated_exts, "cb2";
push @generated_exts, "spl";
push @generated_exts, "nav";
push @generated_exts, "snm";
push @generated_exts, "tdo";
push @generated_exts, "nmo";
push @generated_exts, "brf";
push @generated_exts, "nlg";
push @generated_exts, "nlo";
push @generated_exts, "nls";
push @generated_exts, "synctex.gz";
push @generated_exts, "tex.latexmain";
push @generated_exts, "run.xml";
$latex = 'latex --src-specials %O %S';
$pdflatex = 'pdflatex -file-line-error -synctex=1 -interaction=nonstopmode -shell-escape %O %S';

$new_viewer_always = 0;
$pdf_update_method = 2;
$pdf_update_signal = 'SIGHUP';

add_cus_dep('nlo', 'nls', 0, 'nlo2nls');
sub nlo2nls {
  system("makeindex $_[0].nlo -s nomencl.ist -o $_[0].nls -t $_[0].nlg" );
}

# vim: ft=perl

The vast majority of the LaTeX documents that I write are divided into multiple files. I followed the vimtex guidelines to add headers to the top of a subfile so that it is clear which file is the main file. To reproduce the fault that I am having on complex LaTeX documents, I have produced the following minimal working example.

First, there is a file that is called minimal.tex:

\documentclass{minimal}
\begin{document}
Hello world first page!
\newpage
Hello world second page!
\newpage
\input{sections/first.tex}
\end{document}

Then, there is the file called first.tex that is in the sections directory:

% vim: ft=tex
% !TEX root = minimal.tex

Hello world third page!!

I am going to write a lot of content now. Does this work?

Now, if I load the minimal.tex file into vim and press <leader>ll to start the compilation of the paper in latexmk I would see the following debugging output, as an example:

b:vimtex
  id : 0


g:vimtex#data[0] : minimal
  root : '/home/gkapfham/working/writing/minimal'
  aux : '/home/gkapfham/working/writing/minimal/minimal.aux'
  log : '/home/gkapfham/working/writing/minimal/minimal.log'
  out : '/home/gkapfham/working/writing/minimal/minimal.pdf'
  tex : '/home/gkapfham/working/writing/minimal/minimal.tex'
  tmp : '/tmp/vFC1lkf/1'
  pid : '13489'
  cmd_latexmk_compile : 'cd ''/home/gkapfham/working/writing/minimal'' && max_print_line=2000 latexmk -pdf -e ''$pdflatex =~ s/ / -file-line-error /''
-pvc -e ''$new_viewer_always = "0"'' -e ''$pdf_update_method = "0"'' -e ''$pdf_previewer = "start xdg-open "'' ''minimal.tex'' >/tmp/vFC1lkf/1 2>&1'
  base : 'minimal.tex'
  viewer
    init : function('4')
    view : function('5')
    latexmk_append_argument : function('6')

Now, if I go into the buffer that contains the first.tex file and press <leader>li to get the debugging information again, then I would produce this output:

b:vimtex
  id : 0


g:vimtex#data[0] : minimal
  pid : 0
  root : '/home/gkapfham/working/writing/minimal'
  aux : '/home/gkapfham/working/writing/minimal/minimal.aux'
  log : '/home/gkapfham/working/writing/minimal/minimal.log'
  out : '/home/gkapfham/working/writing/minimal/minimal.pdf'
  tex : '/home/gkapfham/working/writing/minimal/minimal.tex'
  tmp : '/tmp/vFC1lkf/1'
  cmd_latexmk_compile : 'cd ''/home/gkapfham/working/writing/minimal'' && max_print_line=2000 latexmk -pdf -e ''$pdflatex =~ s/ / -file-line-error /''
-pvc -e ''$new_viewer_always = "0"'' -e ''$pdf_update_method = "0"'' -e ''$pdf_previewer = "start xdg-open "'' ''minimal.tex'' >/tmp/vFC1lkf/1 2>&1'
  base : 'minimal.tex'
  viewer
    init : function('4')
    view : function('5')
    latexmk_append_argument : function('6')

As you can see, latexmk has crashed and updates to this file will not be seen in the PDF that has already been loaded in evince. Typing the <leader>lg command also confirms that latexmk is no longer running correctly.

If I try to restart background compilation by going back to the main file and repeating the above steps, I find that the compiler starts running correctly for the minimal working example. However, when I try all of these steps with larger LaTeX papers that contain many files and many pages, then I have to restart latexmk every time I enter a new file in the sections directory (I could not discover the reason why this behavior is different and decided that the aforementioned issues were important enough that I would go ahead and submit the issue now).

Again, I should stress that this behavior only appeared after a recent update to vimtex. Before that point, I have been able to successfully work on many papers that are structured in the same way as the minimal working example.

Can you please advise on what I need to do to ensure that vimtex starts to work correctly again?

[lervag]

Hi @gkapfham, and thank you for submitting this (and the other) issue(s)! This helps me to remove bugs and to improve the plugin, for which I am grateful.

It seems I might have pushed a buggy commit, for which I am sorry. I just want to inform you that I don't have that much time right now. vimtex is developed on my spare time, and right now my spare time is limited.

I'll look into this issue as soon as possible! Hopefully within the end of this week. Then I'll work my way through your other issues when time permits.

[nasenatmer]

One question: So far my projects usually had all tex files contained in one directory which is why I don't know how to use the texroot marker if input files are in sub dirs. You say that your input file has the following header:

% !TEX root = minimal.tex

Would it make a difference if you declare ../minimal.tex instead of just minimal.tex? Or is vimtex agnostic of directories in this respect?

Also, am I right in assuming that this problem should occur any time one starts latexmk in one buffer and then switches to another buffer? I will just go and try this now as I'm going to continue on another paper of mine…

[nasenatmer]

It seems three ( #176, #177, #178) of your current issues basically rest on one problem: multi file projects with tex files in different sub dirs. But as this one here seems to be the central one, I'm going to answer here again.

I can reproduce your problem of vimtex losing track of the cohesion between complex multi-file projects if I seperate the tex files over various levels of directories. This leads to latexmk stopping compilation and instead vimtex opening several instances of viewer (as reported in #177 and #178).

On the other hand, when I keep all tex files in the same folder, there is no problem. You can switch buffers easily, latexmk continues to run, and recompiles after each :w in the input file. The only thing that doesn't work right now is \lv in the input file, vimtex says vimtex viewer can not read PDF file. But maybe keeping tex files in one folder is a workaround for now?

Due to this, it seems that multi-directory projects are not supported by vimtex so far, although there was one issue recently (#155) that dealt with these kinds of projects and I think there was a solution found.

Maybe this can be a way forward for you? Otherwise, we'll have to wait until @lervag will find some time to look into this.

[gkapfham]

Hello @nasenatmer, thanks for your feedback! I have taken your suggestion and created my own minimal working example that has all of the files in the same directory. However, I see the same behavior as I reported in the first description of this issue. That is, once I go into the subfile called first.tex, latexmk is no longer running and I have to return to the main file to restart the compiler.

There are several other key issues that I should draw out as part of my response to your points. First, I have been able to get mutli-file projects -- organized in exactly the format that I outlined in my first statement of the issue -- to work for several months of time when using vimtex. Second, this functionality only stopped working in, say, the past one to two weeks after I did an update of vimtex. Also, it may be the case that the other issue you referenced has a solution to my stated issue; however, it requires me to use different LaTeX packages and I would like to avoid rewriting a lot of LaTeX code that worked in a previous version of vimtex. Finally, if I run latexmk from the command-line and not from vim using vimtex, then I find that it always works correctly for all of my multi-file projects.

[nasenatmer]

Alright, I wasn't aware that complex multi-file projects™ (as I will call them from now on to refer to those of yours that span over at least two hierarchies of directoriers) did work with vimtex before.

I fully agree with you that rewriting your papers to use subfile or any other package that can deal with several input files is not an option if you don't wish to do this yourself.

It is still interesting, though, that I can't observe the problematic behaviour on Arch with vim: Although it seems I use the same version as you when you compiled yours on 31st of May (mine includes patches 1-729, too).

With vim, I can do 1-3, while with gvim, I can do 1-4:

1. open main file, press \ll and get zathura open with synctex highlighted
2. :tabedit test/test.tex, edit, :write and see the pdf updated automatically
3. gt and edit my main file, :write and see the pdf updated automatically
4. In the input file (test/test.tex), \lv opens a second instance of zathura when I use vim. But synctex forward search works, too, and if I do \lv again, there are no additional instances piling up. With gvim, \lv works just fine.

[gkapfham]

In addition, I am writing to let you know that I have noticed the same reported behavior when I am compiling a main file that uses a bibliography and the bibliography changes. That is, latexmk crashes when I edit the bibliography and then I have to restart it by typing \ll in the buffer for the main file. It is possible that this behavior, specific to the bibliography, has been evident in several past editions of vimtex. Please let me know if you develop any insights or solutions!

[lervag]

Ok, I think I found the problem. In c5d0066 I added some bad initialization code for the latexmk module. I've pushed an update now that seems to fix the problem.

Please test and verify if this solves the present issue.

[lervag]

Btw: The problem was that I reset the PID stored in vimtex. Note that this meant that the latexmk process still exists in the system and was NOT properly stopped by vimtex. Thus this will lead to more problems, especially if you then start a new latexmk process on the same project.

[gkapfham]

After using vim-plug to update vimtex (I see "- vimtex: 2 files changed, 18 insertions(+), 12 deletions(-)
"), I ran some quick tests with the same minimal working example that I mentioned in this issue. Unless I am making a mistake, I cannot confirm that the issue has been resolved.

That is, I still see that latexmk crashes when I move into the buffer than contains the sections/first.tex file. I quickly tried this with the use of xdg-open (which goes to evince for me), zathura, and mupdf, and it seems that I have the same results.

Later, I will also try the new fix with some much more complex LaTeX documents. But, for right now, it seems that, at least on my Ubuntu 14.04 machine, I cannot confirm that the issue is resolved.

[lervag]

That is strange. When you do git show, can you confirm that you get output similar to this:

g--> git show
commit d65f92d7b9e29ad7df277495f519e5e4ccffdf19
Author: Karl Yngve Lervåg <karl.yngve+git@gmail.com>
Date:   Fri Jun 5 22:37:11 2015 +0200

    Don't reset PID if already set (fixes #176)

diff --git a/autoload/vimtex/latexmk.vim b/autoload/vimtex/latexmk.vim
index b9f648b..e2b6e04 100644
--- a/autoload/vimtex/latexmk.vim
+++ b/autoload/vimtex/latexmk.vim
@@ -28,7 +28,10 @@ function! vimtex#latexmk#init(initialized) " {{{1
   " Set compiler (this defines the errorformat)
   compiler latexmk

-  let g:vimtex#data[b:vimtex.id].pid = 0
+  " Initialize system PID
+  if !has_key(g:vimtex#data[b:vimtex.id], 'pid')
+    let g:vimtex#data[b:vimtex.id].pid = 0
+  endif

[gkapfham]

It seems that I do have output just like what you have:

commit d65f92d7b9e29ad7df277495f519e5e4ccffdf19
Author: Karl Yngve Lervåg <karl.yngve+git@gmail.com>
Date:   Fri Jun 5 22:37:11 2015 +0200

    Don't reset PID if already set (fixes #176)

diff --git a/autoload/vimtex/latexmk.vim b/autoload/vimtex/latexmk.vim
index b9f648b..e2b6e04 100644
--- a/autoload/vimtex/latexmk.vim
+++ b/autoload/vimtex/latexmk.vim
@@ -28,7 +28,10 @@ function! vimtex#latexmk#init(initialized) " {{{1
   " Set compiler (this defines the errorformat)
   compiler latexmk

-  let g:vimtex#data[b:vimtex.id].pid = 0
+  " Initialize system PID
+  if !has_key(g:vimtex#data[b:vimtex.id], 'pid')
+    let g:vimtex#data[b:vimtex.id].pid = 0
+  endif

   " Define commands
   command! -buffer       VimtexCompile       call vimtex#latexmk#compile()

Thanks for working on this late in the evening, @lervag! I really appreciate your help.

[lervag]

I'm sorry, it seems I was too quick. It now works with my usual vimrc, but it does not work with the minimal vimrc.

[lervag]

Ah, I see what is happening now. If you add set hidden to the vimrc file, then the problem is fixed. However, this shouldn't in principle be necessary. I'll see if I can find a workaround.

If your main vimrc file does set hidden, then the problem here should be resolved, though.

[gkapfham]

I can confirm that adding set hidden to my personal vimrc file allows me to edit the minimal working example --- and two much more complex LaTeX paper structured in roughly the same way --- without latexmk crashing in the way that it was previously. Great!

There is one interesting wrinkle to note about the debugging output. If I run \lg in the main file, it does output that latexmk is running. However, if I run \lg when I am editing one of the other "internal" files (e.g., the first.tex in the minimal working example), then is the output states that latexmk is not running. But, technically, because the %!TEX root indicates that this is part of a bigger project, then it is, in fact, the case that latexmk is running for the subfile since it is running for the main file.

With that said, this is an extremely minor issue that does not concern me at all. I simply brought it to your attention in case you might see this as being important.

I will try many more large LaTeX documents and report back on my results. At least for now, it seems as though this temporary fix of set hidden does solve the problem correctly. Thanks @lervag!

[lervag]

It seems I have already solved this problem as best I can: The code already parses all buffers to check if the current buffer is the last one for a project before it kills the process. That is, it seems set hidden is actually necessary. I am sorry if this is inconvenient!

Regarding the \lg thing: I can not reproduce this. \lg outputs running both with my normal vimrc and with the minimal vimrc (as long as I have set hidden).

[gkapfham]

Hello again @lervag, I am writing to confirm that all of the changes that you made in the past few days seem to have completely resolved this issue. I can now use vimtex with the minimal working example and with several complex multiple-file LaTeX documents. As I revisit past documents that I have written (most in a similar style to the minimal working example), I will update this issue if I find a case where these fixes did not work correctly. Thanks!

[lervag]

Hi @gkapfham, I am happy to hear that the issue is resolved. Thanks for raising it and for providing all the examples and settings necessary to address the issue!

Btw: I think it is often better to open a new issue if you have follow-up problems and instead reference this issue. That makes it easier to keep the issues focused.