zipped synctex

[honekamp]

In older versions of Sumatra PDF, it was possible to have a zipped synctex file and still backwards and forward search would work just fine. Starting 3.4.x, the support for a zipped synctex file seems to have ceased. I have to unzip the synctex file in order to do a forward or backward search. Has this change been applied intentionally? Is there any way to work around the change?

[GitHubRulesOK]

Probably not intentional, last time I looked it was working
Argh! it was not zipped can you provide a good mini sample (one page with few simple variations will do)
mwe.pdf mwe.tex and mwe.synctex.gz (you may need to lump all in a conventional zip to paste in a comment 🙂 )

@kjk
You may have changed synctex gzip library handling inadvertently?

[honekamp]

I‘m in the process of creating a simple example. It may take a little while, I just want to make sure that I get it right.

[honekamp]

Test.zip

I have created a small example that hopefully exemplifies the issue. As long as I don‘t unzip the synctex file, neither forward nor backward search work for me. After unzipping, search works again. Just in case the information is relevant, I‘m using Sublime Text with LaTeX Tools for authoring LaTeX sources.

[GitHubRulesOK]

Thanks
editor should not matter it is the command part of syntax with pdflatex -1 synctex or something (bad memory) where there are many (15?) variations 0=uncompressed

[GitHubRulesOK]

I should have asked which version of SumatraPDF ?

[honekamp]

I‘m using 3.4.3, the latest as far as I know.

[GitHubRulesOK]

Hmmmm
@kjk
It seems to work as expected in 3.5.15009

the file is compressed (large at 32kb but 7 pages and last paragraph starts at mwe.tex 154 = Duis...... which tallies too
I thus have to suggest it is the editor configuration of SumatraPDF
and recollect there were others with similar recent question using VScode due to a plugin change.

[GitHubRulesOK]

Ahh
@kjk 3.4.3 seems to be the culprit 3.5... OK

3.4.3 say no synchronization open issue #2642

[honekamp]

Thanks a lot for the investigation!

[kjk]

@honekamp thanks for the sample.

However, it works for me in 3.4.3 (64 -bit, installer).

What I did:

• extracted the Test.zip file
• I opened mwe.pdf
• I double-clicked on a word
• I have notepad++ installed so sumatra auto-detected it and launched notepad++ with mwe.tex positioned at the right line

This is all using mwe.synctex.gz.

So what happens in your setup when after you double-click on a word? Nothing? An error message?

Are you using a portable version or installer?

[kjk]

I've tested mwe.pdf with portable version of 3.1.2, 3.2, 3.3 and 3.4.3, 64-bit, and they all work with mwe.synctex.gz and notepad++

The auto-detected inverse-search command is: "C:\Program Files\Notepad++\notepad++.exe" -n%l "%f"

So maybe the issue with your setup is inverse-search command. What is it?

[honekamp]

It is strange that the sample does not exhibit the described behavior, but the whole issue is strange.

Maybe I should mention that I‘m a daily user of Sumatra PDF (installer) for a long time (i.e. since before it reached 1.0). My setup is very stable, I‘ve used Sublime Text as my LaTeX editor all these years (with a short excursion to VS Code in between, because new and shiny). Both forward (from Sublime to Sumatra PDF) and inverse search (from Sumatra PDF to Sublime) worked reliably day by day all these years until I switched from Sumatra PDF 3.3.3 to 3.4.3 this week.

For forward search, Sumatra PDF 3.4.3 gives me an error message that the „Synchronization file cannot be opened“. The error message for inverse search is „No synchronization info at this position“.

I tried the 3.5 pre-release build, but that one crashes during the forward search in my real projects. If I install 3.3.3, everything is back to normal, both search directions work with the zipped synctex file.

After reading the error message, I tried to find a hint how to work around the issue. After some experimentation, I found out that the search in both directions started working again in 3.4.3 as soon as I unzipped the zipped synctex file. Unfortunately, I cannot provide a sample of the real projects I‘m working on because these are very complex and the build requires some tooling effort. The documents are typically slightly below or larger than a 1000 pages.

Of course, a possible work-around for me could be to just tweak the build workflow such that the synctex file is left uncompressed.

[kjk]

The crash you saw with pre-release should be fixed.

I've added logging to pre-release around synctex handling. The build should be ready in about 1 hr.

When it is, could you test with pre-release (15032 or later)?

We narrowed it down to Sumatra failing to parse synctex.gz, most likely because it fails to open it.

I've added logging around opening / parsing .synctex.gz files.

When the issue happens with pre-release, you can use Ctrl + K for command palette and then show log to open the log in notepad. Then you could post the logs here.

My only guess so far that it's a timing issue.

Are your projects big? How big is .synctext.gz file?

Maybe when Sumatra is trying to load synctext.gz it isn't yet fully generated and that's why it fails to parse. The logs would provide more info.

[honekamp]

sumatra-log.zip

I have uploaded the log file obtained from build 15033 for the case of a zipped synctex file. Forward search does not work, and the error message is „No synchronization file found“. If I unzip the synctex file and try to do a forward search, then build 15033 simply crashes.

Please note that I have deleted several thousand lines of file states in the log file because these might contain sensitive information.

BTW, the syntex.gz file for the project I ran the experiment on is a about 12 MB.

[kjk]

Please retry with pre-release 15039 or later and submit the logs.

I've added logging of synctex code so that we can pin point what goes wrong there.

Also, I've added a retry with 1 sec so if my theory about timing issue is right, this could fix it.

[honekamp]

I tried with build 15051. Initially, I initially made a mistake because the project I was taking for a text had an non-zipped synctex file. The log obtained from that experiment is probably worthless. I deleted the non-zipped synctex file such that only the zipped version remains.

A forward search yields the error message "No synchronization file found" and does not create a log file. Inverse sync does not work and the error message "No synchronization info at this position" is created.

[GitHubRulesOK]

@honekamp
KJK is the developer thus able to see where the issue may be however, I was at one time able to emulate your problem.
I have to agree it is likely to be that when you double click, SumatraPDF needs to expand a compressed file so as to read, the encoded placement co-ordinates of the tex line reference and associated filename, for the reverse call (open file.tex at line I)

Thus SumatraPDF on a fast machine may not get the slower unpacked co-ordinates from a GZ file rather than the uncompressed version, however without knowing the method I also expect that would potentially take longer on any page due to a bigger GZ needing more time to unpack.

For checking if gz is working I suggest first try a small example such my very small (and poor example) highlight.zip at #2642 (comment)

Unpacking the zip as a folder of three files should never respond in the pdf with "No synchronization file found", that would suggest there is a file system issue (such as not using the folder on conventional local drive)?

There will be only two results one of which is an editor should report "file not found" from the highlighted function, (it should attempt to open a file not in that folder) the other result should be click on title (or many other text area locations and the tex opens at line 23.
OR gz problem says "No synchronization info at this position" (which can be normal for margins or other white spaces)

[honekamp]

I tried the "highlite" example using 3.4.3 and I get very inconsistent results from it. If I start at the TeX file with a forward search while the PDF file is not already open in Sumatra PDF, the PDF opens in Sumatra PDF and the I get the error message "Synchronization file cannot be opened". If I then start an inverse search, the error message is "No synchronization info at this position".

If I load the file into Sumatra PDF and then do an inverse search, everything works like it should. If, after the successful inverse search, I do a forward search, I get the error message "Unknown source file".

[GitHubRulesOK]

"unknown source file" (or similar) message from your editor when clicking places other than Title should be indicating reverse IS working (that is the point of that highlight test) so confused by you saying getting similar in a forward search

The easier test is double click Title should show line 23 in editor confirming current.tex %file location and %line number were correct (synctex is not top to bottom nor based on word locations) That only proves a known compressed file thats quick to decompress is functioning.

More telling is

not already open in Sumatra PDF, the PDF opens in Sumatra PDF and the I get the error message "Synchronization file cannot be opened"

1st rule of Tex compilation is keep every resource in the one working folder (no /sub/chapter or /aux/folders), its tough enough getting remote or relative TextMF folders to work consistently. so SumatraPDF should know the .synctex.gz is in the same working folder.

Hinting that SumatraPDF did not detect the companion file from its presence (should not be the case) or had some problem opening the file to decompress

[kjk]

So far what I can tell from logs is that .synctex.gz file exists and we call synctex_scanner_new_with_output_file which fails for some reason.

This is unfortunately synctex parser code (not written by us) so no idea what's going on there.

I'll add code to pipe synctex parser logs to our logging so that we can get insight on why synctex parser fails.

However, I didn't change synctex parser code in 3.4 so if it worked in 3.3 it should work in 3.4

Given the very large size of synctex.gz file I do suspect timing issue i.e. the file not being fully generated when we try to access it.

Maybe a simple hack of re-doing syntex.gz parsing with 1 sec delay (if the file exists and fails the first time) would be a fix.

[GitHubRulesOK]

@kjk
see above HOWEVER after running round in circles with various versions It seems Portable Works for me (always preferred it for LaTeX)
So wonder if its related to all users installation ?? (which is the one that fails)

[GitHubRulesOK]

@honekamp
are you always testing with installed version (you said 3.4.3) but where is sumatrapdf.exe located ?

[honekamp]

The exe is located in the default install location, i.e. (if memory serves, currently afk) in c:\Programs\SumatraPDF

[GitHubRulesOK]

@honekamp
Can you please try with a portable copy of 3.4.3 using your files in your installation (its a suggested workaround for now)

• Run the portable copy once to generate in its own folder sumatrapdf-settings.txt
• Copy from old settings advanced options the line starting InverseSearchCmdLine = " to a similar position in new portable settings file
• Then ONLY using your biggest pdf with a synctex.gz check if a double-click works

If you wish you can (backup editor settings) point existing editor at portable sumatrapdf.exe
if all is well it should allow resumption of 2 way working

[honekamp]

@GitHubRulesOK, I followed your recipe, but was unable to get the forward or inverse search to work. I got the already known error messages.

Then I took the portable version from the website and gave it a try. With the portable version everything works just fine, forward and inverse.

BTW, the install directory of Sumatra PDF is in %appdata%\local. My recollection from memory was incorrect.

[kjk]

@honekamp I narrowed it down to synctex using zlib's gzopen to try to read the file and failing with rather generic "bad file" error code.

Can you e-mail me (kkowalczyk@gmail.com) the synctex.gz file that's failing?

There's no more logging I can think of that would give me more information.

For some reason synctex is failing to open that particular .gz file. But it works for other .gz files so it's not just generic problem.

If I can step through the code in the debugger, I might learn why this is happening.

At some point I switched from zlib to zlib-ng, which is supposed to be faster. Not sure when but if I did it for 3.4 and zlib-ng has problems with this file, then it would explain why it always worked before 3.4 and now it sometimes fails.

[kjk]

@honekamp I got your file, thanks.

So far I was unable to reproduce the problem.

Here's what I did: I created a PDF with a name matching the synctex.gz file, opened it in Sumatra and double clicked in the PDF. The inverse search works i.e. an editor is launched trying to load the original .tex file (which I don't have but that's not important).

I tried with installed (just released) 3.4.4 64 bit, I tried 64 bit / 32 bit, release / debug build. .synctex.gz works in all of them.

When you do similar steps on your machine, does double-clicking not result in inverse search? If yes, then I could only think about differences in processor that make zlib code not work.

So there goes my theory that this is zlib-ng problem.

And we're back to theory of a timing issue i.e. that when you ask Sumatra to do the commands, the .synctex.gz is not yet fully created and fails to load.

But that's not something I can recreate easily on my machine.

Do you have additional anti-virus software installed? If yes can you disable it and retry?
Also disable Window's anti-virus software (Virus & threat protection settings turn off Real-time protection).
Or exclude the relevant folders from virus scanning (it's Exclusions for Windows and I'm sure other anti-virus software has similar settings).

The reason is that anti-virus software hooks into OS file operations and scans newly created files. That can make those files unavailable for other processes for a while. The bigger the file, the longer the scanning takes.

This would explain why the file would file to open at one time but then could be opened later.

Also, is the drive where .synctex.gz file generated a slow one, like network drive or Dropbox/One Drive or USB drive etc.? Slow drive would increase probability of timing issues.

If yes, try with C:\ drive.

[honekamp]

@kjk I can‘t deactivate the virus protection, sorry. The projects I work on are stored on the internal SSD of my computer, so I would expect that speed should not be an issue.

What I don‘t understand about the whole issue is that I was able to get along with my work environment for years, until I installed 3.4.3. And as I mentioned in one other comment above, I can use the portable 3.4.3 and search works fine in both directions. If I go back to 3.3.3 for the installed version, search works again fine in both directions on the same projects that don’t work on 3.4.3.

As I mentioned, I could work around the issue by making sure the the build toolchain for my projects creates non-zipped syntex files. That way, I can at least continue with my work.

I am really grateful for the way you guys have responded to my problem report, it is very much appreciated. But maybe this is the point where it does not make sense to continue investigating without having further clues. I will think about how I can provide more information that may help to dig deeper.

[kjk]

@honekamp Makes sense, I ran out of ideas to test as well. Thanks for providing the information.

[GitHubRulesOK]

@honekamp
Thanks for feedback
I am running many "dirty" systems (as elevated user) so often cause my own problems, on this occasion It seems to be 64bit inside a Sandbox with Windows PDF search active is possibly having more problems here than without search interfering. However that should not be version related.

One major issue is when users have background file monitors like One Drive, AntiVirus or "Everything" file application etc. Those see the file changes such as unpacking as something to hang on to for inspection. The "One Second" delay should workaround that if they don't hold the file{s} open longer.

[GitHubRulesOK]

@honekamp Are you installing pre-viewer / search filter in all 64bit install ?

[honekamp]

@GitHubRulesOK, I‘m sorry, I don‘t understand the question.

[GitHubRulesOK]

@honekamp
During your install did you select options

[honekamp]

No, I have never touched these options before. I always do the straight install with no options.

[kjk]

@honekamp the 32-bit vs 64-bit would be consistent with issue being zlib-ng. For speed it has assembly implementations that are different for different processor.

I've switched pre-release from zlib-ng back to regular zlib so could you try latest pre-release (both 64-bit and 32-bit release install)?

Another thing I can try is to move reading of .synctex.gz from synctex parser code to my code. But this is a bigger change that I really want to make to synctex parser.

[GitHubRulesOK]

@kjk
just to update that a fresh install of 3.4.4 inside sandbox appears to be working with synctex (aswell as outside),
but I was careful NOT to include iFilter, so unconfirmed if that was my factor.

[honekamp]

I've switched pre-release from zlib-ng back to regular zlib so could you try latest pre-release (both 64-bit and 32-bit release install)?

Sorry, @kjk, trying to download the 64 bit pre-release gives me a 404. Unfortunately, I'll not be able to run further tests in the coming weeks. I'll check back as soon as I can and keep you posted on the results I'll get from pre-release-builds.

[GitHubRulesOK]

try https://kjkpub.s3.amazonaws.com/software/sumatrapdf/prerel/15093/SumatraPDF-prerel-64-install.exe

[GitHubRulesOK]

@kjk since no 32bit pre-rel the 64bit is also 404

[kjk]

@GitHubRulesOK Thanks, 404 should be fixed now.

[honekamp]

I tried the pre-release from AWS, and unfortunately there is no progress. Forward-search gives me "No synchronization file found" (I proved the existence of the sync file later by installing 3.3.3, which worked on the existing synctex.gz file) and inverse search says "No synchronization info at this position". As mentioned, I'll unfortunately not have time for further test, but will pick up again as soon as possible.

[T-F-S]

I followed the discussion and I had the same problem, i.e. synctex search was broken with version 3.4.x, and I had to roll back to version 3.3.3.

Today, I tried 3.4.6 (64 bit installer) and synctex forward and backward search work now for me as in 3.3.3. Version History does not name the problem or its solution, but now it seems to be solved (?).

[honekamp]

Today, I tried both 3.4.6 and the latest pre-release. Using the zipped synctex file, sync unfortunately does not work in either of them on my machine. I only get the error messages "No synchronization file found" and "No synchronization info at this position", as reported earlier already.

Meanwhile, I have found a viable solution for changing the PDF build such that the synctex file does not get zipped any longer. As mentioned before, the forward and inverse search works for me on the basis of a non-zipped synctex file.

[T-F-S]

I have to supplement my last comment: Version 3.4.6 works for me partially for zipped synctex files, meaning that "No synchronization file found" errors appear more or less often. Closing and opening sumatrapdf resets such errors, but they appear frequently and are quite annoying for a larger project.

Using synctex=-1 to work with unzipped synctex files as @honekamp suggested seems to work much better. I switched my settings yesterday and I did not have any error since.

[MarcusTressl]

I have the same problem with all compressed synctex files in all versions of SumatraPDF that I installed after version 3.1.2 (including 3.4.6): If the synctex file is decompressed, forward and inverse search work well. If it is compressed, neither forward nor inverse search works. Inverse search responds with a "No synchronization file found" message, so I guess at least this problem is not related to the TeX editor. Working with Windows 10, SumatraPDF x64, WinEdt 5.5.
[Version 3.1.2 of SumatraPDF x64 works with the compressed synctex file (and the decompressed version).]
Attached is a pdf and its compressed synctex file and the log (if that is of any use) where the problem occurs:
commands.log
commands.synctex.gz
commands.pdf

[ruijiacao]

I encountered problems similar to those reported by @honekamp. A good work-around I found was to use the pdfsync package.