Why does typst generate much larger pdf than TeX?

[Wybxc]

Recently, I started using typst to write my assignments. However, I quickly noticed that the PDF generated by typst is considerably larger (~1.9MB) compared to TeX's output (~120KB).

I am curious if this difference in file size is due to the different fonts used (SourceHansSerif in typst and Fandol series in TeX) or if it is an issue with typst itself?

[alerque]

Write a one line test document using the same font in each, then change up the font and check the size changes in each. That will tell you. I'm guessing you are embedding a full CJK character set font in one situation and not the other. In other words apples and oranges. Report back here if there is an oranges to oranges comparison with a significant difference.

[extua]

I've noticed the same thing with surprisingly large pdfs containing CJK characters, I think it must be font embedding.

[ghost]

My guess (I have only quickly looked at the PDF in a Text Editor):

• PDF codegen is much more compact with PdfTeX, for example ]1 and ] 1 are the same by PDF spec, last one requires one more byte.
• compression is higher (highest) with PdfTeX by default. With Typst (I think) default compression is Default
• PDF Object streams -> puts many pdf objects together (~100+) and compresses all of them in one -> much higher compression than compressing only one pdf object (few bytes) at once.
• PdfTeX has Xref-Streams which is important for creating small xref tables
• more clever codegen for many things. The reason is that PdfLaTeX packages and pdflatex itself are quite good optimized (not perfect) over the last 20 years to create small code, because users complained.
• Typst writes uncompressed XMP Metadata to the PDF file, PdfTeX does not (?)

Possible solution:
optimize the pdf file using pdfcpu.
Run the file using pdfcpu optimize file.pdf and check if it's smaller. If it is, then atleast some of the assumptions above where correct.
See also here.

[laurmaedje]

I think that while of the above things are true, in this case it's pretty much just fonts. The rest is really very small in comparison, typically. Note that XMP Metadata shall be uncompressed so that other programs can find it in the file without understanding the PDF.

[Andrew15-5]

I, too, noticed this. And indeed, the problem is only with Noto Sans CJK JP font (in my case), although I really only tried about 5 different small documents. The size ratio is 1:44.5 for lualatex:typst (53.17 KiB : 2.31 MiB). There is an issue about this, but I don't see it being mentioned in the roadmap. And I wanted to know what is the priority for this.

[Andrew15-5]

I'm currently using mupdf for quick and clean PDF refresh/reload while editing the source file. And since pretty much all documents compile in under 300 ms, it's all good. But I suddenly added a Japanese テスト line (for an unrelated reason), and the results were shocking. The PDF size change from 215.5 KiB to 553.3 KiB, but the real kicker is that compilation time increased by about 3.5 seconds! It was about 250 ms normal, but now about 3.7 seconds. That's a huge increase. I don't know if this is because subsetting isn't optimized or because the font is harder to process(?).

[Andrew15-5]

This is literally what I've added:

+ テスト テスト テスト テスト テスト テスト テスト テスト テスト テスト テスト テスト テスト テスト テスト テスト テスト テスト テスト テスト

If I remove + then the compilation time decreases by about 1.3 seconds.

[KronosTheLate]

I just found that a 33-page paper I wrote was 28 MB. I put it through smallpdf.com, which resulted in a 1.1 MB file that looks the same, has the same hyperlinks etc. In other words, I see no difference, but the file is around 4% of the size. This makes me feel like there is some low-hanging fruit for compression that could and should be applied by default.

For comparison, another paper I wrote a semester before in Overleaf (LaTeX) is 44 pages, but only 4.3 MB. Both papers contain pictures, but I would say the amount and resolution is comparable. This indicates, as people have pointed out above, that LaTeX performs more compression by default.

I am not sure what the downsides are to adding compression, so please point them out. But the benefit does seem rather huge. I discovered this issue because I could not send the paper over facebook messenger due to the 25 MB file size limit. I can also imagine that cloud-storages with PDFs will fill up significantly quicker for typst-users. This kind of practical inconvenience is bound to make the user experience of using typst worse, so the downsides to adding compression should be pretty large before it does not make sense to compress by default.

[0ip]

Consider voting on issue #895 to raise awareness of that problem a little bit 😉
28 MB indeed sounds terrifying for a pdf that has merely 33 pages...

[Enivex]

I am not sure what the downsides are to adding compression

The obvious downside to compressing images is that it's lossy, which typst should definitely not do by default. I suspect that a good chunk of that 28 MB is images.

[Dherse]

It's funny because in my experience typst produces significantly smaller files than latex, maybe it's down to the way I try and use more vectorised graphics since typst has good support for them unlike latex

[yaksher]

This is interesting given that my own long Typst documents (two different ones that are 60 pages) are only 800 kB (neither has any images, but otherwise use a variety of features such as a lot of math, the ctheorems package and correspondingly a bunch of figures, plenty of labels and a good amount of hyperlinks including >100 entry outlines at the front, etc). I also suspect that it's just different decisions in image compression or whatever and if you want to compress your images, you can pretty easily do that yourself.

[Dherse]

I also suspect that it's just different decisions in image compression or whatever and if you want to compress your images, you can pretty easily do that yourself.

We are indeed not optimally compressing png files but other than that we should be pretty much equivalent. Pre-compressing your images would do nothing except for jpeg.

[KronosTheLate]

The obvious downside to compressing images is that it's lossy, which typst should definitely not do by default. I suspect that a good chunk of that 28 MB is images.

This turned out to 100% be the case. I had a 17.2 MB PNG file in the report. When I replaced it with a 4.8 MB JPG, the final file size ended up being 4.9 MB. I am not sure how that works out, as the difference between the JPG and the entrite document is only 0.1 MB, and I have a bunch of text and other pictures as well. Perhaps JPG is somehow compressed by default, as unlike PNG, it is already a lossy format?

So yes, it seems like this one is entirely on me, and that the indication I had for saying that typst produces large PDF's was unfounded. I do have to say that it would be great to be able to opt-in to some amount of compression, as I think for most users the only noticable difference would be smaller files.

[laurmaedje]

We plan to add support for instructing Typst to recompress images at a specific DPI level as an export option in the future.

[laurmaedje]

Regarding 4.8MB and 4.9MB, it's quite possible that Typst compressed the JPG a little smaller than it was before. We do already recompress images, but we don't touch the resolution for now.

[KronosTheLate]

Fantastisk. I am continually impressed by how well typst works, it's continued development, and how present developes (mainly you) are on the github issues and discussions that pop up all the time. Thank you so much for your hard work!

[miliog]

@laurmaedje Would this also apply to SVG files?

[gettalong]

I just ran a sample test with typst using the example from https://github.com/caffeinatedgaze/bare-bones-cv. The resulting PDF from typst is 44790 bytes while compressing it losslessly using HexaPDF leads to a file with just 35097 bytes, i.e. a 22% smaller file. The main difference here is the use of object streams to compress PDF object and to compress streams that aren't yet compressed like the ToUnicode CMap stream.

So adding compression for all streams and adding support for object streams could be an easy fix for larger-than-necessary files.

[Andrew15-5]

why this is something anyone really cares about?

What is the benefit of putting engineering effort into saving 25% of file size?

One of the most obvious reasons is restrictions on file size. Some places can have a limit of 3 MiB (for easy math). Imagine you made a great PDF document that is 4 MiB. Now you are having a major problem just because of file size constraints. You wouldn't have to worry about it if Typst could've saved you an additional 25%, i.e., your file to only be 3 MiB.

If you're using images, then the images are at fault and image compression is an entirely separate question.

True, but images still can be not the only thing that can be optimized (Typst right now can have a hard time optimizing some font formats). There is a new comment about images: #404 (reply in thread).

This seems like more or less the lowest possible priority

IDK if this is really the lowest possible priority, but I can imagine it's definitely not the highest. I would put it at a level of 5/10.

[Enivex]

I can't imagine that many sites have a 3 MB file size limit in 2023. If they do, there are postprocessing tools that can reduce the size of your PDF.

As long as typst's file sizes are reasonable I view this as a complete non-issue.

[yaksher]

Restrictions on file size are increasingly high in my experience—I exceedingly rarely actually hit a file size limit these days. Second, yes, obviously for any given file size limit and any given percent reduction in file size, there would be a collection of files which fit under the threshold. Your example doesn't tell me anymore why I should care than 3 MiB limit, 3.0003 MiB file, and 0.01% reduction in file size. Yes, it is theoretically possible that someone will hit a file size limit somewhere that a size optimization could have prevented, but in that narrow set of circumstances, they can survive and compress the PDF themselves, either via PDF shrinker of the kinds discussed or just through a normal compression mechanism like zipping it. Furthermore, even the 3 MiB example you've provided corresponds to a file hundreds of pages long in Typst from my personal experience, unless it contains images. While those files certainly exist, I feel like any mechanism where you expect to distribute files that are 200 pages long can handle a couple megabytes.

In practice, the smallest file size limit I can remember encountering in the last 5 years is 8 megabytes, which is more than enough for basically any file that doesn't consist of images. Even email tends to support 25 megabytes.

The only circumstance where really optimizing your PDF size seems significant is if it's being served as part of a webpage, where bandwidth is at somewhat of a premium, but in that extremely narrow circumstance, you can simply... pass it through one of the tools used to demonstrate that there's room for improvement (though amusingly, the Typst web app is presumably one such example).

Given modern storage and bandwidth, optimizations for files on the order of a couple megabytes seem like a misuse of time and effort. 40 years ago or 20 years ago (or maybe even 10 years ago, though not really) or whatever, it mattered a lot more and so LaTeX developed accordingly, but that doesn't mean that Typst needs to be in any rush to match up to LaTeX in that category when there are much more practical issues (like basically anything that affects what the output looks like or how easy/convenient Typst is to write) to improve.

[Andrew15-5]

I also forgot to mention that I just want to see the lowest possible value of the file size. It makes me super excited/happy to see a humongous PDF file that's only 512 KiB or something like that. Because I sure felt this way when I was using LaTeX. (Can't remember how good were file sizes from LO Writer/Google Docs/MS Word.) And right now some fonts or text in general isn't as dense (in PDF) as with LaTeX.

I can't imagine that many sites have a 3 MB file size limit in 2023.

Well, I certainly saw restriction of 5 MiB or 10 MiB. As I said, 3 MiB is just to easily get 25% of 4 MiB.

BTW, MB and MiB are completely different things, SI units are deprecated (for information size).

[Andrew15-5]

they can survive and compress the PDF themselves, either via PDF shrinker

Yes, but we are talking about Typst built-in compression (optimization of unoptimized PDF files). It's nice to be able to avoid using 3rd party apps/programs to additionally compress your PDF file.

or just through a normal compression mechanism like zipping it

This can very easily be a non-option, as a "place" can only allow PDF files and not archive files.

Furthermore, even the 3 MiB example you've provided corresponds to a file hundreds of pages long in Typst

As I mentioned before, there are many unoptimized things in Typst. The great example is huge PDF files with just CJK font/text. The font format in which CJK fonts are stored is very poorly subsetted/optimized by Typst. You can easily hit a 3 MiB mark with a few pages, I believe.

Given modern storage and bandwidth, optimizations for files on the order of a couple megabytes seem like a misuse of time and effort.

I have a git repository that have a lot of PDF files that are not as optimized/small as I would like them to be (not made by me). And every additional mebibyte or even half of it can stack up really quickly. And PDF files repo can turn from "text-like file sizes" to "binary-like file size" very quick. And cloud based git repositories can also have restrictions on how large a git repo can be, therefore limiting the amount of (PDF) files it can have.

I clearly understand that it's (file size) not always the problem, but it is in some/many cases, and the end goal is to make file sizes as small as possible (but making Typst still fast to recompile on watch). It is not a first-class issue and therefore can be fixed after everything else has been implemented (in Typst), i.e., no one is rushing to solve this problem.

[gettalong]

Just adding another data point for this discussion. Below you will find the result of a benchmark I created for HexaPDF that shows the running time as well as memory usage and resulting file size for outputting the Gutenberg text of the 'Odyssey' on pages with a width of 100pt and 50pt and a height of 1000pt (granted, this is a very specific benchmark for testing how long words that need to be broken on short lines affect the runtime/memory usage but nonetheless interesting, I think):

|--------------------------------------------------------------------|
|                              ||    Time |     Memory |   File size |
|--------------------------------------------------------------------|
| hexapdf     | 100 ttf        |  1.223ms |  99.592KiB |     606.546 |
| prawn       | 100 ttf        |  6.544ms |  69.692KiB |   1.015.734 |
| reportlab/C | 100 ttf        |  1.336ms |  51.016KiB |     920.045 |
| fpdf2       | 100 ttf        |  2.258ms |  80.660KiB |     888.224 |
| tcpdf       | 100 ttf        |  1.606ms |  43.472KiB |   1.200.745 |
|--------------------------------------------------------------------|
| hexapdf     | 50 ttf         |  3.253ms | 287.232KiB |     767.852 |
| prawn       | 50 ttf         | 10.946ms |  70.120KiB |   1.572.098 |
| reportlab/C | 50 ttf         |  1.643ms |  52.936KiB |   1.249.287 |
| fpdf2       | 50 ttf         |  2.911ms | 100.468KiB |   1.320.952 |
| tcpdf       | 50 ttf         |  2.388ms |  52.428KiB |   1.941.055 |
|--------------------------------------------------------------------|

As you can see HexaPDF produces a file that is markedly smaller than the others, in both cases (still need to optimize the 50pt width case in terms of performance and memory usage, though). This is achieved by compressing all streams and making sure that the output of PDF objects is as compact as possible.

[laurmaedje]

subsetting is unfortunately also built in-house ... rustybuzz dropped harfbuzz's subsetter from the port.

[Dherse]

subsetting is unfortunately also built in-house ... rustybuzz dropped harfbuzz's subsetter from the port.

I my bad, i thought they had abandoned making a fully featured sub setter, this area is still pretty unclear to me

[gettalong]

... better image encoding (i.e png predictors). I have tried looking into the later and i couldn't make heads of tails of what little information i found 🤷‍♂️

The PDF standard is delegating to the PNG spec in this regard, see http://www.w3.org/TR/PNG-Filters.html. I'm not sure if this would be helpful but here is HexaPDF's code in Ruby for this: https://github.com/gettalong/hexapdf/blob/master/lib/hexapdf/filter/predictor.rb#L1C1-L238C12.

Basically, you encode each row of an image separately, for some predictors also referencing values of the previous line. And you always work with bytes regardless of the bit depth set which makes things easier and faster.

If you need help with implementing that part, I might be able to provide general pointers (nothing Rust specific due to my marginal knowledge of Rust).

[Dherse]

@gettalong thanks a lot! I had been looking for implementations, but clearly my Google-fu is out of practice 😅

[gettalong]

@Dherse No problem, feel free to poke me for information if you need something.

[squeaktoy]

I found a very simple way to compress PDFs. I just use this ghostscript command:

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/prepress -dNOPAUSE -dQUIET -dBATCH -sOutputFile=compressed_PDF_file.pdf input_PDF_file.pdf

It works great on a resume PDF which went from 9 MB to just over 100 KB.
I found this command on https://itsfoss.com/compress-pdf-linux/

I also found that pdfcpu didn't decrease the file size on my PDF but ghostscript did.

[Andrew15-5]

gs -sDEVICE=pdfwrite -dPDFSETTINGS=/default -dCompatibilityLevel="$version" \
  -dNOPAUSE -dQUIET -dBATCH -dDetectDuplicateImages \
  -dSubsetFonts=true -dCompressFonts=true -dConvertCMYKImagesToRGB=true \
  -r150 -sOutputFile="$compressed_name" -f "$1"

[Kreijstal]

I don't mind makes pdf more editable if they contain more info (which can be more accessible to people with for example screenreaders), but I can see why you would dislike it, I guess we can have an optimize flag or something like that