memory consumption and cache not releasing memory

[felixpinkert]

I've happily used pdf-view on a relatively high-memory and low-resolution machine so far, but now that I've switched to a lower-memory, high-resolution (3k) machine, I notice that pdf-view hogs huge amounts of memory and that the cache clearing mechanism does not seem to work. For example, 100 page simple B&W pdf document easily makes emacs use 500MB of memory.

Closing the file does not release that memory, and reducing pdf-cache-image-limit has no effect on memory consumption. The cache limit does have an effect on scrolling speed, as pages have to get rendered again if I go over the page limit, but memory consumption keeps increasing with every new page scrolled to even when the limit has already been exceeded. Likewise, the defuns for clearing the cache don't seem to take effect - I wrapped pdf-cache-clear-images in an interactive defun and call it from the buffer in which the file is opened - no luck.

I'm using emacs 24.5.1 64bit on debian unstable in a virtual machine on a Surface Pro 4. Poppler version is 0.38, I think. Can anyone reproduce this problem?

[politza]

Memory managment could definitley be improved.

Emacs has it's own additional image cache, which isn't touched by
pdf-tools. For starters, you could lower the value of
image-cache-eviction-delay.

[felixpinkert]

Many thanks for the super quick reply - you're a star! I haven't thought of emacs having its own cache, so despaired at trying to make pdf-tools release memory. But your tipp fixed the issue well enough for now. I reduced image-cache-eviction-delay to 5 seconds.Emacs now still uses some 400 MB in total, way more than I'd expect for just storing images for 2 rendered pages (I reduced the limit to that), but then does not keep growing, and at times also releases memory back, e.g. a bit after (way more than 5 seconds) I kill the pdf-view buffer. I'm very glad I can keep using my nice setup for pdf reading!

[politza]

Each displayed page uses about .5 MB, which adds up.

[nnicandro]

Couldn't we use image-flush to release memory? For example, keeping track of the image spec used in the call to image-size in pdf-cache--prefetch-pages and then calling image-flush on the saved image spec when releasing pages in pdf-cache-put-image.

Also it would probably be a good idea to call image-flush before displaying images in pdf-isearch-hl-matches and pdf-view-display-region. I have found Emacs eating up a large amount of memory in these cases as well.

[politza]

Flushing the image after removal from the cache sounds good.

The other functions you mentioned don't use this cache and they may produce many images. So, I'm thinking of calling clear-image-cache after these functions are finished.

[nnicandro]

For pdf-isearch-hl-matches and pdf-view-display-region, I was thinking more of calling image-flush on the current image being displayed before displaying the new image but calling clear-image-cache also works.

[yantar92]

I have similar problem with emacs not releasing memory after opening several pdfs. The memory is not released even after closing the pdf buffers.

The graph was generated with the following config

(setq init-flag t)

(eval-and-compile
  (defvar bootstrap-version)
  (let ((bootstrap-file
	 (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
	(bootstrap-version 5))
    (unless (file-exists-p bootstrap-file)
      (with-current-buffer
	  (url-retrieve-synchronously
	   "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
	   'silent 'inhibit-cookies)
	(goto-char (point-max))
	(eval-print-last-sexp)))
    (load bootstrap-file nil 'nomessage)))

(eval-and-compile (straight-use-package 'use-package))

(use-package pdf-tools
  :if init-flag
  :straight t
  :magic ("%PDF" . pdf-view-mode)
  :config
  (pdf-tools-install)
  )
(setq large-file-warning-threshold nil)
(setq image-cache-eviction-delay 5)

;; Open 92 pdfs
;;(find-file "~/Books/articles/_symlinks/[a-d]*.pdf" 'wild) 
;; Then keep pressing 'q' untill all the pdf buffers are buries
;;Kill all the pdf buffers
;;(mapc #'kill-buffer (seq-filter (apply-partially #'string-match ".+.pdf$") (mapcar #'buffer-name (buffer-list))))
;; Try to trigger all kinds of cache clearing
;;(clear-image-cache t)
;;(garbage-collect)

Not sure if the problem is with pdf-tools or with emacs itself.
Any ideas?

[politza]

Have you read #177 (comment) ?

[yantar92]

Yes, I have tried the suggestion from the comment. My example has (setq image-cache-eviction-delay 5), which makes things a little different. Basically, the first memory consumption drop in the graph appears after reducing image-cache-eviction-delay. However, the second part of the graph does not seem to be affected. So, there is no real improvement in the memory usage in the long term.

[politza]

How did you measure memory consumption ? 

[yantar92]

I followed http://notes.secretsauce.net/notes/2015/10/05_memory-leak-debugging-tools.html

[yantar92]

Hmm. I actually observe similar problem when just opening a bunch of images. Not just with pdfs.

[yantar92]

I have reported the bug in emacs-devel (here and here).

I am currently trying to run my emacs with jemalloc memory allocator. The total memory consumption seems to be around twice lower in my system after opening many pdfs over the last week.

[moritzschaefer]

might this one be related: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37006 ?

In any case, the bug report referenced by @yantar92 seems to fixed now (https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38345) (in version 28.1)

[mooseyboots]

i find this issue pretty frustrating when i zoom any pdf files.

a little hack to release some memory on closing: run (clear-image-cache) on closing a pdf file, such as:

(defun pdf-tools-kill-this-buffer ()
  "Clear the image cache (to release memory) after killing a pdf buffer."
  (interactive)
  (kill-this-buffer)
  (clear-image-cache))

(define-key pdf-view-mode-map (kbd "Q") 'pdf-tools-kill-this-buffer)  

found in the emacs-dev list linked above.

[yuuyins]

In any case, the bug report referenced by @yantar92 seems to fixed now (https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38345) (in version 28.1)

no they just closed the bug report because they found it to not be a memory leakage but the fact the memory allocator does not returns memory to the OS, and there wasn't follow up in fixing it. I'm still experiencing this issue in 28.1. I know this repo is not updated anymore, but just wanted to leave this here for anyone who is going to read it in the future.

If that's the case, then this bug report can be closed?

I think so.

Lars Ingebrigtsen <larsi gnus.org> writes:

Eli Zaretskii <eliz gnu.org> writes:

However, looking at the C code, I don't understand then why the
memory consumption is increasing. I can only see that too many images in
c->images array can cause extra memory consumption, which cannot cause
the observed memory leak.

I guess that's because the freed memory is not at the end of the heap,
and this cannot be returned to the OS.

This was half a year ago, and skimming this bug report, it seems like
this was the tentative conclusion -- there's no memory leak, just the
allocator not returning the memory back to the OS.

If that's the case, then this bug report can be closed?

--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no